A multi-window video signal low-delay processing method, system, device and medium

Abstract

The present invention discloses a multi-window video signal low-delay processing method, system, device and medium, the method comprising obtaining original data and window data and analyzing them to obtain a scaling ratio; adjusting according to the scaling ratio, original data and window data to obtain a scaling factor; adjusting according to the scaling factor and original data to obtain the number of window video frames; calculating according to the scaling factor to obtain the window video size; controlling the number of window video frames to obtain the window video playback speed; sampling according to the window data to obtain the sampled data; calculating according to the sampled data to obtain the delay mean; updating according to the delay mean and buffering time to obtain the display buffering time; performing self-adaptive display on the window video according to the window video size, the window video playback speed and the display buffering time to obtain a multi-window synchronous display video image. The method can adjust the window video display image in real time by calculating the scaling factor to improve the video playback effect.

Description

Technical Field

[0001] The present invention relates to the field of video processing technology, and in particular to a method, system, device and medium for low-delay processing of multi-window video signals. Background Art

[0002] In the field of video playback and display, as users' diversified demands for video content presentation effects continue to grow, video image processing technology has become increasingly complex and sophisticated. In order to achieve a richer visual experience and meet the display requirements in specific scenarios, such as multi-screen monitoring, video content comparison and analysis, etc., the original video image is often processed in blocks, that is, it is divided into multiple windows and then displayed independently in each window. However, in actual applications, due to differences in image data processing, transmission, and display device performance in each block area, the image presentation and playback speed will be inconsistent when displaying different windows. This difference ultimately causes the problem of display asynchrony, which seriously affects the smoothness and integrity of users' video viewing and reduces the accuracy and effectiveness of video content communication. This problem needs to be solved urgently to promote the widespread application of video block display technology in various scenarios.

[0003] In one prior art, a specific implementation includes: first, using a collection device, such as multiple cameras, to shoot a scene from different angles to obtain multiple sets of video source data. These data will be transmitted to a video processing chip or processor, and the chip will divide the original video image into multiple areas according to a preset block rule, and generate multiple video windows accordingly.

[0004] However, the prior art only processes the video window in real time, and does not solve the problem of image stretching or overlapping when multiple sub-window videos are displayed in the prior art, resulting in poor video playback effect. Summary of the invention

[0005] The present invention provides a multi-window video signal low-latency processing method, system, device and medium to achieve real-time adjustment of displayed video images by calculating a scaling factor, effectively solving the problem of image stretching or overlapping when displaying multiple sub-window videos in the prior art, thereby improving the window video playback effect.

[0006] In a first aspect, in order to solve the above technical problems, the present invention provides a method for low-delay processing of multi-window video signals, comprising:

[0007] Get original video data and window video data;

[0008] Performing a ratio analysis based on the original video data and the window video data to obtain a scaling ratio;

[0009] Performing proportional adjustment according to the scaling ratio, the original video data, and the window video data to obtain a scaling factor;

[0010] Adjust the frame number according to the scaling factor and the original video data to obtain the frame number of the window video image;

[0011] Calculate the size of the window video according to the scaling factor to obtain the window video image size;

[0012] The window video is controlled in real time according to the frame number of the window video image to obtain the window video image playing speed;

[0013] Sampling is performed according to the window video data to obtain sampled data;

[0014] Performing weighted mean calculation according to the sampled data to obtain a delay mean;

[0015] Performing update calculation according to the delay mean and the pre-stored buffer time to obtain the display buffer time;

[0016] The window video image is adaptively displayed according to the window video image size, the window video image playback speed and the display buffer time to obtain multi-window synchronous display video images.

[0017] In an implementable manner of the first aspect, performing ratio analysis on the original video data and the window video data to obtain a scaling ratio includes:

[0018] Performing convolution processing on the original video data and the window video data to obtain a first attention map and a second attention map;

[0019] Performing weighted pixel statistics according to the first attention map and the second attention map to obtain the number of pixels of the first video image and the number of pixels of the second video image;

[0020] A scaling ratio is obtained by performing a ratio calculation based on the number of pixels of the first video image and the number of pixels of the second video image.

[0021] In an implementable manner of the first aspect, the scaling is adjusted according to the scaling ratio, the original video data, and the window video data to obtain a scaling factor, including:

[0022] Performing degree analysis according to the window video data to obtain node degree values;

[0023] Correction is performed according to the node degree value and the scaling ratio to obtain a corrected scaling ratio;

[0024] Calculate the video entropy value according to the original video data and the window video data to obtain a first video entropy value and a second video entropy value;

[0025] Calibration is performed according to the modified scaling ratio, the first video entropy value, and the second video entropy value to obtain a scaling factor.

[0026] In an implementable manner of the first aspect, the performing correction according to the node degree value and the scaling ratio to obtain the corrected scaling ratio includes:

[0027] The corrected scaling ratio is calculated using the following formula:

[0028]

[0029] in, Indicates the correction scaling, Indicates the number of pre-stored windows, Indicates The zoom ratio of each window, Indicates The node degree value of each window, Indicates the sequence number of the window.

[0030] In an implementable manner of the first aspect, adjusting the frame number according to the scaling factor and the original video data to obtain the frame number of the windowed video image includes:

[0031] Extract the number of frames according to the original video data to obtain the number of frames of the original video image;

[0032] Perform pixel analysis using an optical flow algorithm according to the window video data to obtain an optical flow amplitude;

[0033] Summing the optical flow amplitudes of all pixels in the window video image to obtain a total optical flow amplitude;

[0034] The frame number is adjusted according to the scaling factor, the original video image frame number and the sum of the optical flow amplitudes to obtain the window video image frame number.

[0035] In an implementable manner of the first aspect, adjusting the frame number according to the scaling factor, the original video image frame number, and the sum of the optical flow amplitudes to obtain the window video image frame number includes:

[0036] The number of window video image frames is calculated using the following formula:

[0037]

[0038] in, Indicates The number of window video image frames per window, represents the scaling factor, Indicates the number of frames of the original video image. Indicates The sum of the optical flow amplitudes of windows, is the adjustment coefficient.

[0039] In an implementation manner of the first aspect, performing weighted mean calculation according to the sampled data to obtain a delay mean includes:

[0040] The average delay is calculated by the following formula:

[0041]

[0042] in, represents the mean delay, For the The sampling data weight coefficient, Indicates the number of pre-stored sampling data, Indicates the sequence number of the sampling data. Indicates Sampling data.

[0043] In a second aspect, the present invention provides a multi-window video signal low-delay processing system, comprising:

[0044] A data acquisition module, used for acquiring original video data and window video data;

[0045] A ratio analysis module, used for obtaining a scaling ratio based on a ratio analysis of the original video data and the window video data;

[0046] A scaling module, used for performing scaling according to the scaling ratio, the original video data and the window video data to obtain a scaling factor;

[0047] A frame number adjustment module, used to adjust the frame number according to the scaling factor and the original video data to obtain the frame number of the window video image;

[0048] A size calculation module, used to calculate the size of the window video according to the scaling factor to obtain the window video image size;

[0049] A playback speed control module, used to control the window video in real time according to the frame number of the window video image to obtain the window video image playback speed;

[0050] A data sampling module, used for sampling according to the window video data to obtain sampled data;

[0051] A delay mean calculation module, used to perform weighted mean calculation according to the sampled data to obtain a delay mean;

[0052] A buffer time updating module, used for performing update calculation according to the delay mean and the pre-stored buffer time to obtain a display buffer time;

[0053] The result output module is used to perform adaptive video display on the window video image according to the window video image size, the window video image playback speed and the display buffer time to obtain multi-window synchronous display video images.

[0054] In an implementable manner of the second aspect, performing ratio analysis on the original video data and the window video data to obtain a scaling ratio includes:

[0055] Performing convolution processing on the original video data and the window video data to obtain a first attention map and a second attention map;

[0056] Performing weighted pixel statistics according to the first attention map and the second attention map to obtain the number of pixels of the first video image and the number of pixels of the second video image;

[0057] A scaling ratio is obtained by performing a ratio calculation based on the number of pixels of the first video image and the number of pixels of the second video image.

[0058] In an implementable manner of the second aspect, the scaling is performed according to the scaling ratio, the original video data, and the window video data to obtain a scaling factor, including:

[0059] Performing degree analysis according to the window video data to obtain node degree values;

[0060] Correction is performed according to the node degree value and the scaling ratio to obtain a corrected scaling ratio;

[0061] Calculate the video entropy value according to the original video data and the window video data to obtain a first video entropy value and a second video entropy value;

[0062] Calibration is performed according to the modified scaling ratio, the first video entropy value, and the second video entropy value to obtain a scaling factor.

[0063] In an implementable manner of the second aspect, the performing correction according to the node degree value and the scaling ratio to obtain the corrected scaling ratio includes:

[0064] The corrected scaling ratio is calculated using the following formula:

[0065]

[0066] in, Indicates the correction scaling, Indicates the number of pre-stored windows, Indicates The zoom ratio of each window, Indicates The node degree value of each window, Indicates the sequence number of the window.

[0067] In an implementable manner of the second aspect, adjusting the frame number according to the scaling factor and the original video data to obtain the frame number of the windowed video image includes:

[0068] Extract the number of frames according to the original video data to obtain the number of frames of the original video image;

[0069] Perform pixel analysis using an optical flow algorithm according to the window video data to obtain an optical flow amplitude;

[0070] Summing the optical flow amplitudes of all pixels in the window video image to obtain a total optical flow amplitude;

[0071] The frame number is adjusted according to the scaling factor, the original video image frame number and the sum of the optical flow amplitudes to obtain the window video image frame number.

[0072] In an implementable manner of the second aspect, the frame number is adjusted according to the scaling factor, the original video image frame number, and the sum of the optical flow amplitudes to obtain the window video image frame number, including:

[0073] The number of window video image frames is calculated using the following formula:

[0074]

[0075] in, Indicates The number of window video image frames per window, represents the scaling factor, Indicates the number of frames of the original video image. Indicates The sum of the optical flow amplitudes of windows, is the adjustment coefficient.

[0076] In an implementation manner of the second aspect, performing weighted mean calculation according to the sampled data to obtain a delay mean includes:

[0077] The average delay is calculated by the following formula:

[0078]

[0079] in, represents the mean delay, For the The sampling data weight coefficient, Indicates the number of pre-stored sampling data, Indicates the sequence number of the sampling data. Indicates Sampling data.

[0080] In a third aspect, the present invention also provides an electronic device comprising a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein when the processor executes the computer program, the low-delay processing method for multi-window video signals described in any one of the above is implemented.

[0081] In a fourth aspect, the present invention also provides a computer-readable storage medium, which includes a stored computer program, wherein when the computer program is running, the device where the computer-readable storage medium is located is controlled to execute any one of the multi-window video signal low-delay processing methods described above.

[0082] Compared with the prior art, the present invention has the following beneficial effects:

[0083] The present invention discloses a multi-window video signal low-delay processing method, including obtaining original video data and window video data; performing proportional analysis according to the original video data and the window video data to obtain a scaling ratio; performing proportional adjustment according to the scaling ratio, the original video data and the window video data to obtain a scaling factor; performing frame number adjustment according to the scaling factor and the original video data to obtain the number of frames of the window video image; performing size calculation of the window video according to the scaling factor to obtain the size of the window video image; performing real-time control of the window video according to the number of frames of the window video image to obtain the playback speed of the window video image; performing sampling according to the window video data to obtain sampling data; performing weighted mean calculation according to the sampling data to obtain a delay mean; performing update calculation according to the delay mean and a pre-stored buffer time to obtain a display buffer time; performing self-adaptive video display on the window video image according to the window video image size, the window video image playback speed and the display buffer time to obtain a multi-window synchronous display video image. By calculating the scaling factor to adjust the displayed video image in real time, the problem of image stretching or overlapping that occurs when multiple sub-window videos are displayed in the prior art is effectively solved, thereby improving the window video playback effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0084] Figure 11 is a flowchart of a method for low-delay processing of multi-window video signals provided by a first embodiment of the present invention;

[0085] Figure 2 It is a schematic diagram of the structure of a multi-window video signal low-delay processing system provided by the second embodiment of the present invention. DETAILED DESCRIPTION

[0086] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0087] Reference Figure 1 The first embodiment of the present invention provides a multi-window video signal low-delay processing method, comprising the following steps:

[0088] S1, obtain original video data and window video data;

[0089] S2, performing a ratio analysis on the original video data and the window video data to obtain a scaling ratio;

[0090] S3, performing proportional adjustment according to the scaling ratio, the original video data and the window video data to obtain a scaling factor;

[0091] S4, adjusting the frame number according to the scaling factor and the original video data to obtain the frame number of the window video image;

[0092] S5, calculating the size of the window video according to the scaling factor to obtain the window video image size;

[0093] S6, controlling the window video in real time according to the frame number of the window video image to obtain the window video image playback speed;

[0094] S7, sampling according to the window video data to obtain sampled data;

[0095] S8, performing weighted mean calculation according to the sampled data to obtain a delay mean;

[0096] S9, performing update calculation according to the delay mean and the pre-stored buffer time to obtain a display buffer time;

[0097] S10, performing adaptive video display on the window video image according to the window video image size, the window video image playback speed and the display buffer time to obtain multi-window synchronous display video images.

[0098] In step S1, original video data and window video data are obtained.

[0099] In a specific embodiment, the original video data refers to the original video file that has not been processed by too many window segmentation. When processing multi-window video signals, it is necessary to first obtain this original video file, which contains complete video information and is the basis for subsequent calculation of scaling factors and video synchronization processing. Exemplarily, the storage path of the original video file can be found and determined through the search function of the file system or in a specific video library so that its data can be accurately read.

[0100] In a specific embodiment, the acquisition of the window video data is to read the corresponding frame image from the corresponding video file according to the set window position. Since the multi-window video comes from different video source files, it is necessary to accurately extract the frame data from the corresponding file according to the position information of each window. Exemplarily, it can involve the programming interface of the file reading operation, and use these interfaces to locate and read the required image data in the video file according to the coordinate range of the window to ensure that each window can obtain the correct video content.

[0101] In step S2, a scaling ratio is obtained by performing a ratio analysis based on the original video data and the window video data.

[0102] In the above step S2, the ratio analysis is performed according to the original video data and the window video data to obtain the scaling ratio, which specifically includes the following steps:

[0103] S21, performing convolution processing on the original video data and the window video data to obtain a first attention map and a second attention map;

[0104] S22, performing weighted pixel statistics according to the first attention map and the second attention map to obtain the number of pixels of the first video image and the number of pixels of the second video image;

[0105] S23, performing a ratio calculation according to the number of pixels of the first video image and the number of pixels of the second video image to obtain a scaling ratio.

[0106] It should be noted that the above steps S21 to S23 are implemented by using a convolutional neural network (CNN) combined with a self-attention mechanism to process the video frames of each window and the original video frames. For each frame of the image, the self-attention mechanism can focus on the importance of different areas and generate a corresponding attention map. Specifically, suppose the window No. The frame video image is , after passing through the CNN containing the multi-head self-attention module, the output attention map ,Original video Frame Image Generating Attention Map According to the attention map, weights are assigned to pixels to calculate the number of valid pixels. The weighted pixel count of the window video image is calculated as:

[0107]

[0108] Similarly, the weighted pixel count of the original video can be obtained ,here Indicates pixel coordinates. This replaces simple pixel point statistics to obtain a more accurate scaling ratio for multi-window videos when outputting as follows:

[0109]

[0110] in, Display Window The scaling factor, Indicates The weighted pixel count of the video window, Represents the weighted pixel count of the original video, Indicates the sequence number of the window.

[0111] In this embodiment, the first attention map refers to the attention map generated according to the original video image; the second attention map refers to the attention map generated by the window video image; the number of pixels of the first video image refers to the weighted number of pixels obtained by weighted calculation of the attention map generated by the original video image; the number of pixels of the second video image refers to the weighted number of pixels obtained by weighted calculation of the attention map generated by the window video image.

[0112] In step S3, a scaling factor is obtained by performing scaling adjustment according to the scaling ratio, the original video data and the window video data.

[0113] In the above step S3, the scaling is adjusted according to the scaling ratio, the original video data and the window video data to obtain a scaling factor, which specifically includes the following steps:

[0114] S31, performing degree analysis according to the window video data to obtain node degree values.

[0115] It should be noted that the node degree value is to regard the image in each window video data as a node in graph theory, and the visual similarity between windows is used as the weight of the edge. The structural similarity index (SSIM) is used to measure the similarity between windows and construct an undirected weighted graph. ,in Is a collection of window nodes ,side Weight For Window With window Specifically, the edge Weight For Window With window The structural similarity index (SSIM) value of the node is The weights of the edges connected to all other nodes are summed to obtain the degree of the node, which is used to measure the degree of connection between the window and other windows.

[0116] S32, performing correction according to the node degree value and the scaling ratio to obtain a corrected scaling ratio.

[0117] It should be noted that the correction scaling ratio is calculated by the following formula:

[0118]

[0119] in, Indicates the correction scaling, Indicates the number of pre-stored windows, Indicates The zoom ratio of each window, Indicates The node degree value of each window, Indicates the sequence number of the window.

[0120] The formula used in this embodiment is based on the traditional proportional relationship formula, which is modified by introducing the graph Laplace matrix after considering the topological structure of the graph. This formula combines the degree value of the node with the window scaling ratio, so that the calculation of the scaling factor incorporates the visual association information between windows. By correcting the traditional proportional relationship formula through the graph Laplace matrix, it can better reflect the relationship between windows and improve the accuracy of scaling ratio calculation when outputting multi-window videos.

[0121] It should be noted that the Laplacian Matrix is ​​a core concept in graph theory and spectral graph theory, and it plays an important role in the design of graph convolutional neural networks (GCNs). For example, for a graph G with n vertices, its Laplacian matrix is ​​defined as: , where D is the degree matrix of graph G and A is the adjacency matrix of graph G.

[0122] S33, calculating the video entropy value according to the original video data and the window video data to obtain a first video entropy value and a second video entropy value.

[0123] It should be noted that video entropy The distribution disorder of pixel values ​​in the video frame and the changes in the time dimension are comprehensively considered. The video entropy calculation process is to calculate the entropy value of each window video and the original video respectively. For example, by adopting the spatiotemporal entropy calculation method, integrating the intra-frame entropy and the inter-frame entropy, the video entropy value can be obtained.

[0124] In this embodiment, the first video entropy value refers to the entropy value of the original video; the second video entropy value refers to the entropy value of each window video.

[0125] S34, calibrating according to the modified scaling ratio, the first video entropy value, and the second video entropy value to obtain a scaling factor.

[0126] It should be noted that the entropy value is incorporated into the correction scaling ratio as a calibration factor to calculate the scaling factor using the following formula:

[0127]

[0128] in, represents the scaling factor, Indicates the correction scaling, Indicates the number of pre-stored windows, Indicates The entropy value of the window video of the window, Represents the original video entropy value.

[0129] In this embodiment, by introducing video entropy, the scaling factor can better adapt to differences in video content complexity and improve overall calculation accuracy.

[0130] In step S4, the frame number is adjusted according to the scaling factor and the original video data to obtain the frame number of the window video image.

[0131] In the above step S4, the frame number is adjusted according to the scaling factor and the original video data to obtain the frame number of the window video image, which specifically includes the following steps:

[0132] S41, extracting the number of frames according to the original video data to obtain the number of frames of the original video image.

[0133] Exemplarily, the number of frames of the original video image can be directly extracted by inputting instructions to the original video data using a multimedia processing tool.

[0134] S42, performing pixel analysis using an optical flow algorithm based on the window video data to obtain an optical flow amplitude.

[0135] In one feasible method, the original video data is first obtained and input into the selected optical flow algorithm model; based on the basic assumptions of the optical flow algorithm, such as constant brightness, temporal continuity and spatial consistency, the pixels between adjacent frames in the video are analyzed, and the displacement vector of each pixel in the adjacent frame image is calculated to determine the movement direction and speed of the pixel; the square root sum is performed according to the movement direction and speed to obtain the optical flow amplitude. For example, the Lucas-Kanade optical flow algorithm uses the Taylor expansion approximation in a small image block to construct a linear equation system to solve the optical flow vector of the pixel, and then obtain the optical flow amplitude of each pixel.

[0136] It should be noted that the Lucas-Kanade optical flow algorithm is a classic computer vision technique used to estimate the motion of pixels in an image sequence. The algorithm is based on the assumption of constant brightness, that is, the brightness value of the same object point in two consecutive frames of images remains unchanged. The implementation process includes: first selecting feature points in the image, then constructing optical flow constraint equations, and estimating the motion vectors of these feature points by minimizing the brightness error. The Lucas-Kanade method combines the optical flow constraint equations of multiple pixel points in a small window to form a fixed set of linear equations, thereby estimating a unified displacement vector. This method can be used for sparse optical flow estimation, that is, only tracking a subset of certain points in the image, which are corner points or points with obvious features.

[0137] S43, summing up the optical flow amplitudes of all pixels in the window video image to obtain a total optical flow amplitude.

[0138] Exemplarily, the total optical flow amplitude is obtained by summing the optical flow amplitudes of all pixels in the image according to the length and width of the image.

[0139] S44, adjusting the frame number according to the scaling factor, the original video image frame number and the sum of the optical flow amplitudes to obtain the window video image frame number.

[0140] It should be noted that the number of window video image frames is calculated using the following formula:

[0141]

[0142] in, Indicates The number of window video image frames per window, represents the scaling factor, Indicates the number of frames of the original video image. No. The sum of the optical flow amplitudes of windows, is the adjustment coefficient.

[0143] In this embodiment, the adjustment coefficient is a parameter used in dynamic frame rate adaptation, which is used to adjust the number of video frames to adapt to the motion intensity of the video content. Specifically, the adjustment coefficient is multiplied by the sum of the optical flow amplitudes to measure the impact of the motion intensity on the number of frames. For example, when the motion intensity is high, by increasing The value can increase the frame rate appropriately. , thus ensuring the smoothness of video playback.

[0144] In step S5, the size of the window video is calculated according to the scaling factor to obtain the window video image size.

[0145] In an achievable manner, the size of the window video is calculated according to the scaling factor to obtain the size of the window video image, including: obtaining the height of the window video image output and the height of the original video image ; Calculate the number of pixels corresponding to each window video image according to the determined zoom ratio, the window video image height and the original video image height; rearrange the pixels in the window video image from left to right and from top to bottom according to the calculated number of pixels corresponding to each window video image, and keep the pixel arrangement order of the original video image consistent, and use the first pixel in the first row after the reset as the origin of the coordinate system; determine the size of the window video image according to the new pixel arrangement.

[0146] In step S6, the window video is controlled in real time according to the frame number of the window video image to obtain the window video image playback speed.

[0147] For example, the speed of the video image is controlled in real time by setting a timer and adjusting a counter. When playing, first get the end time of the previous video frame, then set the timer. When the timer reaches the time to play each frame, trigger the counter to add one, and the time of each counter addition is equal. When the counter value is not less than the number of video frames corresponding to the window video When the counter value is less than the number of video frames in the window video When the pixel points of the previous window video image are moved sequentially to the coordinate system of the next window video image, each time the counter increments, the counter value increments by one, while maintaining the video frame playback rate per second. Then, get the end time of the next video frame, repeat the above steps to continue playing, and get the window video image playing speed.

[0148] In step S7, sampling is performed according to the window video data to obtain sampled data.

[0149] For example, during the multi-window video playback and display process, the same window is displayed at different positions to observe the duration from the appearance to the disappearance of the window. This operation is continued during the entire video playback, and the time information of the appearance and disappearance of the window at different display positions is collected to form a series of time data sequences. These collected time data are used as sample data for delay testing and are temporarily stored in random access memory (RAM).

[0150] In step S8, a weighted mean is calculated based on the sampled data to obtain a delay mean.

[0151] It should be noted that the average delay is calculated by the following formula:

[0152]

[0153] in, represents the mean delay, For the The sampling data weight coefficient, Indicates the number of pre-stored sampling data, Indicates the sequence number of the sampling data. Indicates Sampling data.

[0154] In this embodiment, the sampling data weight coefficient is assigned according to the different degrees of influence of system noise and network jitter on different sampling points. For example, the sampling point is judged by the degree of influence of factors such as system noise and network jitter. The less affected the sampling point is, the more stable it is, and the higher the weight value is assigned to it.

[0155] In step S9, an update calculation is performed based on the delay mean value and the pre-stored buffer time to obtain the display buffer time.

[0156] It should be noted that the pre-stored buffering time refers to the original video buffering time. The calculation formula for the displayed buffering time calculated based on the delay mean and the original video buffering time is:

[0157]

[0158] in, It is the final display buffer time after adjustment; It is the video buffering time originally set by the display device, which is determined by the characteristics and initial settings of the display device itself; This is the mean delay calculated previously.

[0159] In step S10, the window video image is adaptively displayed according to the window video image size, the window video image playback speed and the display buffer time to obtain a multi-window synchronous display video image.

[0160] For example, in terms of size, the pixels in the window video image are rearranged in order from left to right and from top to bottom, and the pixel arrangement order of the original video image is kept consistent, and the first pixel in the first row after the reset is used as the origin of the coordinate system. In this way, the size of the window video image is determined according to the new pixel arrangement, so that the window video can maintain the same image size ratio as the original video during synchronous playback, effectively solving the problem of image stretching or overlapping when displaying multiple sub-window videos, and ensuring the stability and consistency of the video playback effect.

[0161] For example, in terms of playback speed, the speed of the video image can be controlled in real time by setting a timer and adjusting a counter. When playing, first get the end time of the previous video frame, then set the timer. When the timer reaches the time to play each frame, trigger the counter to add one, and the time for each counter to add one is equal. When the counter value is not less than the number of video frames corresponding to the window video When the counter value is less than the number of video frames in the window video When the pixel points of the previous window video image are moved sequentially to the coordinate system of the next window video image, each time the counter increments, the counter value increments by one, while maintaining the video frame playback rate per second. Then, the end time of the next video frame is obtained, and the above steps are repeated to continue playing, so that the playing speed of the window video image and the original video image remains consistent.

[0162] For example, in terms of signal delay, by using the original video buffering time of the display device Subtract the mean delay , you can get a new display buffer time ,This new buffer time can be adjusted according to the actual delay of the ,video signal, so as to optimize the video playback effect and ,reduce the impact of delay.

[0163] In this embodiment, by adjusting the size, play speed and signal delay of each window video image, multi-window synchronous display video images are obtained.

[0164] In summary, the present invention discloses a low-delay processing method for multi-window video signals, including obtaining original video data and window video data; performing proportional analysis based on the original video data and the window video data to obtain a scaling ratio; performing proportional adjustment based on the scaling ratio, the original video data and the window video data to obtain a scaling factor; performing frame number adjustment based on the scaling factor and the original video data to obtain the number of window video image frames; performing size calculation of the window video based on the scaling factor to obtain the size of the window video image; performing real-time control of the window video based on the number of window video image frames to obtain the window video image playback speed; performing sampling based on the window video data to obtain sampling data; performing weighted mean calculation based on the sampling data to obtain a delay mean; performing update calculation based on the delay mean and a pre-stored buffer time to obtain a display buffer time; performing adaptive video display on the window video image based on the window video image size, the window video image playback speed and the display buffer time to obtain multi-window synchronous display video images. By calculating the scaling factor and adjusting the displayed video image in real time, the problem of image stretching or overlapping when displaying multiple sub-window videos in the prior art is effectively solved, thereby improving the window video playback effect.

[0165] Reference Figure 2 A second embodiment of the present invention provides a multi-window video signal low-delay processing system, comprising:

[0166] The data acquisition module 101 is used to acquire original video data and window video data;

[0167] A ratio analysis module 102, configured to obtain a scaling ratio based on a ratio analysis between the original video data and the window video data;

[0168] A scaling module 103, configured to perform scaling according to the scaling ratio, the original video data and the window video data to obtain a scaling factor;

[0169] A frame number adjustment module 104, configured to adjust the frame number according to the scaling factor and the original video data to obtain the frame number of the window video image;

[0170] A size calculation module 105 is used to calculate the size of the window video according to the scaling factor to obtain the window video image size;

[0171] The play speed control module 106 is used to control the window video in real time according to the frame number of the window video image to obtain the play speed of the window video image;

[0172] A data sampling module 107, configured to perform sampling according to the window video data to obtain sampled data;

[0173] A delay mean calculation module 108, configured to perform weighted mean calculation based on the sampled data to obtain a delay mean;

[0174] A buffer time updating module 109, configured to perform update calculation according to the delay mean and the pre-stored buffer time to obtain a display buffer time;

[0175] The result output module 110 is used to perform adaptive video display on the window video image according to the window video image size, the window video image playback speed and the display buffer time to obtain multi-window synchronous display video images.

[0176] It should be noted that a multi-window video signal low-delay processing system provided in an embodiment of the present invention is used to execute all process steps of a multi-window video signal low-delay processing method in the above embodiment. The working principles and beneficial effects of the two correspond one to one, and therefore will not be repeated here.

[0177] An embodiment of the present invention further provides an electronic device. The electronic device includes: a processor, a memory, and a computer program stored in the memory and executable on the processor, such as a multi-window video signal low-delay processing method program. When the processor executes the computer program, the steps in each of the above multi-window video signal low-delay processing method embodiments are implemented, such as Figure 1 Alternatively, when the processor executes the computer program, the functions of the modules / units in the above-mentioned device embodiments are realized, such as the data acquisition module.

[0178] Exemplarily, the computer program may be divided into one or more modules / units, which are stored in the memory and executed by the processor to implement the present invention. The one or more modules / units may be a series of computer program instruction segments capable of implementing specific functions, which are used to describe the execution process of the computer program in the electronic device.

[0179] The electronic device may be a computing device such as a desktop computer, a notebook, a PDA, and a smart tablet. The electronic device may include, but is not limited to, a processor and a memory. Those skilled in the art will appreciate that the above components are merely examples of electronic devices and do not constitute a limitation on the electronic device. The electronic device may include more or fewer components than the above components, or may combine certain components, or different components. For example, the electronic device may also include input and output devices, network access devices, buses, etc.

[0180] The processor may be a central processing unit (CPU), other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc. The processor is the control center of the electronic device, and uses various interfaces and lines to connect various parts of the entire electronic device.

[0181] The memory can be used to store the computer program and / or module, and the processor realizes various functions of the electronic device by running or executing the computer program and / or module stored in the memory, and calling the data stored in the memory. The memory can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area can store data created according to the use of the mobile phone (such as audio data, a phone book, etc.), etc. In addition, the memory can include a high-speed random access memory, and can also include a non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), at least one disk storage device, a flash memory device, or other volatile solid-state storage devices.

[0182] Wherein, if the module / unit integrated in the electronic device is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on such an understanding, the present invention implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Wherein, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, U disk, mobile hard disk, disk, optical disk, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electric carrier signal, telecommunication signal and software distribution medium. It should be noted that the content contained in the computer-readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electric carrier signals and telecommunication signals.

[0183] It should be noted that the device embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. In addition, in the accompanying drawings of the device embodiments provided by the present invention, the connection relationship between the modules indicates that there is a communication connection between them, which may be specifically implemented as one or more communication buses or signal lines. A person of ordinary skill in the art may understand and implement it without paying any creative effort.

[0184] The specific embodiments described above further illustrate the purpose, technical solutions and beneficial effects of the present invention. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. It is particularly pointed out that for those skilled in the art, any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention should be included in the scope of protection of the present invention.

Claims

1. A method for processing multi-window video signals with low delay, characterized in that: Executed by a computer, including: Get original video data and window video data; Performing a ratio analysis based on the original video data and the window video data to obtain a scaling ratio; Proportional adjustment is performed according to the scaling ratio, the original video data, and the window video data to obtain a scaling factor, including: Performing degree analysis according to the window video data to obtain node degree values; Correction is performed according to the node degree value and the scaling ratio to obtain a corrected scaling ratio; Calculate the video entropy value according to the original video data and the window video data to obtain a first video entropy value and a second video entropy value; Calibrate according to the modified scaling ratio, the first video entropy value and the second video entropy value to obtain a scaling factor; The frame number is adjusted according to the scaling factor and the original video data to obtain the frame number of the window video image, including: Extract the number of frames according to the original video data to obtain the number of frames of the original video image; Perform pixel analysis using an optical flow algorithm according to the window video data to obtain an optical flow amplitude; Summing the optical flow amplitudes of all pixels in the window video image to obtain a total optical flow amplitude; Adjust the frame number according to the scaling factor, the frame number of the original video image and the sum of the optical flow amplitudes to obtain the frame number of the windowed video image; The step of adjusting the frame number according to the scaling factor, the frame number of the original video image and the sum of the optical flow amplitudes to obtain the frame number of the windowed video image includes: The number of window video image frames is calculated using the following formula: in, Indicates The number of window video image frames per window, represents the scaling factor, Indicates the number of frames of the original video image. Indicates The sum of the optical flow amplitudes of windows, is the adjustment coefficient; Calculate the size of the window video according to the scaling factor to obtain the window video image size; The window video is controlled in real time according to the frame number of the window video image to obtain the window video image playing speed; Sampling is performed according to the window video data to obtain sampled data; Performing weighted mean calculation according to the sampled data to obtain a delay mean; Performing update calculation according to the delay mean and the pre-stored buffer time to obtain the display buffer time; The window video image is adaptively displayed according to the window video image size, the window video image playback speed and the display buffer time to obtain multi-window synchronous display video images.

2. The multi-window video signal low-delay processing method according to claim 1, characterized in that: The performing ratio analysis according to the original video data and the window video data to obtain a scaling ratio includes: Performing convolution processing on the original video data and the window video data to obtain a first attention map and a second attention map; Performing weighted pixel statistics according to the first attention map and the second attention map to obtain the number of pixels of the first video image and the number of pixels of the second video image; A scaling ratio is obtained by performing a ratio calculation based on the number of pixels of the first video image and the number of pixels of the second video image.

3. The multi-window video signal low-delay processing method according to claim 1, characterized in that: The step of performing correction according to the node degree value and the scaling ratio to obtain a corrected scaling ratio includes: The corrected scaling ratio is calculated using the following formula: in, Indicates the correction scaling, Indicates the number of pre-stored windows, Indicates The zoom ratio of each window, Indicates The node degree value of each window, Indicates the sequence number of the window.

4. The multi-window video signal low-delay processing method according to claim 1, characterized in that: The weighted mean calculation is performed according to the sampled data to obtain the delay mean, including: The average delay is calculated by the following formula: in, represents the mean delay, For the The sampling data weight coefficient, Indicates the number of pre-stored sampling data, Indicates the sequence number of the sampling data. Indicates Sampling data.

5. A multi-window video signal low-delay processing system, used to implement the multi-window video signal low-delay processing method according to any one of claims 1 to 4, characterized in that: include: A data acquisition module, used for acquiring original video data and window video data; A ratio analysis module, used for obtaining a scaling ratio based on a ratio analysis of the original video data and the window video data; A scaling module, used for performing scaling according to the scaling ratio, the original video data and the window video data to obtain a scaling factor; A frame number adjustment module, used to adjust the frame number according to the scaling factor and the original video data to obtain the frame number of the window video image; A size calculation module, used to calculate the size of the window video according to the scaling factor to obtain the window video image size; A playback speed control module, used to control the window video in real time according to the frame number of the window video image to obtain the window video image playback speed; A data sampling module, used for sampling according to the window video data to obtain sampled data; A delay mean calculation module, used to perform weighted mean calculation according to the sampled data to obtain a delay mean; A buffer time updating module, used for performing update calculation according to the delay mean and the pre-stored buffer time to obtain a display buffer time; The result output module is used to perform adaptive video display on the window video image according to the window video image size, the window video image playback speed and the display buffer time to obtain multi-window synchronous display video images.

6. An electronic device, characterized in that: The method comprises a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein when the processor executes the computer program, the method for low-delay processing of multi-window video signals as described in any one of claims 1 to 4 is implemented.

7. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored computer program, wherein when the computer program is executed, the device where the computer-readable storage medium is located is controlled to execute the multi-window video signal low-delay processing method according to any one of claims 1 to 4.

Images