Remote control video stream dynamic hierarchical transmission system and method

Abstract

The invention is suitable for the field of video stream data transmission processing, and provides a remote control video stream dynamic hierarchical transmission system and method, and the system comprises an image collection module which is used for capturing an original video stream; the image preprocessing module is used for converting a grayscale image; the state analysis module is used for comparing continuous gray frames; the image classification unit divides an image into a plurality of blocks and classifies an overall image state into a full dynamic state, a local dynamic state or a static state; the transmission control module is used for providing bandwidth optimization for data transmission and correspondingly calling the full dynamic processing unit, the local dynamic optimization unit and the static control unit; the network transmission cache module is used for receiving a decision of the transmission control module; and the output module is used for outputting pictures. Therefore, a transmission strategy can be dynamically adjusted according to the change degree of the picture, transmission is paused when the picture is static, only difference blocks are sent when the picture is locally changed, full-frame-rate transmission is started when the picture is fully dynamic, the picture state is automatically identified, and graded accurate distribution of transmission resources is realized.

Description

Technical Field

[0001] The present invention is applicable to the field of video stream data transmission and processing, and provides a remote control video stream dynamic hierarchical transmission system and method. Background Art

[0002] Remote video streaming is a technology that transmits video content from one device to another over a network. It is commonly used in scenarios such as remote monitoring, video conferencing, teleworking, and online learning. Its main purpose is to transmit real-time video data over the network so that remote users can view and interact with the video content on their terminal devices.

[0003] Typically, remote video streaming involves encoding video data and transmitting it over a network. During this process, the video signal is captured by an acquisition device and encoded into digital data. This data is then transmitted over the network to the receiving end, where it is decoded and displayed. To ensure real-time transmission and smooth video playback, the video is typically compressed to reduce the data volume. This ensures stable video transmission even with limited network bandwidth.

[0004] Existing remote control technologies typically use a fixed frame rate for continuous video streaming. Regardless of whether the video content changes, the transmission strategy remains consistent. This means that even if the image is static or only partially changes (such as mouse movement), all frames are continuously pushed through full frame rate transmission, resulting in wasted network bandwidth. Mouse synchronization delay is also a common problem. In remote control, the mouse positions of the controlled and remote ends need to be linked in real time, which results in frequent transmission of mouse coordinate information, increasing network load and latency. Even when the image content remains static, the static image still occupies transmission resources and cannot effectively adapt to unstable or weak network bandwidth environments, resulting in unnecessary bandwidth waste and a negative user experience. Summary of the Invention

[0005] In view of the above-mentioned shortcomings, the present invention aims to provide a remote control video stream dynamic hierarchical transmission system and method, in order to solve the problems raised in the background technology. The system includes:

[0006] An image acquisition module for capturing the original video stream of the monitored terminal screen; the image acquisition module can intercept and read the frame buffer and output the underlying graphics;

[0007] Image preprocessing module for converting underlying graphics into grayscale images;

[0008] A state analysis module for comparing successive grayscale frames; the state analysis module divides the image into multiple blocks, quantifies the structural similarity between the current frame and the previous frame in each block, and classifies the overall image state as fully dynamic, partially dynamic, or static;

[0009] A transmission control module for providing bandwidth optimization for data transmission; the transmission control module corresponds to three overall picture state classifications, and respectively calls a full dynamic processing unit, a local dynamic optimization unit, and a static control unit;

[0010] A network transmission buffer module for receiving decisions made by the transmission control module;

[0011] Output module for receiving data and outputting images.

[0012] A remote control video stream dynamic hierarchical transmission method is based on a remote control video stream dynamic hierarchical transmission system, and the method includes the following steps:

[0013] S1, the image acquisition module collects the frame buffer of the port in real time, obtains the underlying graphics of the screen image to be refreshed to the output end, and passes it to the image preprocessing module;

[0014] S2, the image preprocessing module converts the image into a grayscale image;

[0015] S3, the state analysis module compares and classifies the previous and next frames, and classifies the overall picture state into three overall state identifiers: "full dynamic, partial dynamic or static";

[0016] For the local dynamic "overall state identifier", the system divides the image into n×n blocks, defines the block state identifier corresponding to each block, and marks the blocks with the lowest similarity as dynamically changing blocks;

[0017] S4. The transmission control module selects a corresponding sub-processing unit to process and transmit the data according to the received overall status identifier.

[0018] S5. The network transmission cache module performs network transmission or caches data according to the transmission unit called by the transmission control module;

[0019] S6. The remote end renders the received data according to the current mode and outputs the rendered image.

[0020] Furthermore, the state analysis module uses difference calculation to perform comparison and classification of previous and next frames using the structural similarity index SSIM.

[0021] Furthermore, the structural similarity index SSIM is used to quantify the similarity between two frames of images. Its calculation combines the comparison of three dimensions: brightness, contrast, and structure. The formula is as follows:

[0022]

[0023] C1=(k1L) 2 ;

[0024] C2=(k2L) 2 ;

[0025] Where: μx, μy: the average pixel values ​​of images x and y, representing brightness; σx, σy: the standard deviations of images x and y, representing contrast; σxy: the covariance of images x and y, representing structural similarity; C1, C2: stability constants to prevent the denominator from being zero, usually k1 = 0.01, k2 = 0.03, and L is the dynamic range of pixel values.

[0026] Furthermore, in step S4, the transmission control module:

[0027] For full motion, the full motion processing unit is used to encode the complete frame sequence after calling the encoder inside the transmission control module to decide the overall transmission;

[0028] For local dynamics, the local dynamic optimization unit is used to locate the changing area, and only the underlying graphics corresponding to the dynamically changing blocks are selectively encoded, and the code is determined and transmitted;

[0029] For static, the static control unit is used to pause the video stream transmission and start the "heartbeat frame mechanism", periodically sending heartbeat frames to verify the screen status.

[0030] Furthermore, in step S4, the heartbeat frame mechanism generates a low-resolution detection frame that only occupies 1 / 20 of the original data volume at every specified time interval, and the specified time interval is 5S.

[0031] Furthermore, the heartbeat frame mechanism includes the steps: S4.1, when the "heartbeat frame mechanism" detects that the remote end's active operation exceeds the set value, it packages and transmits the mouse coordinate data, and only outputs the processed video data packet obtained after the "heartbeat frame mechanism" is triggered to the network transmission module.

[0032] Furthermore, the system is optimized by using an operation event binding mechanism; the mouse coordinates can be packaged and transmitted when an actual click on the remote end is detected or the state analysis module obtains a local dynamic result according to the above method.

[0033] Furthermore, during the operation of the heartbeat frame mechanism, if the heartbeat detection fails three times in a row, the edge cache mechanism is activated to temporarily store the image data locally; after the network is restored, the accumulated difference frames are transmitted first.

[0034] This system addresses the bandwidth waste caused by fixed frame rate transmission by implementing precise transmission control through the SSIM state recognition mechanism. In static scenes, the system suspends regular video streaming and uses low-resolution heartbeat frames to detect dynamic changes, effectively resolving the issue of static images constantly occupying the transmission channel. For locally dynamic scenes, the system locates the changing areas through grayscale, extracting and transmitting only the changed pixel blocks, thus avoiding the waste of full-frame transmission resources due to small local changes.

[0035] To address the issue of mouse synchronization delay, the system uses an operation event binding mechanism for optimization, which solves the problem of hundreds of empty coordinate transmissions per second when there is no operation, and improves the lag caused by frequent coordinate synchronization.

[0036] Regarding the issue of network load adaptability, when the network is normal, the heartbeat frame mechanism only maintains a connection overhead of 1 / 20 of the regular data volume; when three consecutive heartbeats time out (network interruption), the edge cache is automatically started to temporarily store the image differences in local storage; after the network is restored, the backlog of changed blocks is prioritized for retransmission, effectively preventing connection interruptions caused by continuous retransmission of traditional solutions, and ensuring operational continuity in weak network environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 Flowchart of a system and method for remotely controlling dynamic hierarchical transmission of video streams; DETAILED DESCRIPTION

[0038] The following will be combined with the accompanying drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the embodiments described are only part of the embodiments of the present invention, rather than all the embodiments. It should be understood that the specific embodiments described herein are only used to illustrate the present invention and are not intended to limit the present invention.

[0039] The remote control video stream dynamic hierarchical transmission system includes six core modules: image acquisition module, image preprocessing module, state analysis module, transmission control module, network transmission cache module, and output module.

[0040] The image acquisition module serves as the data input, directly capturing the raw video stream from the monitored screen. Specifically, this module directly intercepts and reads the frame buffer, enabling low-level graphics output. Since the system's GPU renders images in the frame buffer, operating system-level APIs (such as Windows DirectX or Linux DRM) intercept the underlying graphics before they are refreshed to the screen at 50ms intervals, enabling the fastest acquisition of current raw comparison image samples.

[0041] The image preprocessing module converts it into a grayscale image with a resolution of 1080P, that is, it performs dimensionality reduction on the underlying graphics in the frame buffer. The frame sequence after dimensionality reduction is output to the downstream state analysis module.

[0042] Typically, if a system uses the RGB888 color mode to process frame buffer images, comparing the image samples taken before and after the comparison consumes significant computing resources. RGB888 is a 24-bit color mode, where "888" represents 8 bits each for red, green, and blue. This means each color has 2^8 = 256 levels, from 0 to 255. Therefore, the total number of colors is 256 * 256 * 256 = 16,777,216. While RGB565 has fewer colors, each pixel uses only two bytes. However, comparing the two samples still requires significant computing resources. Therefore, this step first performs dimensionality reduction on the collected comparison samples, grayscaling the images captured in the frame buffer at a specified time length to simplify the information capacity of the comparison image samples.

[0043] The state analysis module receives continuous grayscale frames from the image preprocessing module and performs comparative processing on them. This module is implemented using the GPU-accelerated SSIM perception model (Structure Similarity Index Measure) computation engine. It divides the image into multiple blocks, such as 16×16 blocks, and processes them in parallel, quantifying the structural similarity between the current frame and the previous frame for each block.

[0044] Specifically, the structural similarity judgment is actually to classify the overall picture state into fully dynamic, partially dynamic, or static based on the preset SSIM threshold (0.8 / 0.999). The three states of the overall picture, the "overall state identifier", correspond to different working modes of the transmission control module;

[0045] More specific:

[0046] Under the overall state identifier of the static state, only the overall state identifier is transmitted to the system internal log via the transmission control module, and the static underlying graphics are still temporarily stored in blocks as the front sample for block comparison with the rear sample of the local dynamic state, but the static underlying graphics are not subsequently rendered and refreshed.

[0047] Based on the overall state identifier of the local dynamic state, the system defines a block state identifier for each of the n×n blocks divided by the SSIM perception model. Blocks with the lowest similarity are marked as "dynamically changing blocks." This means that similarity is determined between the corresponding block in the "previous sample stored in the block" and the "dynamically changing block." The underlying graphics corresponding to this "dynamically changing block" are subsequently rendered and refreshed on the output image. The similarity determination results for the underlying graphics of other blocks are only transmitted to the system's internal log via the transmission control module and are not subsequently rendered or refreshed.

[0048] Under the overall state identifier of the fully dynamic state, the system renders and refreshes the entire underlying graphics.

[0049] The above analysis results transmit each block with dynamic identification together with the original frame data to the transmission control module to provide a judgment basis for hierarchical transmission.

[0050] The transmission control module provides bandwidth optimization for data transmission. Its corresponding status analysis module's overall image status classification (full dynamic, partial dynamic, or static) consists of three sub-processing units:

[0051] The full dynamic processing unit is activated when SSIM < 0.8, and the encoder inside the transmission control module is called to encode the complete frame sequence and then decide on the overall transmission.

[0052] The local dynamic optimization unit is activated when 0.8≤SSIM<0.999. By locating the changed area, it selectively encodes only the underlying graphics corresponding to the dynamically changed blocks and decides on the transmission of the code.

[0053] When SSIM ≥ 0.999, the static control unit takes over the system, suspends the video stream and starts the "heartbeat frame mechanism", that is, it generates a low-resolution detection frame that only accounts for 1 / 20 of the original data volume at each specified time interval. That is, when the static control unit receives the overall status identifier of the static state, the static control unit suspends streaming and periodically sends heartbeat frames to verify the picture status.

[0054] The "heartbeat frame mechanism" is designed to synchronously detect mouse events. Mouse coordinate data is only packaged and transmitted when active remote manipulation exceeds a set threshold, preventing invalid position information from occupying bandwidth. Only processed video packets generated after the "heartbeat frame mechanism" is triggered are output to the network transmission module.

[0055] The network transmission buffer module receives the decision of the transmission control module and transmits the data to the output module.

[0056] See also Figure 1The application aims to provide a remote control video stream dynamic grading transmission method, based on the above remote control video stream dynamic grading transmission system, comprising the steps of:

[0057] S1, the image acquisition module of the controlled end acquires the frame buffer of the port in real time to obtain the bottom layer graphics of the screen picture to be refreshed to the output end, and transmits to the image preprocessing module.

[0058] S2, the image preprocessing module converts the picture into a gray image.

[0059] S3, the state analysis module performs frame comparison and classification; the difference calculation uses the structural similarity index SSIM to compare the current frame (the latter sample) and the previous frame (the former sample).

[0060] Specifically, the structural similarity index is used to classify the overall picture state into three states of "overall dynamic, local dynamic or static" "overall state identifier", so as to correspond to different modes of subsequent transmission.

[0061] For the "overall state identifier" of local dynamic, the system defines the block state identifier corresponding to each block according to the n×n blocks divided by the SSIM perception model, and marks the blocks with low similarity (reaching the SSIM threshold) as "dynamic change blocks".

[0062] Specifically, the structural similarity index SSIM is used to quantify the similarity between two frames of images, which combines the comparison of three dimensions of luminance (Luminance), contrast (Contrast) and structure (Structure). The formula is as follows:

[0063]

[0064] In the formula, the parameters are defined as follows:

[0065] μx, μy: the average value of pixels of images x and y, representing luminance;

[0066] σx, σy: standard deviation of images x and y, representing contrast;

[0067] σxy: covariance of images x and y, representing structural similarity;

[0068] C1=(k1L) 2 ;

[0069] C2=(k2L) 2 :

[0070] C1 and C2 are stable constants to prevent the denominator from being zero. Usually, k1=0.01 and k2=0.03 are taken. L is the dynamic range of pixel value (such as L=255 for 8-bit image and L=1023 for 10-bit image).

[0071] As an embodiment, the specific calculation steps are:

[0072] The image is divided into blocks of 16×16 pixels, and the SSIM value is calculated block by block. The SSIM values ​​of all blocks are averaged to obtain a global similarity score, which ranges from 0 to 1, with 1 indicating complete consistency.

[0073] The status is determined based on the score: SSIM < 0.8 corresponds to full dynamics, i.e., the entire image changes; 0.8 ≤ SSIM < 0.95 corresponds to partial dynamics, such as only the mouse or a small area changes; SSIM ≥ 0.95 corresponds to statics, i.e., no image changes.

[0074] S4. Based on the status conclusions drawn by the status analysis module, the transmission control module optimizes bandwidth for data transmission and determines the data transmission mode. The three sub-processing units process and transmit data in the three states of "full dynamic, partial dynamic, or static" classified by the status analysis module.

[0075] For full motion, the full motion processing unit is used to encode the complete frame sequence after calling the encoder inside the transmission control module to decide the overall transmission;

[0076] For local dynamics, the local dynamic optimization unit is used to locate the changing area, and only the underlying graphics corresponding to the dynamically changing blocks are selectively encoded, and the code is determined and transmitted;

[0077] For static conditions, the static control unit is used to pause the video stream transmission and start the "heartbeat frame mechanism", that is, a low-resolution detection frame that only accounts for 1 / 20 of the original data volume is generated at each specified time interval, and heartbeat frames are periodically sent to verify the screen status.

[0078] S4.1. The "heartbeat frame mechanism" packages and transmits mouse coordinate data only when it detects that the remote end's active operation exceeds the set value. Only the processed video data packets obtained after the "heartbeat frame mechanism" is triggered are output to the network transmission module to avoid invalid position information occupying bandwidth.

[0079] To address mouse synchronization delays, the system uses an event binding mechanism to optimize mouse coordinates. This allows for data transmission when a click is detected on the remote end or when the state analysis module detects local dynamic results using the aforementioned methods. This eliminates hundreds of empty coordinate transmissions per second when no operation is performed, and reduces the lag caused by frequent coordinate synchronization.

[0080] It is worth explaining that SSIM ≥ 0.999 is one of the implementations of the "heartbeat frame mechanism" to detect mouse events. Technicians can adjust the trigger threshold according to actual needs, such as the pixel ratio of the image cursor on the overall underlying graphics. In this case, the 1 / 20 low-resolution detection frame defined by the "heartbeat frame mechanism" will also change.

[0081] An embodiment of the heartbeat frame mechanism:

[0082] Every 5 seconds, a low-resolution heartbeat frame (Ht) is sent from the controlled end to the remote end. After the remote end receives Ht, it triggers the controlled end to re-execute step S101 (acquiring the current frame Ft); the SSIM value of Ft and the previous valid frame Ft-5s is calculated: if SSIM < 0.99: it is determined that the picture has changed, the static mode is exited, and the current static picture is refreshed; if SSIM ≥ 0.999: the static mode is maintained and the next heartbeat cycle is continued.

[0083] If three consecutive heartbeat detection failures (network interruption) occur, the edge caching mechanism is activated to temporarily store the image data locally. Once the network is restored, the accumulated difference frames are transmitted first. This eliminates the problem of pseudo-static image changes being unrecognizable under high-similarity SSIM.

[0084] S5. The network transmission cache module performs network transmission or caches data according to the transmission unit called by the transmission control module.

[0085] S6. The remote end outputs the received data by rendering the screen according to the current mode, that is, refreshing the latest screen on the remote screen.

[0086] In summary, the core of this remote-controlled video streaming dynamic hierarchical transmission system lies in intelligently identifying the degree of image change and adjusting the data transmission strategy based on actual dynamic needs. When the controlled computer screen is essentially static, the system significantly reduces data transmission. When the image changes locally (such as when the mouse moves), only the small changed area is transmitted. Only when the entire screen changes dramatically does full frame rate transmission occur. This hierarchical processing achieves this effect by first employing the SSIM algorithm, which quantifies image differences, then establishing a three-level state determination mechanism, and finally by binding heartbeat frames to operations.

[0087] The operating process is as follows: After the controlled end starts, the image acquisition module directly captures the original image from the graphics card buffer, and the preprocessing module immediately converts it to grayscale to reduce subsequent processing. The state analysis module cuts the current frame into 16×16 pixel blocks and calculates the SSIM similarity of each block with the previous frame. If the average similarity is less than 0.8 (such as when playing a video), full-frame encoding and transmission is triggered; if it is between 0.8 and 0.999, only the changed blocks are marked for transmission; if it reaches above 0.999, the system enters a dormant state.

[0088] Correspondingly, the transmission control module operates in three modes: encoding the entire frame for full motion; compressing only the changed blocks for partial motion; and pausing transmission during static conditions and initiating an intelligent heartbeat mechanism—sending a detection frame to the remote end every 5 seconds. The system synchronizes coordinate data only when the user operates the mouse, avoiding idle transmissions when no operation is performed. The network module continuously monitors the connection, automatically caching image changes in the event of a network interruption and prioritizing retransmission upon restoration.

[0089] The remote output module intelligently renders based on packet type: Receiving full-frame data results in a full refresh; receiving block data results in a partial update; and maintaining the original image during static conditions. When the heartbeat frame detects a change in the image (such as a sudden user action), it immediately wakes up the transmission channel, achieving a "zero-touch, zero-traffic, and zero-delay" experience.

[0090] Of course, the present invention may have many other embodiments. Without departing from the spirit and essence of the present invention, those skilled in the art may make various corresponding changes and modifications based on the present invention, but these corresponding changes and modifications should all fall within the scope of protection of the claims attached to the present invention.

Claims

1. A remote control video stream dynamic hierarchical transmission system, characterized in that: include: An image acquisition module for capturing the original video stream of the monitored terminal screen; The image acquisition module can intercept and read the frame buffer and output the underlying graphics; Image preprocessing module for converting underlying graphics into grayscale images; A state analysis module for comparing successive grayscale frames; the state analysis module divides the image into multiple blocks, quantifies the structural similarity between the current frame and the previous frame in each block, and classifies the overall image state as fully dynamic, partially dynamic, or static; A transmission control module for providing bandwidth optimization for data transmission; the transmission control module corresponds to three overall picture state classifications, and respectively calls a full dynamic processing unit, a local dynamic optimization unit, and a static control unit; A network transmission buffer module for receiving decisions made by the transmission control module; Output module for receiving data and outputting images.

2. A remote control video stream dynamic hierarchical transmission method, characterized in that: Based on claim 1, the remote control video stream dynamic hierarchical transmission system comprises the following steps: S1, the image acquisition module collects the frame buffer of the port in real time, obtains the underlying graphics of the screen image to be refreshed to the output end, and passes it to the image preprocessing module; S2, the image preprocessing module converts the image into a grayscale image; S3, the state analysis module compares and classifies the previous and next frames, and classifies the overall picture state into three overall state identifiers: "full dynamic, partial dynamic, or static"; For the local dynamic "overall state identifier", the system divides the image into n×n blocks, defines the block state identifier corresponding to each block, and marks the blocks with the lowest similarity as dynamically changing blocks; S4. The transmission control module selects a corresponding sub-processing unit to process and transmit the data according to the received overall status identifier; S5. The network transmission cache module performs network transmission or caches data according to the transmission unit called by the transmission control module; S6. The remote end renders the received data according to the current mode and outputs the rendered image.

3. The remote control video stream dynamic hierarchical transmission system and method according to claim 2, characterized in that: The state analysis module uses difference calculation to perform front and back frame comparison and classification using the structural similarity index SSIM.

4. The remote control video stream dynamic hierarchical transmission method according to claim 3, characterized in that: The structural similarity index SSIM is used to quantify the similarity between two frames of images. Its calculation combines the comparison of three dimensions: brightness, contrast, and structure. The formula is as follows: C1=(k1L) 2 ; C2=(k2L) 2 ; Where: μx, μy: the average pixel values ​​of images x and y, representing brightness; σx, σy: the standard deviations of images x and y, representing contrast; σxy: the covariance of images x and y, representing structural similarity; C1, C2: stability constants to prevent the denominator from being zero, usually k1 = 0.01, k2 = 0.03, and L is the dynamic range of pixel values.

5. The remote control video stream dynamic hierarchical transmission method according to claim 2, characterized in that: In step S4, the transmission control module: For full motion, the full motion processing unit is used to encode the complete frame sequence after calling the encoder inside the transmission control module to decide the overall transmission; For local dynamics, the local dynamic optimization unit is used to locate the changing area, and only the underlying graphics corresponding to the dynamically changing blocks are selectively encoded, and the code is determined and transmitted; For static, the static control unit is used to pause the video stream transmission and start the "heartbeat frame mechanism", periodically sending heartbeat frames to verify the screen status.

6. The remote control video stream dynamic hierarchical transmission method according to claim 5, characterized in that: In step S4, the heartbeat frame mechanism generates a low-resolution detection frame that only occupies 1 / 20 of the original data volume at every specified time interval, and the specified time interval is 5S.

7. The remote control video stream dynamic hierarchical transmission method according to claim 2, characterized in that: The heartbeat frame mechanism includes the following steps: S4.1, when the "heartbeat frame mechanism" detects that the remote end's active operation exceeds the set value, it packages and transmits the mouse coordinate data, and only outputs the processed video data packet obtained after the "heartbeat frame mechanism" is triggered to the network transmission module.

8. The remote control video stream dynamic hierarchical transmission method according to claim 2, characterized in that: The system is optimized by using an operation event binding mechanism; the mouse coordinates can be packaged and transmitted when an actual click on the remote end is detected or the state analysis module obtains a local dynamic result according to the above method.

9. The remote control video stream dynamic hierarchical transmission method according to claim 7, characterized in that: During the operation of the heartbeat frame mechanism, if the heartbeat detection fails three times in a row, the edge cache mechanism is activated to temporarily store the image data locally; after the network is restored, the accumulated difference frames are transmitted first.

Images