Method and device for realizing screen projection control based on WIFI-Display protocol

Abstract

The invention discloses a method and device for achieving screen projection control based on a WIFI-Display protocol, and relates to the technical field of wireless communication and multimedia transmission, and the method comprises the steps: firstly, building a WIFI-Display connection session between a source device and a receiving end; secondly, a dynamic updating area of the screen is intelligently captured by monitoring a graphic buffer area and using a difference block detection algorithm, and an original screen data stream is only generated for a changing area so as to improve the efficiency; then, dynamically generating coding control parameters by analyzing image motion complexity and real-time network state parameters; and finally, carrying out real-time coding and adaptive network packaging on the data stream based on the parameter to form a screen projection data stream, and sending the screen projection data stream to a receiving end. Through the above mode, the method effectively reduces the bandwidth consumption and the coding calculation amount, and remarkably improves the transmission fluency and the image quality stability of the dynamic screen content in a complex network environment through the adaptive code rate and the packaging strategy.

Description

Technical Field

[0001] This application relates to the field of wireless communication and multimedia transmission technology, and in particular to a method and apparatus for screen projection control based on the WIFI-Display protocol. Background Technology

[0002] In existing technologies, screen mirroring solutions based on the WIFI-Display protocol typically employ a fixed frame rate and encoding parameters to continuously capture and encode the entire screen image for transmission. This method has significant limitations when facing dynamic scenes or complex network fluctuations: firstly, continuous full-screen encoding generates a large amount of redundant data, especially when only a small portion of the image is updated, resulting in a huge waste of valuable wireless bandwidth resources and high bandwidth consumption; secondly, a fixed encoding strategy cannot adaptively adjust to the motion complexity of the image content and real-time network conditions. When bandwidth is limited or network jitter is severe, it easily leads to increased data transmission latency and packet loss rate, resulting in noticeable video stuttering and image quality degradation at the receiving end, severely impacting the user experience.

[0003] The above content is only used to help understand the technical solution of the present invention and does not represent an admission that the above content is prior art. Summary of the Invention

[0004] The main purpose of this application is to provide a screen projection control method and device based on the WIFI-Display protocol, which aims to solve the technical problems of screen projection lag and high bandwidth consumption in the prior art.

[0005] To achieve the above objectives, this application provides a screen mirroring control method based on the WIFI-Display protocol, the method comprising: Based on the session establishment request initiated by the source device, perform capability negotiation with the receiving device and establish a WIFI-Display connection session; Based on monitoring changes in the graphics buffer, a difference tile detection algorithm is used to extract the dynamically updated region between adjacent frames, and the original screen data stream is obtained based on the dynamically updated region. Determine the motion complexity of the image in the original screen data stream, and generate encoding control parameters based on the real-time feedback network state parameters and the motion complexity; Based on the encoding control parameters, the original screen data stream is encoded in real time and encapsulated using network adaptive methods to obtain a projection data stream. The projection data stream is then sent to the receiving device through an established WIFI-Display connection session. The network adaptive encapsulation dynamically adjusts the encapsulation granularity of the data packets according to the estimated network jitter.

[0006] In one embodiment, the step of extracting dynamically updated regions between adjacent frames using a difference tile detection algorithm based on monitoring changes in the graphics buffer includes: Monitor changes in the graphics buffer to determine the current frame and the reference frame; The current frame and the reference frame are divided into multiple tiles of the same size by a difference tile detection algorithm, and the pixel difference between the tiles is calculated. Based on a preset difference threshold, the patches whose pixel difference exceeds the preset difference threshold are marked as candidate dynamic patches; The candidate dynamic map pieces are subjected to morphological dilation to connect adjacent small variation regions, generating dynamically updated regions.

[0007] In one embodiment, the step of performing morphological dilation on the candidate dynamic map tiles to connect adjacent regions of minor variation and generate the final dynamically updated region includes: Based on the preset convolution kernel size, the candidate dynamic image blocks are dilated to obtain preliminary connected candidate regions; The candidate regions that are initially connected are marked, and adjacent minor variation regions are connected according to the marks to obtain several independent connected regions; Based on the minimum region area threshold, connected components with areas smaller than the minimum region area threshold are filtered to obtain the filtered connected components. A dynamically updated region is generated based on the filtered connected components.

[0008] In one embodiment, the step of obtaining the original screen data stream based on the dynamically updated region includes: Based on the coordinate information of the dynamically updated region, the corresponding region pixel data is extracted from the graphics buffer; Add frame header information to the pixel data of the region to identify its position in the full screen, and generate incremental data frames; During non-first transmission cycles, the incremental data frames are treated as the original screen data stream.

[0009] In one embodiment, the step of adding frame header information to the region pixel data to identify its position in the full screen and generating an incremental data frame includes: A location descriptor is generated based on the coordinates of the top-left corner vertex of the dynamically updated region and the region's width and height information. Generate an inter-frame dependency identifier based on the sequence number of the current frame and the sequence number of the reference frame; The location descriptor and the inter-frame dependency identifier are combined to form frame header information that identifies the location in the full screen; The frame header information is appended before the region pixel data to generate an incremental data frame.

[0010] In one embodiment, the step of determining the motion complexity of an image in the original screen data stream includes: Optical flow calculation is performed on multiple consecutive frames of images in the original screen data stream, and the average motion amplitude between frames is calculated based on the motion vector field of the pixels. Based on image gradient feature detection technology, high-frequency texture regions and key interest regions in the multi-frame images are identified, and the area ratio is calculated based on the high-frequency texture regions and the key interest regions. The motion complexity is obtained based on the average motion amplitude and the area ratio.

[0011] In one embodiment, the step of performing optical flow calculation on multiple consecutive frames of images in the original screen data stream and calculating the average motion amplitude between frames based on the motion vector field of the pixels includes: Optical flow calculation is performed on multiple consecutive frames of images in the original screen data stream to determine the motion vector field of each pixel in the current frame and the previous frame; The intra-frame motion amplitude distribution map is obtained based on the motion vector field of all pixels; The effective motion amplitude is obtained by thresholding the motion amplitude value of each pixel in the intra-frame motion amplitude distribution map. The effective motion amplitude is arithmetically averaged to obtain the average motion amplitude between frames.

[0012] In one embodiment, the step of generating coding control parameters based on real-time feedback network state parameters and the motion complexity includes: Based on the packet loss rate, latency jitter, and available bandwidth periodically fed back from the receiver, a current network status assessment vector is constructed. The motion complexity and the situation evaluation vector are input into a pre-trained reinforcement learning model to obtain the target bit rate, target frame rate, and keyframe insertion strategy for the next time window. Encoding control parameters are generated based on the target bitrate, the target frame rate, and the keyframe insertion strategy.

[0013] In one embodiment, the step of performing real-time encoding and network adaptive encapsulation on the original screen data stream based on the encoding control parameters to obtain the projection data stream includes: Based on the target bitrate and target frame rate in the encoding control parameters, the hardware encoder is invoked to perform intra-frame compression and inter-frame predictive coding on the original screen data stream to obtain the encoded basic stream data. Based on the keyframe insertion strategy in the encoding control parameters, keyframes are periodically inserted into the basic stream data, and random access points are marked. Based on the estimated network jitter, the maximum transmission unit size of the current encapsulated data packet is determined, and based on the maximum transmission unit size, the basic stream data with the random access point is encapsulated into a network transmission packet, and the network transmission packets are combined to obtain the screen projection data stream.

[0014] Furthermore, to achieve the above objectives, this application also proposes a screen projection control device based on the WIFI-Display protocol, which includes: The session establishment module is used to negotiate capabilities with the receiving device and establish a WIFI-Display connection session based on the session establishment request initiated by the source device. The intelligent capture module is used to extract the dynamically updated region between adjacent frames based on the monitoring of changes in the graphics buffer and the difference tile detection algorithm, and obtain the original screen data stream based on the dynamically updated region; The parameter decision module is used to determine the motion complexity of the image in the original screen data stream and generate encoding control parameters based on the real-time feedback network state parameters and the motion complexity. The encoding and sending module is used to perform real-time encoding and network adaptive encapsulation on the original screen data stream based on the encoding control parameters to obtain the projection data stream, and send the projection data stream to the receiving device through the established WIFI-Display connection session. The network adaptive encapsulation dynamically adjusts the encapsulation granularity of the data packets according to the estimated network jitter.

[0015] In addition, to achieve the above objectives, this application also proposes a screen projection control device based on the WIFI-Display protocol. The screen projection control device based on the WIFI-Display protocol includes: a memory, a processor, and a computer program stored in the memory and capable of running on the processor. The computer program is configured to implement the steps of the screen projection control method based on the WIFI-Display protocol as described above.

[0016] In addition, to achieve the above objectives, the present invention also proposes a storage medium, which is a computer-readable storage medium, and stores a computer program on the storage medium. When the computer program is executed by a processor, it implements the steps of the screen projection control method based on the WIFI-Display protocol described above.

[0017] In addition, to achieve the above objectives, this application also provides a computer program product, which includes a computer program. When the computer program is executed by a processor, it implements the steps of the screen projection control method based on the WIFI-Display protocol described above.

[0018] This application provides a screen projection control method based on the WIFI-Display protocol. First, a WIFI-Display connection session is established between the source device and the receiving end. Second, by monitoring the graphics buffer and using a difference tile detection algorithm, dynamically updated areas of the screen are intelligently captured, generating only the original screen data stream for changed areas to improve efficiency. Then, by analyzing image motion complexity and real-time network state parameters, encoding control parameters are dynamically generated. Finally, based on these parameters, the data stream is encoded in real-time and adaptively encapsulated via network, forming a projection data stream which is then sent to the receiving end. Through these methods, this approach effectively reduces bandwidth consumption and encoding computation, and significantly improves the smoothness and image quality stability of dynamic screen content transmission in complex network environments through adaptive bitrate and encapsulation strategies. Attached Figure Description

[0019] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0020] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0021] Figure 1 This is a flowchart illustrating Embodiment 1 of the screen projection control method based on the WIFI-Display protocol of this application; Figure 2 This is a schematic diagram of a binary mask representing an embodiment of the screen projection control method based on the WIFI-Display protocol in this application. Figure 3 This is a schematic diagram of the module structure of the screen projection control device based on the WIFI-Display protocol in an embodiment of this application; Figure 4 This is a schematic diagram of the device structure of the hardware operating environment involved in the screen projection control method based on the WIFI-Display protocol in the embodiments of this application.

[0022] The realization of the purpose, functional features and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0023] It should be understood that the specific embodiments described herein are merely illustrative of the technical solutions of this application and are not intended to limit this application.

[0024] To better understand the technical solution of this application, a detailed description will be provided below in conjunction with the accompanying drawings and specific implementation methods.

[0025] The main solution of this application embodiment is: based on the session establishment request initiated by the source device, to negotiate capabilities with the receiving device and establish a WIFI-Display connection session; Based on monitoring changes in the graphics buffer, a difference tile detection algorithm is used to extract the dynamically updated region between adjacent frames, and the original screen data stream is obtained based on the dynamically updated region. Determine the motion complexity of the image in the original screen data stream, and generate encoding control parameters based on the real-time feedback network state parameters and the motion complexity; Based on the encoding control parameters, the original screen data stream is encoded in real time and encapsulated using network adaptive methods to obtain a projection data stream. The projection data stream is then sent to the receiving device through an established WIFI-Display connection session. The network adaptive encapsulation dynamically adjusts the encapsulation granularity of the data packets according to the estimated network jitter.

[0026] Currently, existing screen mirroring solutions based on the Wi-Fi-Display protocol typically employ a fixed frame rate and encoding parameters to continuously capture and encode the entire screen image for transmission. This method has significant limitations when facing dynamic scenes or complex network fluctuations: firstly, continuous full-screen encoding generates a large amount of redundant data, especially when only a small portion of the image is updated, resulting in a huge waste of valuable wireless bandwidth resources and high bandwidth consumption; secondly, a fixed encoding strategy cannot adaptively adjust to the motion complexity of the image content and real-time network conditions. When bandwidth is limited or network jitter is severe, it easily leads to increased data transmission latency and packet loss rate, resulting in noticeable video stuttering and image quality degradation at the receiving end, severely impacting the user experience.

[0027] This application provides a solution that first establishes a Wi-Fi-Display connection session between the source device and the receiving end; second, by monitoring the graphics buffer and using a difference tile detection algorithm, it intelligently captures dynamically updating areas of the screen, generating only the original screen data stream for the changed areas to improve efficiency; then, by analyzing the image motion complexity and real-time network state parameters, it dynamically generates encoding control parameters; finally, based on these parameters, it performs real-time encoding and adaptive network encapsulation on the data stream to form a projection data stream and sends it to the receiving end. Through these methods, this approach effectively reduces bandwidth consumption and encoding computation, and significantly improves the smoothness of transmission and image quality stability of dynamic screen content in complex network environments through adaptive bitrate and encapsulation strategies.

[0028] It should be noted that the executing entity in this embodiment can be a computing service device with data processing, network communication, and program execution functions, such as a tablet computer, personal computer, or mobile phone, or an electronic device capable of performing the above functions, such as a screen projection control device based on the WIFI-Display protocol. This embodiment does not specifically limit it in this regard. The following uses a screen projection control device based on the WIFI-Display protocol as an example to describe this embodiment and the following embodiments.

[0029] All actions involving the acquisition of signals, information, or data in this application are carried out in accordance with the relevant data protection laws and policies of the country where the application is located, and with the authorization of the owner of the relevant device.

[0030] This application provides a screen mirroring control method based on the WIFI-Display protocol, referring to... Figure 1 , Figure 1 This is a flowchart illustrating the first embodiment of the screen projection control method based on the WIFI-Display protocol in this application.

[0031] In this embodiment, the screen projection control method based on the WIFI-Display protocol includes steps S10 to S40: Step S10: Based on the session establishment request initiated by the source device, perform capability negotiation with the receiving device and establish a WIFI-Display connection session.

[0032] It should be noted that, within the WIFI-Display protocol framework, the source device refers to the initiating end that carries the content to be projected, such as a mobile phone, tablet, or computer. The receiving end device is the terminal that displays the content, such as a smart TV or projector.

[0033] It should be understood that capability negotiation is a key step in connection establishment, and is essentially an interactive handshake process based on the Real-Time Streaming Protocol (RTSP).

[0034] In practice, the source device first discovers the receiver via Wi-Fi P2P or within a shared wireless LAN, and then initiates an RTSP session establishment request. Both parties negotiate key parameters, such as M3, M4, and M5 messages, by exchanging a series of standard RTSP signaling messages. These messages include, but are not limited to, video encoding formats (such as H.264 / AVC or HEVC), supported resolutions and frame rate profiles, audio encoding formats, and display control capabilities. This ensures that both ends agree on a common set of media processing capabilities, establishing a stable audio and video transmission connection session.

[0035] Step S20: Based on monitoring changes in the graphics buffer, extract the dynamically updated region between adjacent frames using a difference tile detection algorithm, and obtain the original screen data stream based on the dynamically updated region.

[0036] It's important to note that the graphics buffer refers to the memory area used by the operating system to synthesize and store the final frame image of the current screen. The difference tile detection algorithm is an image processing technique that divides the screen into several regular tiles and identifies regions where content has changed by comparing the pixel data or checksums of corresponding tiles in the buffer between consecutive frames. The dynamically updated region is the set of all tiles that have undergone pixel-level changes, detected by the difference tile detection algorithm. The raw screen data stream refers to the sequence of unencoded RGB or YUV format screen frames captured from the graphics buffer.

[0037] Understandably, the system listens for commit or refresh events in the graphics buffer in real time by hooking into or polling the underlying graphics compositor interface of the operating system. Once a new frame is detected, a difference tile detection process is triggered. First, the buffer data of the current frame and the previous frame are divided into grids of the same size, such as 16x16 pixel blocks. Then, an efficient block-by-block comparison algorithm quickly locates the tiles whose content has changed and marks these tiles as dynamically updated areas. Finally, the original pixel data corresponding to the tiles marked as dynamically updated areas is extracted and encapsulated to form a stream of raw screen data to be encoded, thereby reducing the amount of data that needs to be processed.

[0038] In one feasible implementation, the step of extracting dynamically updated regions between adjacent frames using a difference tile detection algorithm based on monitoring changes in the graphics buffer includes: Monitor changes in the graphics buffer to determine the current frame and the reference frame; The current frame and the reference frame are divided into multiple tiles of the same size by a difference tile detection algorithm, and the pixel difference between the tiles is calculated. Based on a preset difference threshold, the patches whose pixel difference exceeds the preset difference threshold are marked as candidate dynamic patches; The candidate dynamic map pieces are subjected to morphological dilation to connect adjacent small variation regions, generating dynamically updated regions.

[0039] It's important to note that a reference frame is an image from the previous frame or a baseline frame used for comparison with the most recently captured current frame. Pixel difference is a metric used to quantify the degree of content difference between two tiles, typically calculated by the sum of absolute errors (SAD) or mean squared errors of corresponding pixel values. The preset difference threshold is a configurable threshold value used to determine whether tile differences are significant enough to be considered a change; its purpose is to filter out minor changes caused by noise or slight rendering jitter. Candidate dynamic tiles are tiles whose pixel difference exceeds the preset threshold.

[0040] In the implementation, updates to the graphics buffer are continuously monitored through the operating system's underlying interface. When a new frame is detected, it is used as the current frame, and the previously cached frame is used as the reference frame. The core of difference detection is to divide both frames into N x M fixed-size patches (e.g., 32x32 pixels), and then calculate the pixel difference for each corresponding patch. A commonly used method for calculating pixel difference is the sum of absolute errors, with the formula:

[0041] in, and These represent a tile in the current frame and a reference frame, respectively. and This represents the width and height of the tile. The larger the SAD value, the greater the difference between the two tiles.

[0042] Next, the system compares the SAD value of each patch with a preset difference threshold. Compare. If If the value is not specified, the tile is marked as a candidate dynamic tile, forming an initial binary mask image, where white (value 1) represents a candidate tile and black (value 0) represents an unchanged tile. (Refer to...) Figure 2 , Figure 2This is a schematic diagram of a binary mask image. The image shows the screen divided into 32×32 pixel tiles. White tiles represent candidate dynamic tiles (changing regions) detected by the SAD algorithm, while black tiles represent unchanged regions. Since minute changes or noise in screen rendering can cause changing regions to be detected as multiple discrete small tiles, directly encoding these scattered regions is inefficient. Therefore, morphological dilation processing of the mask image is required. The dilation operation uses a structuring element (e.g., a 3x3 all-one matrix kernel) to convolve the mask. Its effect is to expand the boundaries of white regions and fill the small gaps between them, thereby merging adjacent candidate dynamic tiles into a more complete and connected dynamically updated region.

[0043] In one feasible implementation, the step of performing morphological dilation on the candidate dynamic map tiles to connect adjacent regions of minor variation and generate the final dynamically updated region includes: Based on the preset convolution kernel size, the candidate dynamic image blocks are dilated to obtain preliminary connected candidate regions; The candidate regions that are initially connected are marked, and adjacent minor variation regions are connected according to the marks to obtain several independent connected regions; Based on the minimum region area threshold, connected components with areas smaller than the minimum region area threshold are filtered to obtain the filtered connected components. A dynamically updated region is generated based on the filtered connected components.

[0044] It's important to note that the kernel size defines the shape and size of the structuring element used in the morphological dilation operation. It's typically a small matrix (e.g., 3×3) with its center point as the anchor point, used to determine the scanning range of the pixel's neighborhood. The dilation operation scans the binary image using the structuring element. When at least one foreground pixel (with a value of 1) exists in the overlapping area between the structuring element and the image, the pixel at the anchor point is set as foreground, thus expanding the foreground region, filling holes, and connecting breakpoints. The candidate region for initial connection refers to the set of partially connected dynamic patches obtained after the first dilation operation. A connected component is the largest connected region composed of adjacent pixels (using 4-connectivity or 8-connectivity rules). The minimum region area threshold is used to filter out connected components that are too small to be meaningful due to noise or minor interference.

[0045] In the specific implementation, a preset convolution kernel K is used to perform sliding convolution on the binary mask image I. Let K be a matrix of size m×n, with its anchor point located at the center. The dilation operation can be formally defined as:

[0046] This means that the pixel value at (x, y) in the output image is the sum of all pixel values ​​in the region of the input image I that overlaps with K. This operation causes the boundaries of the candidate dynamic patches to expand outward, thereby connecting those small adjacent regions that were originally separated due to grid division or detection errors, forming initially connected candidate regions.

[0047] Subsequently, a double-pass scanning method is performed on the dilated binary image to assign a unique label to each connected pixel set, thereby identifying several independent connected regions. Next, the area of ​​each connected region, i.e., the total number of pixels it contains, is calculated and compared with a preset minimum region area threshold. Compare. For any connected component. If its area If the area is not specified, it is considered noise or an irrelevant update and is removed from the candidate regions. Ultimately, all regions with an area greater than or equal to... The set of connected components constitutes the dynamically updated region.

[0048] In one feasible implementation, the step of obtaining the original screen data stream based on the dynamically updated region includes: Based on the coordinate information of the dynamically updated region, the corresponding region pixel data is extracted from the graphics buffer; Add frame header information to the pixel data of the region to identify its position in the full screen, and generate incremental data frames; During non-first transmission cycles, the incremental data frames are treated as the original screen data stream.

[0049] It's important to note that the graphics buffer is a dedicated memory area used to store the currently rendered screen content, typically maintained by the graphics processing unit (GPU) or display compositor. Region pixel data refers to rectangular pixel blocks extracted from the graphics buffer based on the bounding box coordinates of dynamically updated regions. Frame header information is metadata appended to the pixel data, containing key information such as the incremental frame's position in the full screen coordinate system (e.g., top-left x, y coordinates), dimensions (width, height), and timestamp. An incremental data frame is a concise data unit containing only the pixel data of the changed region and its frame header. The raw screen data stream is the final output sequence of screen data, suitable for encoding or network transmission. The transmission cycle refers to the cyclic interval between screen data acquisition and transmission.

[0050] In the specific implementation, the specific position (x, y) and size (width, height) of the dynamically updated region in the screen coordinate system are first calculated based on the bounding rectangle of the region. Then, the raw RGB or RGBA pixel data within this rectangular region is read directly from the graphics buffer via memory mapping or a graphics API. Next, the system constructs the frame header information, whose data structure can be defined as a binary packet header containing fields such as magic number, version, frame type (incremental frame), coordinates, size, and timestamp. The final composition of the incremental data frame is: frame header information + region pixel data.

[0051] During data transmission, a state machine can be maintained to manage the data transfer. In the first transmission cycle, or when the system detects a failed reference frame, it switches to full transmission mode. This is for example, when a large-area change exceeding a threshold is detected by comparing the hash value of the entire screen, or when a resolution change event is received. In this mode, the entire screen frame is captured, and a data frame labeled as a full frame is generated as the raw screen data stream. Under normal conditions, outside the first cycle and when the reference frame is valid, the system continuously generates and outputs incremental data frames. This hybrid transmission strategy ensures data integrity during connection initialization or drastic changes, while significantly reducing bandwidth consumption in most cases through incremental updates.

[0052] In one feasible implementation, the step of adding frame header information to the region pixel data to identify its position in the full screen and generating an incremental data frame includes: A location descriptor is generated based on the coordinates of the top-left corner vertex of the dynamically updated region and the region's width and height information. Generate an inter-frame dependency identifier based on the sequence number of the current frame and the sequence number of the reference frame; The location descriptor and the inter-frame dependency identifier are combined to form frame header information that identifies the location in the full screen; The frame header information is appended before the region pixel data to generate an incremental data frame.

[0053] It should be noted that the position descriptor is a data block encoded from coordinates and size information. The sequence number is a monotonically increasing unique identifier assigned to each captured screen frame, used to establish the temporal relationship between frames. The inter-frame dependency identifier is the sequence number of the reference frame that the current incremental frame depends on.

[0054] In the specific implementation, the spatial information of the incremental frame first needs to be encapsulated. The position descriptor P can be formally represented as a tuple: P = (x, y, width, height). Here, (x, y) represents the pixel coordinates of the top-left corner of the bounding rectangle of the dynamically updated region in the full screen coordinate system, and width and height represent the width and height of the rectangle, respectively. These integer values ​​are usually converted into a fixed-length byte sequence (e.g., each value is represented by a 4-byte unsigned integer) and arranged sequentially to form the position descriptor data segment.

[0055] Next, the system needs to establish the temporal dependencies of the incremental frames. Assume the current screen frame sequence number is... The sequence number of the reference frame used as the comparison benchmark when calculating the dynamic update region of the current frame is... Therefore, the core content of the inter-frame dependency identifier D is... Its data structure and Together, they form a key part of the frame header used for control logic. A simplified frame header structure can be defined as: ,in This field is used to distinguish between full frames and incremental frames.

[0056] Ultimately, a complete incremental data frame is composed of a frame header and region pixel data: IncrementalFrame = Header + PixelData. At the receiving end, the decoder first parses the frame header to verify the continuity of the sequence number. It does this through D (i.e., The system locates the reference frame in the local cache, and then draws the received PixelData onto the specified area of ​​the reference frame according to the location descriptor P, thereby synthesizing the sequence number. The new complete frame is used as the reference frame for the next compositing operation.

[0057] Step S30: Determine the motion complexity of the image in the original screen data stream, and generate encoding control parameters based on the real-time feedback network state parameters and the motion complexity.

[0058] It should be noted that motion complexity is a quantitative indicator that measures the degree of drastic change in the temporal and spatial domains of image content in the raw screen data stream. It reflects the amplitude, range, and frequency of pixel value changes between consecutive frames and is usually related to features such as macroblock motion vector amplitude, inter-frame residuals, or image gradients. Real-time feedback network status parameters refer to a set of indicators dynamically acquired from the transmission channel monitoring module that reflect the current network conditions. These mainly include available bandwidth, round-trip time, packet loss rate, and network jitter. Encoding control parameters are a series of internal settings of the video encoder that are dynamically adjusted based on the above inputs. Their core objective is to balance encoding efficiency, video quality, and real-time performance. Key parameters include, but are not limited to, target bitrate, frame rate, quantization parameters, GOP structure, motion search range, and entropy coding mode.

[0059] Understandably, motion complexity is first quantified by analyzing the differences between consecutive frames. Simultaneously, network state parameters, particularly estimates of available bandwidth, are continuously acquired through network feedback. The controller then uses these two core inputs as the basis for decision-making: when motion complexity is high or network bandwidth is limited, the bitrate or frame rate is reduced by increasing the quantization parameters to ensure the quality of keyframes; conversely, the bitrate or frame rate is increased, provided the network allows, to improve the visual experience.

[0060] In one feasible implementation, the step of determining the motion complexity of the image in the original screen data stream includes: Optical flow calculation is performed on multiple consecutive frames of images in the original screen data stream, and the average motion amplitude between frames is calculated based on the motion vector field of the pixels. Based on image gradient feature detection technology, high-frequency texture regions and key interest regions in the multi-frame images are identified, and the area ratio is calculated based on the high-frequency texture regions and the key interest regions. The motion complexity is obtained based on the average motion amplitude and the area ratio.

[0061] It's important to note that the motion vector field is the collection of motion vectors from all pixels, describing the dense motion patterns of the entire image from one frame to the next. The average motion amplitude is the statistical average of the magnitudes of all pixel motion vectors in the motion vector field, quantifying the intensity of global motion between frames. Image gradient features reflect the intensity and direction of brightness changes at each pixel, typically calculated using convolution operators such as Sobel and Scharr. Their magnitude is used to identify edges and textures. High-frequency texture regions refer to local areas in the image with high gradient amplitudes and drastic pixel value changes (such as hair or textile textures). Regions of interest are areas that are visually sensitive or task-critical, determined through visual saliency models or user interaction hotspot analysis (such as mouse pointers, input cursors, and active window title bars). Area percentage refers to the percentage of the total area of ​​the aforementioned high-frequency texture regions or regions of interest within the overall effective area of ​​the image.

[0062] In the specific implementation, the optical flow algorithm is first applied to consecutive frames. For each pixel (i, j), its motion vector is calculated. ,in and These represent the velocity components in the horizontal and vertical directions, respectively. The motion vector field of the entire frame is thus formed by... The set of average motion amplitude. It is obtained by calculating the arithmetic mean of the motion vector magnitudes of all pixels, using the following formula:

[0063] in This represents the total number of pixels within the frame. The larger the value, the more intense the overall motion between frames.

[0064] Simultaneously, the system performs image gradient calculations on the current frame. This is typically done using the Sobel operator (convolution kernel). and Calculate the gradient components in the horizontal and vertical directions respectively, and then obtain the gradient magnitude of each pixel:

[0065] By setting an empirical threshold ,Will The pixels are classified as high-frequency texture pixels. The area percentage of the high-frequency texture region... This is the ratio of the total number of these pixels to the total number of pixels in the image. Similarly, focus on the area ratio of specific regions. The pixel coverage ratio can be determined by a predefined ROI detection algorithm (such as face detection, window focus tracking, or mouse position heatmap).

[0066] Ultimately, motion complexity It is a weighted combination of the average motion amplitude and the proportion of the two areas, and can be expressed as: .in, , , These are weighting coefficients that are optimized based on experience or scenarios.

[0067] In one feasible implementation, the step of performing optical flow calculation on multiple consecutive frames of images in the original screen data stream and calculating the average motion amplitude between frames based on the motion vector field of the pixels includes: Optical flow calculation is performed on multiple consecutive frames of images in the original screen data stream to determine the motion vector field of each pixel in the current frame and the previous frame; The intra-frame motion amplitude distribution map is obtained based on the motion vector field of all pixels; The effective motion amplitude is obtained by thresholding the motion amplitude value of each pixel in the intra-frame motion amplitude distribution map. The effective motion amplitude is arithmetically averaged to obtain the average motion amplitude between frames.

[0068] It should be noted that the intra-frame motion amplitude distribution map is a scalar map formed by converting the motion vector of each pixel in the motion vector field into its Euclidean magnitude, which intuitively reflects the local motion intensity of each region of the image. The effective motion amplitude is the result of filtering the motion amplitude distribution map using a threshold segmentation algorithm.

[0069] In the specific implementation, a dense optical flow algorithm is first used to process two consecutive grayscale images. and This algorithm assumes that pixels within a small neighborhood move in unison and solves the optical flow equation based on polynomial expansion, calculating a motion vector for each pixel (x, y). ,in and These are the horizontal and vertical velocity components, respectively. The motion vectors of all pixels constitute a dense motion vector field. .

[0070] Subsequently, the system traverses the motion vector field. For each pixel in the graph, calculate the magnitude (i.e., amplitude) of its motion vector. All Arranged according to the original pixel positions, this generates an intra-frame motion amplitude distribution map. This distribution map visually indicates the dynamic range of each location in the image.

[0071] To eliminate noise interference and focus on effective motion, the system analyzes the motion amplitude distribution map. Threshold segmentation is performed. A key step is determining an adaptive threshold. This can be based on Otsu's method (maximum between-class variance method). The value is calculated from the histogram or set to an empirically fixed value (e.g., 5% of the maximum total amplitude range). The segmentation process is defined as:

[0072]

[0073] Finally, for all non-zero values... Calculate the arithmetic mean, which gives the average motion amplitude between frames:

[0074] Where N is satisfied The total number of pixels.

[0075] In one feasible implementation, the step of generating coding control parameters based on real-time feedback network state parameters and the motion complexity includes: Based on the packet loss rate, latency jitter, and available bandwidth periodically fed back from the receiver, a current network status assessment vector is constructed. The motion complexity and the situation evaluation vector are input into a pre-trained reinforcement learning model to obtain the target bit rate, target frame rate, and keyframe insertion strategy for the next time window. Encoding control parameters are generated based on the target bitrate, the target frame rate, and the keyframe insertion strategy.

[0076] In the implementation, the receiving end periodically reports the network status to the sending end via RTCP messages or a custom feedback protocol. After collecting these metrics, the sending end first performs preprocessing: mapping Packet Loss Rate (PLR) and Jitter to negative evaluation scores for network congestion and stability, while using Available Bandwidth as a positive capacity indicator. Since these metrics have different dimensions, they need to be normalized and finally concatenated into a fixed-length status assessment vector. This vector and the motion complexity scalar calculated at the same moment. Together they constitute the complete state input of the reinforcement learning model. .

[0077] The system employs a pre-trained reinforcement learning agent as the decision-making core. This model is learned during the training phase using a large amount of simulated or real network trajectory data. Its goal is to maximize a reward function that comprehensively considers video quality (such as PSNR or SSIM), quality fluctuations, end-to-end latency, and stuttering duration. During the inference phase (i.e., in practical applications), the model uses the current state... The input is fed into its policy network, and the output is an action. This action is typically a discrete or continuous space, corresponding to the target bitrate, target frame rate, and keyframe insertion strategy for the next time window.

[0078] Finally, the encoding controller will output the action from the model. Decoding is then performed on specific encoding control parameters. The target bitrate and target frame rate are directly passed to the encoder's rate control module. The keyframe insertion strategy is then translated into the encoder's forced frame refresh command or the setting of the GOP size (N value).

[0079] Step S40: Based on the encoding control parameters, the original screen data stream is encoded in real time and encapsulated using network adaptive methods to obtain a projection data stream. The projection data stream is then sent to the receiving device through an established WIFI-Display connection session. The network adaptive encapsulation dynamically adjusts the encapsulation granularity of the data packets according to the estimated network jitter.

[0080] Understandably, the encoder module receives encoding control parameters from the previous step and compresses the captured raw screen frames in real time accordingly. Subsequently, the encapsulation module monitors historical network latency data, uses methods such as sliding window or Kalman filtering to estimate the current network jitter level, and dynamically determines the payload size of each data packet based on preset mapping rules (e.g., reducing encapsulation granularity when jitter is high, dividing a frame of data into more and smaller RTP packets to reduce the impact of single packet loss and transmission burstiness; increasing encapsulation granularity when jitter is low, aggregating multiple frame slices or more NAL units into one RTP packet to reduce protocol header overhead). Finally, the encapsulated projection data stream is sent to the receiving end through the established WIFI-Display session signaling and AV data transmission channel using the UDP / RTP protocol stack.

[0081] In one feasible implementation, the step of performing real-time encoding and network adaptive encapsulation on the original screen data stream based on the encoding control parameters to obtain the projection data stream includes: Based on the target bitrate and target frame rate in the encoding control parameters, the hardware encoder is invoked to perform intra-frame compression and inter-frame predictive coding on the original screen data stream to obtain the encoded basic stream data. Based on the keyframe insertion strategy in the encoding control parameters, keyframes are periodically inserted into the basic stream data, and random access points are marked. Based on the estimated network jitter, the maximum transmission unit size of the current encapsulated data packet is determined, and based on the maximum transmission unit size, the basic stream data with the random access point is encapsulated into a network transmission packet, and the network transmission packets are combined to obtain the screen projection data stream.

[0082] In its implementation, the system first calls the API interface of the underlying hardware encoder. The encoding process strictly follows the encoding control parameters: the target bitrate is set as the encoder's rate control target, guiding the selection of its quantization parameter (QP) to achieve optimal rate-distortion performance within a limited bandwidth; the target frame rate determines the rhythm at which the encoder skips or captures raw frames. The core of the encoding algorithm is a hybrid coding framework, performing intra-frame compression (I-frames, using spatial redundancy elimination) or inter-frame predictive coding (P-frames / B-frames, utilizing temporal redundancy elimination) on each frame. This process outputs basic streaming data conforming to the H.264 / AVC or H.265 / HEVC standards, i.e., a series of network abstraction layer units.

[0083] Subsequently, the system dynamically manages the GOP structure according to a keyframe insertion strategy. This strategy can specify a fixed GOP length (e.g., inserting a keyframe every 30 frames) or an adaptive strategy based on scene transition detection. When a keyframe needs to be inserted, the encoder generates an I-frame, which serves as a random access point. This means the decoder can correctly decode from this point without relying on previous frames. In the streaming data, these I-frames are specially marked for encapsulation and receiver identification.

[0084] Finally, network adaptive encapsulation is performed. The system continuously monitors network status and predicts network jitter by calculating the standard deviation of the round-trip time (RTT) or similar metrics over a recent period. Based on the jitter magnitude, a heuristic algorithm is applied to dynamically adjust the maximum transmission unit (MTU): when the network is stable and jitter is low, a larger MTU is allowed, aggregating multiple NAL units into a single RTP packet to reduce header overhead; when network jitter is high, a smaller MTU is used, splitting the data into smaller RTP packets to reduce the impact of single packet loss and burstiness of transmission, thereby improving transmission robustness. The encapsulated RTP packets are sorted by sequence number and combined into the final projection data stream, ready to be sent via Socket.

[0085] This embodiment provides a screen projection control method based on the WIFI-Display protocol. First, a WIFI-Display connection session is established between the source device and the receiving end. Second, by monitoring the graphics buffer and using a difference tile detection algorithm, dynamically updated areas of the screen are intelligently captured, generating only the original screen data stream for changed areas to improve efficiency. Then, by analyzing image motion complexity and real-time network state parameters, encoding control parameters are dynamically generated. Finally, based on these parameters, the data stream is encoded in real-time and adaptively encapsulated via network, forming the projection data stream, which is then sent to the receiving end. Through this approach, the method effectively reduces bandwidth consumption and encoding computation, and significantly improves the smoothness of transmission and image quality stability of dynamic screen content in complex network environments through adaptive bitrate and encapsulation strategies.

[0086] It should be noted that the above examples are only for understanding this application and do not constitute a limitation on the screen projection control method based on the WIFI-Display protocol. Any simple modifications based on this technical concept are within the protection scope of this application.

[0087] This application also provides a screen projection control device based on the WIFI-Display protocol. Please refer to [link / reference]. Figure 3 The screen projection control device based on the WIFI-Display protocol includes: Session establishment module 10 is used to negotiate capabilities with the receiving device and establish a WIFI-Display connection session based on the session establishment request initiated by the source device. The intelligent capture module 20 is used to extract the dynamically updated region between adjacent frames based on the monitoring of changes in the graphics buffer and the difference tile detection algorithm, and to obtain the original screen data stream based on the dynamically updated region; Parameter decision module 30 is used to determine the motion complexity of the image in the original screen data stream and generate encoding control parameters based on the real-time feedback network state parameters and the motion complexity. The encoding and sending module 40 is used to perform real-time encoding and network adaptive encapsulation on the original screen data stream based on the encoding control parameters to obtain the projection data stream, and send the projection data stream to the receiving device through the established WIFI-Display connection session. The network adaptive encapsulation dynamically adjusts the encapsulation granularity of the data packets according to the estimated network jitter.

[0088] In one feasible implementation, the intelligent capture module 20 is also used to monitor changes in the graphics buffer and determine the current frame and the reference frame; The current frame and the reference frame are divided into multiple tiles of the same size by a difference tile detection algorithm, and the pixel difference between the tiles is calculated. Based on a preset difference threshold, the patches whose pixel difference exceeds the preset difference threshold are marked as candidate dynamic patches; The candidate dynamic map pieces are subjected to morphological dilation to connect adjacent small variation regions, generating dynamically updated regions.

[0089] In one feasible implementation, the intelligent capture module 20 is further configured to perform a dilation operation on the candidate dynamic image block according to a preset convolution kernel size to obtain a preliminary connected candidate region; The candidate regions that are initially connected are marked, and adjacent minor variation regions are connected according to the marks to obtain several independent connected regions; Based on the minimum region area threshold, connected components with areas smaller than the minimum region area threshold are filtered to obtain the filtered connected components. A dynamically updated region is generated based on the filtered connected components.

[0090] In one feasible implementation, the intelligent capture module 20 is further configured to extract corresponding region pixel data from the graphics buffer based on the coordinate information of the dynamically updated region; Add frame header information to the pixel data of the region to identify its position in the full screen, and generate incremental data frames; During non-first transmission cycles, the incremental data frames are treated as the original screen data stream.

[0091] In one feasible implementation, the intelligent capture module 20 is further configured to generate a location descriptor based on the coordinates of the upper left corner vertex of the dynamically updated region and the region width and height information; Generate an inter-frame dependency identifier based on the sequence number of the current frame and the sequence number of the reference frame; The location descriptor and the inter-frame dependency identifier are combined to form frame header information that identifies the location in the full screen; The frame header information is appended before the region pixel data to generate an incremental data frame.

[0092] In one feasible implementation, the parameter decision module 30 is further used to perform optical flow calculation on multiple consecutive frames of images in the original screen data stream, and calculate the average motion amplitude between frames based on the motion vector field of the pixel. Based on image gradient feature detection technology, high-frequency texture regions and key interest regions in the multi-frame images are identified, and the area ratio is calculated based on the high-frequency texture regions and the key interest regions. The motion complexity is obtained based on the average motion amplitude and the area ratio.

[0093] In one feasible implementation, the parameter decision module 30 is further used to perform optical flow calculation on multiple consecutive frames of images in the original screen data stream to determine the motion vector field of each pixel in the current frame and the previous frame image. The intra-frame motion amplitude distribution map is obtained based on the motion vector field of all pixels; The effective motion amplitude is obtained by thresholding the motion amplitude value of each pixel in the intra-frame motion amplitude distribution map. The effective motion amplitude is arithmetically averaged to obtain the average motion amplitude between frames.

[0094] In one feasible implementation, the parameter decision module 30 is further configured to construct a current network status assessment vector based on the packet loss rate, latency jitter, and available bandwidth periodically fed back by the receiver. The motion complexity and the situation evaluation vector are input into a pre-trained reinforcement learning model to obtain the target bit rate, target frame rate, and keyframe insertion strategy for the next time window. Encoding control parameters are generated based on the target bitrate, the target frame rate, and the keyframe insertion strategy.

[0095] In one feasible implementation, the encoding and sending module 40 is further configured to call a hardware encoder to perform intra-frame compression and inter-frame predictive coding on the original screen data stream according to the target bit rate and target frame rate in the encoding control parameters, so as to obtain encoded basic stream data. Based on the keyframe insertion strategy in the encoding control parameters, keyframes are periodically inserted into the basic stream data, and random access points are marked. Based on the estimated network jitter, the maximum transmission unit size of the current encapsulated data packet is determined, and based on the maximum transmission unit size, the basic stream data with the random access point is encapsulated into a network transmission packet, and the network transmission packets are combined to obtain the screen projection data stream.

[0096] The screen projection control device based on the WIFI-Display protocol provided in this application, employing the screen projection control method based on the WIFI-Display protocol in the above embodiments, can solve the technical problems of screen projection lag and high bandwidth consumption. Compared with the prior art, the beneficial effects of the screen projection control device based on the WIFI-Display protocol provided in this application are the same as those of the screen projection control method based on the WIFI-Display protocol provided in the above embodiments, and other technical features in the screen projection control device based on the WIFI-Display protocol are the same as those disclosed in the methods of the above embodiments, and will not be repeated here.

[0097] This application provides a screen projection control device based on the WIFI-Display protocol. The screen projection control device based on the WIFI-Display protocol includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the screen projection control method based on the WIFI-Display protocol in the above embodiment 1.

[0098] The following is for reference. Figure 4 This document illustrates a structural diagram suitable for implementing a screen projection control device based on the WIFI-Display protocol in the embodiments of this application. The screen projection control device based on the WIFI-Display protocol in the embodiments of this application can include, but is not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Description), PMPs (Portable Media Players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 4 The screen projection control device shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.

[0099] like Figure 4As shown, a screen projection control device based on the WIFI-Display protocol may include a processing unit 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to programs stored in ROM (Read Only Memory) 1002 or programs loaded from storage device 1003 into RAM (Random Access Memory) 1004. RAM 1004 also stores various programs and data required for operating the screen projection control device based on the WIFI-Display protocol. The processing unit 1001, ROM 1002, and RAM 1004 are interconnected via bus 1005. Input / output (I / O) interface 1006 is also connected to the bus. Typically, the following systems can be connected to I / O interface 1006: input devices 1007 including, for example, touchscreens, touchpads, keyboards, mice, image sensors, microphones, accelerometers, gyroscopes, etc.; output devices 1008 including, for example, LCDs (Liquid Crystal Displays), speakers, vibrators, etc.; storage devices 1003 including, for example, magnetic tapes, hard disks, etc.; and communication devices 1009. Communication device 1009 allows the screen projection control device based on the WIFI-Display protocol to wirelessly or wiredly communicate with other devices to exchange data. Although the figure shows a screen projection control device based on the WIFI-Display protocol with various systems, it should be understood that it is not required to implement or have all the systems shown. More or fewer systems can be implemented alternatively.

[0100] Specifically, according to the embodiments disclosed in this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from storage device 1003, or installed from ROM 1002. When the computer program is executed by processing device 1001, it performs the functions defined in the methods of the embodiments disclosed in this application.

[0101] The screen projection control device based on the WIFI-Display protocol provided in this application, employing the screen projection control method based on the WIFI-Display protocol in the above embodiments, can solve the technical problem of screen projection control based on the WIFI-Display protocol. Compared with the prior art, the beneficial effects of the screen projection control device based on the WIFI-Display protocol provided in this application are the same as the beneficial effects of the screen projection control method based on the WIFI-Display protocol provided in the above embodiments, and other technical features in this screen projection control device based on the WIFI-Display protocol are the same as those disclosed in the method of the previous embodiment, and will not be repeated here.

[0102] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples.

[0103] The above are merely specific embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0104] This application provides a computer-readable storage medium having computer-readable program instructions (i.e., a computer program) stored thereon, which are used to execute the screen projection control method based on the WIFI-Display protocol in the above embodiments.

[0105] The computer-readable storage medium provided in this application may be, for example, a USB flash drive, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, RAM (Random Access Memory), ROM (Read Only Memory), EPROM (Erasable Programmable Read Only Memory or Flash Memory), optical fibers, CD-ROM (CD-Read Only Memory), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, system, or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination thereof.

[0106] The aforementioned computer-readable storage medium may be included in a screen projection control device based on the WIFI-Display protocol; or it may exist independently and not be assembled into a screen projection control device based on the WIFI-Display protocol.

[0107] The aforementioned computer-readable storage medium carries one or more programs. When the aforementioned one or more programs are executed by the screen projection control device based on the WIFI-Display protocol, the screen projection control device based on the WIFI-Display protocol: according to the session establishment request initiated by the source device, negotiates capabilities with the receiving device and establishes a WIFI-Display connection session. Based on monitoring changes in the graphics buffer, a difference tile detection algorithm is used to extract the dynamically updated region between adjacent frames, and the original screen data stream is obtained based on the dynamically updated region. Determine the motion complexity of the image in the original screen data stream, and generate encoding control parameters based on the real-time feedback network state parameters and the motion complexity; Based on the encoding control parameters, the original screen data stream is encoded in real time and encapsulated using network adaptive methods to obtain a projection data stream. The projection data stream is then sent to the receiving device through an established WIFI-Display connection session. The network adaptive encapsulation dynamically adjusts the encapsulation granularity of the data packets according to the estimated network jitter.

[0108] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including LAN (Local Area Network) or WAN (Wide Area Network)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0109] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0110] The modules described in the embodiments of this application can be implemented in software or hardware. The names of the modules do not necessarily limit the functionality of the unit itself.

[0111] The readable storage medium provided in this application is a computer-readable storage medium that stores computer-readable program instructions (i.e., a computer program) for executing the above-described screen projection control method based on the WIFI-Display protocol, thereby solving the technical problem of screen projection control based on the WIFI-Display protocol. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as those of the screen projection control method based on the WIFI-Display protocol provided in the above embodiments, and will not be repeated here.

[0112] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the screen projection control method based on the WIFI-Display protocol as described above.

[0113] The computer program product provided in this application can solve the technical problem of screen projection control based on the WIFI-Display protocol. Compared with the prior art, the beneficial effects of the computer program product provided in this application are the same as those of the screen projection control method based on the WIFI-Display protocol provided in the above embodiments, and will not be repeated here.

[0114] The above are only some embodiments of this application and do not limit the patent scope of this application. All equivalent structural transformations made under the technical concept of this application and using the contents of the specification and drawings of this application, or direct / indirect applications in other related technical fields, are included in the patent protection scope of this application.

Claims

1. A method for implementing screen projection control based on a WIFI-Display protocol, characterized in that, The screen projection control method based on the WIFI-Display protocol comprises the following steps: According to the session establishment request initiated by the source device, the capability negotiation with the receiving end device is performed and the WIFI-Display connection session is established; Based on the monitoring of the change of the graphic buffer, the dynamic update area between adjacent frames is extracted by using a difference block detection algorithm, and the original screen data stream is obtained according to the dynamic update area; The motion complexity of the image in the original screen data stream is determined, and the encoding control parameter is generated according to the real-time feedback network state parameter and the motion complexity; Based on the encoding control parameter, the original screen data stream is real-time encoded and network adaptive packaged to obtain the screen projection data stream, and the screen projection data stream is sent to the receiving end device through the established WIFI-Display connection session, wherein the network adaptive packaging dynamically adjusts the packaging granularity of the data packet according to the estimated network jitter.

2. The method of claim 1, wherein, The step of extracting the dynamic update area between adjacent frames by using a difference block detection algorithm based on the monitoring of the change of the graphic buffer comprises the following steps: The change of the graphic buffer is monitored to determine the current frame and the reference frame; The current frame and the reference frame are divided into a plurality of blocks of the same size by using a difference block detection algorithm, and the pixel difference degree between the blocks is calculated; Based on a preset difference degree threshold, the blocks with the pixel difference degree exceeding the preset difference degree threshold are marked as candidate dynamic blocks; The candidate dynamic blocks are morphologically dilated to connect adjacent small change areas to generate a dynamic update area.

3. The method of claim 2, wherein, The step of morphologically dilating the candidate dynamic blocks to connect adjacent small change areas to generate a final dynamic update area comprises the following steps: According to a preset convolution kernel size, the candidate dynamic blocks are dilated to obtain a preliminary connected candidate area; The preliminary connected candidate area is marked, and adjacent small change areas are connected according to the mark to obtain a plurality of independent connected domains; According to a minimum area threshold, the connected domains with an area less than the minimum area threshold are filtered to obtain filtered connected domains; The dynamic update area is generated according to the filtered connected domains.

4. The method of claim 1, wherein, The step of obtaining the original screen data stream according to the dynamic update area comprises the following steps: Based on the coordinate information of the dynamic update area, the corresponding area pixel data is intercepted from the graphic buffer; The area pixel data is added with frame header information for identifying the position in the complete screen to generate an incremental data frame; In a non-first transmission period, the incremental data frame is taken as the original screen data stream.

5. The method of claim 4, wherein, The step of adding the area pixel data with the frame header information for identifying the position in the complete screen to generate an incremental data frame comprises the following steps: According to the top-left vertex coordinate and the area width and height information of the dynamic update area, a position descriptor is generated; Based on the sequence number of the current frame and the sequence number of the reference frame, a frame interdependence identifier is generated; The position descriptor and the frame interdependence identifier are combined into the frame header information for identifying the position in the complete screen. The frame header information is attached before the region pixel data to generate an incremental data frame.

6. The method of claim 1, wherein, The step of determining the motion complexity of the images in the original screen data stream comprises: Optical flow calculation is performed on the continuous multiple frames of images in the original screen data stream, and the average motion amplitude between frames is calculated according to the motion vector field of the pixel points. Based on the image gradient feature detection technology, the high-frequency texture region and the key attention region in the multiple frames of images are identified, and the area ratio is calculated based on the high-frequency texture region and the key attention region. The motion complexity is obtained according to the average motion amplitude and the area ratio.

7. The method of claim 6, wherein, The step of performing optical flow calculation on the continuous multiple frames of images in the original screen data stream and calculating the average motion amplitude between frames according to the motion vector field of the pixel points comprises: Optical flow calculation is performed on the continuous multiple frames of images in the original screen data stream to determine the motion vector field of each pixel point in the current frame and the previous frame of images. The motion amplitude distribution map within the frame is obtained according to the motion vector field of all pixel points. The motion amplitude value of each pixel point in the motion amplitude distribution map within the frame is threshold segmented to obtain the effective motion amplitude. The effective motion amplitude is arithmetically averaged to obtain the average motion amplitude between frames.

8. The method of claim 1, wherein, The step of generating the encoding control parameter according to the real-time feedback network state parameter and the motion complexity comprises: According to the packet loss rate, delay jitter and available bandwidth periodically fed back by the receiving end, a condition evaluation vector of the current network is constructed. The motion complexity and the condition evaluation vector are input into a pre-trained reinforcement learning model to obtain the target code rate, the target frame rate and the key frame insertion strategy of the next time window. The encoding control parameter is generated according to the target code rate, the target frame rate and the key frame insertion strategy.

9. The method of claim 1, wherein, Based on the encoding control parameter, the original screen data stream is real-time encoded and network adaptively packaged to obtain the screen projection data stream, which comprises: According to the target code rate and the target frame rate in the encoding control parameter, a hardware encoder is called to perform intra-frame compression and inter-frame prediction encoding on the original screen data stream to obtain the encoded basic stream data. Based on the key frame insertion strategy in the encoding control parameter, a key frame is periodically inserted in the basic stream data, and a random access point is marked. According to the estimated network jitter, the maximum transmission unit size of the current packaged data packet is determined, and based on the maximum transmission unit size, the basic stream data with the random access point is packaged into a network transmission packet, and the network transmission packet is combined to obtain the screen projection data stream.

10. A screen projection control device based on a WIFI-Display protocol, characterized in that, The screen projection control device based on the WIFI-Display protocol comprises: A session establishment module is configured to perform capability negotiation with a receiving end device and establish a WIFI-Display connection session according to a session establishment request initiated by a source device. An intelligent capture module is configured to extract a dynamic update region between adjacent frames by a difference block detection algorithm based on monitoring of changes in a graphics buffer, and obtain an original screen data stream according to the dynamic update region. a parameter decision module, configured to determine motion complexity of images in the original screen data stream, and generate an encoding control parameter according to real-time feedback of network state parameters and the motion complexity; an encoding and sending module, configured to perform real-time encoding and network adaptive encapsulation on the original screen data stream based on the encoding control parameter, to obtain a screen projection data stream, and send the screen projection data stream to a receiving end device through an established WIFI-Display connection session, wherein the network adaptive encapsulation dynamically adjusts encapsulation granularity of data packets according to estimated network jitter.

Images