Wireless screen transmission code rate control method and device, wireless screen transmission system and sending end equipment

Abstract

The application is suitable for the field of wireless screen transmission technology, and provides a wireless screen transmission code rate control method and device, a wireless screen transmission system and a sending end equipment, which comprises: acquiring system state data corresponding to the t time in the wireless screen transmission process; wherein the system state data comprises characteristic data of a first communication level and characteristic data of a second communication level between two devices for wireless screen transmission; the first communication level comprises a data link layer and a physical layer, and the second communication level is an upper communication level of the first communication level; performing decision on the system state data corresponding to the t time based on a pre-trained decision model to obtain a control parameter corresponding to the t time; and adjusting the code rate at the t time in the wireless screen transmission process according to the control parameter corresponding to the t time. Through the above method, the wireless screen transmission delay can be effectively reduced, and the continuity of the screen transmission picture can be ensured.

Description

Technical Field

[0001] This application belongs to the field of wireless screen sharing technology, and in particular relates to a wireless screen sharing bitrate control method, device, wireless screen sharing system and transmitting end equipment. Background Technology

[0002] Wireless screen sharing technology refers to the technology of transmitting images or videos from one terminal device to the display screen of another terminal device via wireless communication. For example, a common application scenario is transmitting photos or videos from a mobile phone to a projector, television, or large monitor screen. Currently, in a good wireless network environment, wireless screen sharing latency is low, ensuring the continuity of the transmitted image. However, in complex wireless network environments, wireless screen sharing latency is usually high, easily causing stuttering and other issues that negatively impact the user experience. Summary of the Invention

[0003] This application provides a wireless screen sharing bitrate control method, device, system, and transmitting device, which can effectively reduce wireless screen sharing latency and help ensure the continuity of the screen sharing image.

[0004] In a first aspect, embodiments of this application provide a wireless screen transmission bitrate control method, including: Acquire system status data at time t during the wireless screen sharing process; wherein, the system status data includes feature data of the first communication layer and feature data of the second communication layer between the two devices performing wireless screen sharing; the first communication layer includes the data link layer and the physical layer, and the second communication layer is the upper communication layer of the first communication layer; The system state data at time t is used to make decisions based on the pre-trained decision model to obtain the control parameters at time t. Adjust the bitrate at time t during the wireless screen sharing process according to the control parameters corresponding to time t.

[0005] In this embodiment, when deciding on the control parameters for the wireless screen sharing bitrate, data from multiple communication layers are considered. These multiple communication layers include the data link layer, the physical layer, and upper communication layers. Since the indicators of the data link layer and the physical layer can directly reflect the state of the wireless network hardware, the decision model can promptly perceive underlying hardware problems, thereby improving the timeliness of decision-making. Furthermore, relying solely on data from the lower communication layers may overlook video buffer overload issues, while relying solely on data from the upper communication layers may fail to distinguish whether stuttering is caused by channel interference or an excessively high bitrate. Combining data from different communication layers allows for a more accurate reflection of the wireless network's state, thus improving decision accuracy. Through this approach, the bitrate of wireless screen sharing can be controlled promptly and accurately, effectively reducing wireless screen sharing latency and ensuring the continuity of the shared video.

[0006] In one possible implementation of the first aspect, the pre-trained decision model makes a decision on the system state data at time t to obtain the control parameters at time t, including: Obtain a pre-trained decision model; The system state data at time t is input into the decision model, and the control parameters at time t are output.

[0007] In the above implementation method, the pre-trained decision model does not need to run the training process in actual application. The trained decision model can be directly applied, which can reduce the decision time and ensure the decision accuracy.

[0008] In one possible implementation of the first aspect, the decision model includes a first network and a second network; wherein the first network is used to extract feature information from the input data; and the second network is used to make a decision based on the feature information output by the first network. After adjusting the bitrate at time t during wireless screen sharing according to the control parameters corresponding to time t, the method further includes: Calculate the reward value corresponding to time t; wherein, the reward value corresponding to time t is used to characterize the system latency after adjusting the bitrate at time t; Obtain the system status data at time t+1 during the wireless screen sharing process; Generate the training sample corresponding to time t; wherein, the training sample corresponding to time t includes the system state data corresponding to time t, the control parameters corresponding to time t, the reward value corresponding to time t, and the system state data corresponding to time t+1; The network parameters of the second network are adjusted according to the training samples corresponding to time t to obtain the adjusted second network. The adjusted decision model is determined based on the first network and the adjusted second network.

[0009] In the above implementation, the first network in the decision model used to extract feature information is fixed. Training samples are generated based on data from the actual control process. The second network in the decision model used for decision-making is then fine-tuned based on the training samples. This allows the decision model to gradually "adapt" to the current decision-making environment, thereby improving decision accuracy. Furthermore, this method combines offline training with online fine-tuning. Typically, the model complexity of the first network is higher than that of the second network. This approach improves the adaptive capability of the decision model while reducing the computational overhead of fine-tuning the model on the edge.

[0010] In one possible implementation of the first aspect, calculating the reward value corresponding to time t includes: Calculate the first score corresponding to the time t; wherein the first score corresponding to the time t is used to characterize the wireless screen sharing image quality corresponding to the time t. Calculate the second score corresponding to time t; wherein the second score corresponding to time t is used to characterize the system delay at time t. The reward value corresponding to time t is calculated based on the first score and the second score.

[0011] In the above method, two core indicators, screen transmission quality and system latency, are quantified separately to obtain dual scores, which are then combined to generate a reward value. This allows the fine-tuning of the decision-making model to simultaneously take into account the two optimization goals of image clarity and low transmission latency, achieving a balanced and optimal control of image quality and latency.

[0012] In one possible implementation of the first aspect, calculating the second score corresponding to time t includes: Obtain the system delay corresponding to time t; If the system delay is less than or equal to a preset threshold, the reward value at time t is calculated based on the first penalty weight. If the system delay is greater than a preset threshold, the reward value at time t is calculated according to the second penalty weight; wherein the first penalty weight is less than the second penalty weight.

[0013] In the above method, by distinguishing whether the system latency exceeds the threshold and setting two levels of penalty weights, a stronger penalty is applied to high-latency scenarios, guiding the model to prioritize controlling screen transmission latency, avoid severe stuttering, and accurately constrain the upper limit of transmission latency.

[0014] In one possible implementation of the first aspect, adjusting the bitrate at time t during the wireless screen transmission process according to the control parameters corresponding to time t includes: Map the control parameters corresponding to time t to adjustment coefficients; Calculate the target bitrate at time t based on the adjustment coefficient and the bitrate at time t-1. The bitrate at time t during the wireless screen sharing process is adjusted to the target bitrate corresponding to time t.

[0015] In the above implementation, the control parameters output by the decision model are converted into adjustment coefficients, and the target bitrate at the current moment is calculated by combining the bitrate of the previous moment. This achieves a smooth and gradual adjustment of the bitrate, avoiding drastic fluctuations in image clarity and instantaneous network congestion caused by sudden increases or decreases in bitrate.

[0016] In one possible implementation of the first aspect, the method further includes: If the target bitrate at time t is lower than the preset bitrate, then the frame rate and resolution at time t are reduced.

[0017] In the above method, when the target bitrate is too low, the frame rate and resolution are simultaneously reduced, thereby reducing the amount of data transmitted and alleviating lag in weak network conditions. In addition, the coordinated adjustment of the target bitrate, frame rate, and resolution enables the system to achieve multi-level adjustments, reducing the bitrate for slight network congestion and reducing the bitrate, frame rate, and resolution for severe congestion.

[0018] In one possible implementation of the first aspect, the characteristic data of the first communication layer includes signal fluctuation gradient, MAC layer retransmission rate, and physical layer negotiation rate; wherein the signal fluctuation gradient is used to characterize the attenuation trend of the wireless signal; the MAC layer retransmission rate is used to characterize the probability of retransmission at the MAC layer; and the physical layer negotiation rate is used to characterize the theoretical rate of wireless signal negotiation at the physical layer. The characteristic data of the second communication layer includes round-trip time, one-way delay gradient, and encoder backlog depth; wherein, the round-trip time is used to characterize the data round-trip time of the transport layer; the one-way delay gradient is used to characterize the data backlog of the wireless screen sharing network queue; and the encoder backlog depth is used to characterize the congestion level of the application layer's sending queue.

[0019] The above approach collects multi-dimensional and accurate quantitative indicators from the physical layer / link layer and the upper transport layer / application layer to fully depict the real transmission status of the entire wireless channel, transmission queue, and coding buffer, providing the model with comprehensive and reliable input for accurate code rate decision-making.

[0020] Secondly, embodiments of this application provide a wireless screen transmission bitrate control device, comprising: The acquisition unit is used to acquire system status data at time t during the wireless screen sharing process; wherein, the system status data includes feature data of the first communication layer and feature data of the second communication layer between the two devices performing wireless screen sharing; the first communication layer includes the data link layer and the physical layer, and the second communication layer is the upper communication layer of the first communication layer; The decision unit is used to make decisions based on the system state data at time t based on the pre-trained decision model, and obtain the control parameters at time t. The control unit is used to adjust the bit rate at time t during the wireless screen transmission process according to the control parameters corresponding to time t.

[0021] Thirdly, embodiments of this application provide a wireless screen sharing system, including a transmitting device and a receiving device; the transmitting device and the receiving device are connected wirelessly. The transmitting device is configured to determine the wireless screen transmission code rate according to the method described in any one of the first aspects, and to transmit data to the receiving end based on the wireless screen transmission code rate.

[0022] Fourthly, embodiments of this application provide a transmitting device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the wireless screen transmission rate control method as described in any one of the first aspects above.

[0023] Fifthly, embodiments of this application provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements the wireless screen transmission rate control method as described in any one of the first aspects above.

[0024] In a sixth aspect, embodiments of this application provide a computer program product that, when run on a terminal device, causes the terminal device to execute the wireless screen transmission rate control method described in any one of the first aspects.

[0025] It is understood that the beneficial effects of the second to sixth aspects mentioned above can be found in the relevant descriptions in the first aspect mentioned above, and will not be repeated here. Attached Figure Description

[0026] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0027] Figure 1 This is a schematic diagram of an application scenario for wireless screen sharing provided in an embodiment of this application; Figure 2 This is a schematic diagram of an application scenario for wireless screen sharing provided in another embodiment of this application; Figure 3 This is a flowchart illustrating the wireless screen transmission bitrate control method provided in an embodiment of this application; Figure 4 This is a schematic diagram of the fine-tuning process of the decision model provided in the embodiments of this application; Figure 5 This is a schematic diagram of the wireless screen sharing system provided in an embodiment of this application; Figure 6 This is a schematic diagram of the structure of the wireless screen transmission bitrate device provided in the embodiments of this application; Figure 7 This is a schematic diagram of the structure of the transmitting device provided in an embodiment of this application. Detailed Implementation

[0028] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this application with unnecessary detail.

[0029] It should be understood that, when used in this application specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or a collection thereof.

[0030] It should also be understood that the term “and / or” as used in this application specification and the appended claims means any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0031] As used in this application specification and the appended claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection." Similarly, the phrase "if determined" or "if detected [the described condition or event]" may be interpreted, depending on the context, as meaning "once determined," "in response to determination," "once detected [the described condition or event]," or "in response to detection [the described condition or event]."

[0032] Furthermore, in the description of this application and the appended claims, the terms "first," "second," "third," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0033] References to "one embodiment" or "some embodiments" in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized.

[0034] Wireless screen sharing technology refers to the technology of transmitting images or videos from one terminal device to the display screen of another terminal device through wireless communication.

[0035] In some application scenarios, see Figure 1 This is a schematic diagram illustrating an application scenario for wireless screen sharing provided in an embodiment of this application. For example... Figure 1 As shown, terminal device A transmits photos or videos to terminal device B via wireless screen sharing technology. Terminal device A and terminal device B are connected wirelessly. It is understandable that... Figure 1 In the application scenario shown, both terminal device A and terminal device B have wireless communication capabilities.

[0036] In other application scenarios, see Figure 2 This is a schematic diagram illustrating an application scenario for wireless screen sharing provided in another embodiment of this application. For example... Figure 2 As shown, terminal device A and terminal device B wirelessly transmit data via a screen sharing device. In this scenario, terminal device A transmits data such as photos or videos to the screen sharing device, which then transmits the data to terminal device B.

[0037] Among them, screen sharing devices can also be called screen projectors, screen mirroring devices, or wireless screen sharing equipment. Screen sharing devices have wireless communication capabilities and can replace high-definition multimedia interface (HDMI) cables to achieve wireless screen sharing between two terminal devices.

[0038] like Figure 2In the application scenarios shown, in some cases, terminal device A may not have wireless communication capabilities. In this case, the screen sharing device can be physically connected to terminal device A to achieve wireless screen sharing between terminal device A and terminal device B. In other cases, terminal device B also lacks wireless communication capabilities. In this case, the screen sharing device can be physically connected to terminal device B to achieve wireless screen sharing between terminal device A and terminal device B. In still other cases, neither terminal device A nor terminal device B has wireless communication capabilities. In this case, two screen sharing devices can be used: one physically connected to terminal device A and the other physically connected to terminal device B, achieving wireless screen sharing between terminal device A and terminal device B through the two screen sharing devices.

[0039] Optionally, the screen sharing device connects to the terminal device via a Universal Serial Bus (USB) interface. In this case, the terminal device is responsible for encoding. For example, terminal device A is physically connected to the screen sharing device via a USB interface. Terminal device A encodes the data to be transmitted, obtains a data packet, and sends the data packet to terminal device B through the screen sharing device. In this example, the screen sharing device only forwards data.

[0040] Optionally, the screen sharing device connects to the terminal device via a Type-C interface. In this case, the screen sharing device is responsible for encoding. For example, terminal device A is physically connected to the screen sharing device via a Type-C interface. Terminal device A sends the data to be transmitted to the screen sharing device, which encodes the data to be transmitted, obtains a data packet, and sends the data packet to terminal device B. In this example, the screen sharing device is responsible for data encoding and data forwarding.

[0041] In the above application scenarios, the terminal device can be a mobile phone, laptop, desktop computer, tablet, smart screen, wearable device, projector, smart TV, large monitor, etc. For example, terminal device A is a mobile phone, and terminal device B is a smart TV, meaning the mobile phone can wirelessly project photos, videos, documents, and other data onto the smart TV. As another example, terminal device A is a laptop, and terminal device B is a large monitor or projector, meaning the laptop can wirelessly project photos, videos, documents, and other data onto the large monitor or projector. This application does not specifically limit the application scenarios for wireless screen sharing.

[0042] Currently, in good wireless network environments, wireless screen sharing latency is low, ensuring the continuity of the shared screen. However, in complex wireless network environments, wireless screen sharing latency is usually high, which can easily cause stuttering and other issues, affecting the user experience.

[0043] Based on this, this application provides a method for controlling the bitrate of wireless screen sharing. In this application, when deciding on the control parameters for the wireless screen sharing bitrate, data from multiple communication layers are considered. These multiple communication layers include the data link layer, the physical layer, and upper communication layers. Since the indicators of the data link layer and the physical layer can directly reflect the state of the wireless network hardware, the decision model can promptly perceive underlying hardware problems, thereby improving the timeliness of the decision. Furthermore, relying solely on data from the lower communication layers may overlook video buffer overload issues, and relying solely on data from the upper communication layers may fail to distinguish whether stuttering is caused by channel interference or an excessively high bitrate. Combining data from different communication layers allows for a more accurate reflection of the wireless network's state, thereby improving decision accuracy. Through this method, the bitrate of wireless screen sharing can be controlled in a timely and accurate manner, effectively reducing wireless screen sharing latency and ensuring the continuity of the shared screen image.

[0044] See Figure 3 This is a flowchart illustrating the wireless screen transmission rate control method provided in this application embodiment. It is intended as an example and not a limitation. The method may include steps S301-S303, which are described in detail below.

[0045] S301, acquire the system status data at time t during the wireless screen sharing process.

[0046] The system status data includes feature data of the first communication layer and feature data of the second communication layer between the two devices performing wireless screen sharing; the first communication layer includes the data link layer and the physical layer, and the second communication layer is the upper communication layer of the first communication layer.

[0047] In some application scenarios, the communication layers between two terminal devices in wireless screen sharing can adopt a network architecture of L1-L7. Among them, L1 is the physical layer, L2 is the data link layer (MAC / Wi-Fi underlying layer), L3 is the network layer (IP), L4 is the transport layer (TCP / UDP), L5 is the session layer, L6 is the presentation layer, and L7 is the application layer (business program).

[0048] In some implementations, the first communication layer includes L1 and L2 layers, and the second communication layer includes L4 and L7 layers.

[0049] In some implementations, the characteristic data of the first communication layer includes signal fluctuation gradient, MAC layer retransmission rate, and physical layer negotiation rate. Among them, the signal fluctuation gradient is used to characterize the attenuation trend of the wireless signal; the MAC layer retransmission rate is used to characterize the probability of retransmission at the MAC layer; and the physical layer negotiation rate is used to characterize the theoretical rate of wireless signal negotiation at the physical layer.

[0050] Optionally, the signal fluctuation gradient can be the rate of change of signal fluctuation within a preset time window before time t. The signal fluctuation gradient can more sensitively capture the signal attenuation trend caused by device movement or obstruction.

[0051] Optionally, the MAC layer retransmission rate can be the number of retransmitted frames within a preset time window before time t or the total number of transmitted frames. It is understandable that if the MAC layer retransmission rate exceeds a preset threshold, even if no packet loss has occurred at the application layer, this indicates increased channel contention or co-channel interference. For example, the MAC layer retransmission rate can be obtained through system kernel hooks or driver interfaces.

[0052] Optionally, the physical layer negotiation rate can be the highest theoretical rate negotiated by the current wireless network.

[0053] In some implementations, the characteristic data of the second communication layer includes round-trip time, one-way delay gradient, and encoder backlog depth. Round-trip time characterizes the data round-trip delay of the transport layer; one-way delay gradient characterizes the data backlog in the wireless screen sharing network queue; and encoder backlog depth characterizes the congestion level of the application layer's sending queue.

[0054] Optionally, the round-trip time delay can be data after smoothing the round-trip time delay.

[0055] Optionally, the encoder backlog depth can be the ratio of the total size of unsent video frames in the encoding queue to the current target transmission rate.

[0056] In some implementations of related technologies, decisions are made based on characteristic data from L4 and L7 layers (such as packet loss rate and / or receiver feedback delay). In wireless LANs, covert terminal issues or microwave interference at wireless access nodes can instantly trigger a large number of retransmissions at the MAC layer. By the time the application layer detects packet loss, the underlying buffer has already become severely congested, causing screen sharing to experience stuttering for hundreds of milliseconds or even seconds.

[0057] Compared to the implementation methods in related technologies, the embodiments of this application collect multi-dimensional and accurate quantitative indicators of the physical layer / link layer and the upper transport layer / application layer to fully depict the real transmission status of the entire link of wireless channel, transmission queue, and coding buffer, providing the model with comprehensive and reliable input for accurate decision-making on the code rate.

[0058] In some implementations, the characteristic data of the first communication layer is obtained through the system kernel interface using a high-frequency polling method. For example, in macOS, the system kernel interface can be CoreWLAN / NetworkExtension. In Windows, the system kernel interface can be the WMI / NDIS driver layer interface.

[0059] In some implementations, the individual feature data in the system feature data can be normalized.

[0060] In some implementations, the system state data at time t may include the system state data at time t itself, as well as the system state data at multiple times within a preset time window before time t.

[0061] S302, Based on the pre-trained decision model, make a decision on the system state data at time t to obtain the control parameters at time t.

[0062] Specifically, S302 includes: acquiring a pre-trained decision model; inputting the system state data at time t into the decision model; and outputting the control parameters at time t.

[0063] Optionally, the decision-making model can be a large model or an agent.

[0064] Understandably, the pre-training process of the decision model can be an offline process. Alternatively, the pre-training process can be performed by another terminal, and the trained decision model can then be deployed on the application terminal. For example, the decision model can be pre-trained in the cloud, and then deployed on the terminal device used to implement the wireless screen transmission rate control method.

[0065] In the above implementation method, the pre-trained decision model does not need to run the training process in actual application. The trained decision model can be directly applied, which can reduce the decision time and ensure the decision accuracy.

[0066] S303, adjusts the bit rate at time t during wireless screen transmission according to the control parameters corresponding to time t.

[0067] In one embodiment, S303 includes: Map the control parameters corresponding to time t to adjustment coefficients; Calculate the target bitrate at time t based on the adjustment coefficient and the bitrate at time t-1. Adjust the bitrate at time t during the wireless screen sharing process to the target bitrate corresponding to time t.

[0068] In one implementation, the control parameters at time t are nonlinearly mapped to obtain the adjustment coefficient. Alternatively, the adjustment coefficient can be obtained according to the formula... Perform nonlinear mapping. For adjustment coefficients, These are the control parameters corresponding to time t. This is a single adjustment step size.

[0069] Optional, according to the formula Calculate the target bitrate at time t. Let be the target bitrate at time t. Let be the bitrate at time t-1.

[0070] In the above implementation, the control parameters output by the decision model are converted into adjustment coefficients, and the target bitrate at the current moment is calculated by combining the bitrate of the previous moment. This achieves a smooth and gradual adjustment of the bitrate, avoiding drastic fluctuations in image clarity and instantaneous network congestion caused by sudden increases or decreases in bitrate.

[0071] In some cases, when the network deteriorates drastically (such as when TargetBitrate_t falls below the minimum quality threshold Bitrate_min, such as 1Mbps), simply reducing the bitrate is no longer sufficient to solve the latency problem.

[0072] To address the above problems, in one embodiment, the method further includes: If the target bitrate at time t is lower than the preset bitrate, then reduce the frame rate and resolution at time t.

[0073] Optionally, the reduction ratio of frame rate and resolution can be selected based on the difference between the target bitrate and the preset bitrate at time t. Specifically, if the difference between the target bitrate and the preset bitrate at time t is less than a first threshold, the frame rate and resolution at time t are reduced according to a first adjustment ratio; if the difference between the target bitrate and the preset bitrate at time t is greater than the first threshold, the frame rate and resolution at time t are reduced according to a second adjustment ratio. The first ratio is less than the second ratio.

[0074] It is understandable that frame rate and resolution can use different thresholds and different adjustment ratios.

[0075] In the above method, when the target bitrate is too low, the frame rate and resolution are simultaneously reduced, thereby reducing the amount of data transmitted and alleviating lag in weak network conditions. In addition, the coordinated adjustment of the target bitrate, frame rate, and resolution enables the system to achieve multi-level adjustments, reducing the bitrate for slight network congestion and reducing the bitrate, frame rate, and resolution for severe congestion.

[0076] Figure 3In the illustrated embodiment, when determining the control parameters for the wireless screen sharing bitrate, data from multiple communication layers are considered. These layers include the data link layer, the physical layer, and upper communication layers. Since the metrics of the data link layer and the physical layer directly reflect the state of the wireless network hardware, the decision-making model can promptly detect underlying hardware problems, thereby improving the timeliness of the decision. Furthermore, relying solely on data from the lower communication layers may overlook video buffer overload issues, while relying solely on data from the upper communication layers may fail to distinguish whether stuttering is caused by channel interference or an excessively high bitrate. Combining data from different communication layers provides a more accurate reflection of the wireless network's state, thus improving decision accuracy. Through this approach, the bitrate of wireless screen sharing can be controlled promptly and accurately, effectively reducing wireless screen sharing latency and ensuring the continuity of the shared video.

[0077] Understandably, the pre-trained decision model can be a general-purpose model, and in practical applications, it can be deployed on different types of terminals. However, because the handling of buffer length and queuing strategies by various terminals is a "black box" phenomenon, the decision model may exhibit fitting bias on the terminal side.

[0078] To improve the generalization ability of the decision-making model and thus enhance decision-making accuracy, in one embodiment, the decision-making model can be fine-tuned at the edge.

[0079] In this embodiment of the application, the decision model includes a first network and a second network; wherein, the first network is used to extract feature information from the input data; and the second network is used to make decisions based on the feature information output by the first network.

[0080] See Figure 4 This is a schematic diagram illustrating the fine-tuning process of the decision-making model provided in the embodiments of this application. It is intended as an example and not a limitation. Figure 4 As shown, after S303, the method further includes steps S401-S405, which are described in detail below.

[0081] S401, calculate the reward value at time t.

[0082] The reward value at time t is used to characterize the system latency after adjusting the bitrate at time t.

[0083] In one implementation, S401 includes: Calculate the first score corresponding to time t; where the first score corresponding to time t is used to characterize the wireless screen sharing image quality at time t. Calculate the second score corresponding to time t; where the second score corresponding to time t is used to characterize the system delay at time t. The reward value at time t is calculated based on the first and second scores.

[0084] In the above method, two core indicators, screen transmission quality and system latency, are quantified separately to obtain dual scores, which are then combined to generate a reward value. This allows the fine-tuning of the decision-making model to simultaneously take into account the two optimization goals of image clarity and low transmission latency, achieving a balanced and optimal control of image quality and latency.

[0085] Optionally, a linear function can be used to calculate the first score.

[0086] Due to the principle of diminishing marginal returns in screen sharing scenarios, the perceived improvement in image quality is more significant when the bitrate increases from 1Mbps to 2Mbps, but the difference is smaller when increasing from 10Mbps to 11Mbps. Based on this consideration, this embodiment uses a nonlinear function to calculate the first score.

[0087] Optionally, the calculation process for the first score includes: according to the formula Calculate the first score. Wherein, The minimum baseline bitrate set for the system to ensure that the basic visuals do not crash, for example, Set to 1Mbps. Let be the target bitrate actually output by the system at time t.

[0088] Optionally, a piecewise nonlinear penalty can be used to calculate the second score.

[0089] In one implementation, the process of calculating the second score includes: Obtain the system delay at time t; If the system delay is less than or equal to the preset threshold, the reward value at time t is calculated based on the first penalty weight. If the system delay exceeds a preset threshold, the reward value at time t is calculated based on the second penalty weight; wherein the first penalty weight is less than the second penalty weight.

[0090] The preset threshold can be set as the operation delay between the two terminal devices in wireless screen sharing. This operation delay represents the time it takes for the mouse to move from the sending end to the receiving end and for the corresponding frame to be rendered. For example, the preset threshold can be set to 90ms. For example, if the system delay is less than or equal to the preset threshold, then according to the formula... Calculate the reward value at time t; where, As the first penalty weight, The system delay at time t is... Let be the reward value (or penalty value) at time t. If the system delay exceeds a preset threshold, then according to the formula... Calculate the reward value at time t; where, As the second penalty weight, This is a preset threshold.

[0091] Understandably, the first penalty weight is less than the second penalty weight. If the system latency is less than or equal to a preset threshold, a smaller penalty weight is used. In this case, the penalty for latency is smaller, and to pursue a higher reward value, the bitrate will tend to be increased to increase image quality gains. This means that the bitrate can be moderately increased to improve image quality within a controllable latency range. However, if the system latency exceeds a preset threshold, the penalty will increase exponentially, thereby increasing the severity of the penalty. Through this stringent negative feedback, the decision-making model can immediately choose a "cliff-like" bitrate reduction strategy when it receives a congestion warning from the L1 / L2 layer, even if no packet loss has occurred yet, to ensure the "responsiveness" of the interaction without sacrificing image quality.

[0092] Compared with the fixed weight method, the above method sets two levels of penalty weights, large and small, to distinguish whether the system latency exceeds the threshold. This applies stronger penalties to high-latency scenarios, guides the model to prioritize controlling screen transmission latency, avoids severe stuttering, and accurately constrains the upper limit of transmission latency.

[0093] In one implementation, the reward value at time t is calculated based on the first score, the second score, and the packet loss rate.

[0094] For example, the reward value is calculated according to the formula: Calculate the reward value. Among them, Let be the first score at time t, and be the second score at time t. Let be the packet loss rate at time t. , These are preset weights.

[0095] In the above implementation method, the current screen transmission status is comprehensively quantified by integrating three indicators: weighted image quality gain, latency penalty, and packet loss rate penalty. This guides the model to simultaneously optimize image clarity, transmission latency, and packet loss issues, thereby achieving multi-objective balanced control of the wireless screen transmission bitrate.

[0096] S402, acquire the system status data at time t+1 during the wireless screen sharing process.

[0097] This step is implemented in the same way as S301, and can be found in the description in the S301 embodiment. It will not be repeated here.

[0098] S403 generates the training sample corresponding to time t.

[0099] The training samples at time t include the system state data at time t, the control parameters at time t, the reward value at time t, and the system state data at time t+1.

[0100] S404, adjust the network parameters of the second network according to the training samples corresponding to time t to obtain the adjusted second network.

[0101] Of course, optionally, the network parameters of the second network can be adjusted based on the training samples corresponding to time t and the training samples corresponding to multiple times within a preset time window before time t.

[0102] Understandably, a larger number of training samples leads to higher tuning accuracy for the second network, but also increases computational overhead. The settings can be adjusted according to actual needs.

[0103] S405, Determine the adjusted decision model based on the first network and the adjusted second network.

[0104] In the embodiments described in S401-S405, the first network used for extracting feature information in the decision-making model is fixed. Training samples are generated based on data from the actual control process. The second network used for decision-making in the decision-making model is then fine-tuned based on the training samples. This allows the decision-making model to gradually "adapt" to the current decision-making environment, thereby improving decision accuracy. Furthermore, the above method is equivalent to combining offline training with online fine-tuning. Typically, the model complexity of the first network is higher than that of the second network. This method improves the adaptive capability of the decision-making model while reducing the computational overhead of fine-tuning the model on the edge.

[0105] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0106] See Figure 5 This is a schematic diagram of the wireless screen sharing system provided in an embodiment of this application. Figure 5 As shown, the wireless screen sharing system may include a transmitting device 51 and a receiving device 52. The transmitting device 51 and the receiving device 52 are connected wirelessly.

[0107] The transmitting device 51 is used to determine the wireless transmission rate according to the steps of the above-described embodiments of the wireless transmission rate control method, and to send data to the receiving device 52 based on the determined wireless transmission rate.

[0108] In some examples, such as Figure 1In the application scenario shown, the sending device 51 is terminal device A, and the receiving device 52 is terminal device B.

[0109] In other examples, such as Figure 2 In the application scenario shown, if terminal device A and the screen sharing device are connected via a USB interface, in this case, terminal device A is responsible for encoding, that is, terminal device A is the frame rate controller. In this case, the sending device 51 is terminal device A, and the receiving device 52 is terminal device B.

[0110] In some other examples, such as Figure 2 In the application scenario shown, if terminal device A and screen sharing device are connected via a Type-C interface, the screen sharing device is responsible for encoding, that is, the screen sharing device is the frame rate controller. In this case, the sending device 51 is the screen sharing device, and the receiving device 52 is the terminal device B.

[0111] In some other examples, such as Figure 2 In the application scenario shown, if terminal device A is connected to a screen sharer via a Type-C interface, and terminal device B is connected to another screen sharer via a Type-C interface, in this case, the screen sharer physically connected to terminal device A is responsible for encoding, that is, the screen sharer physically connected to terminal device A is the frame rate controller. In this case, the sending end device 51 is the screen sharer physically connected to terminal device A, and the receiving end device 52 is the screen sharer physically connected to terminal device B.

[0112] Corresponding to the wireless screen transmission bitrate control method described in the above embodiments, Figure 6 This is a structural block diagram of the wireless screen transmission rate control device provided in the embodiments of this application. For ease of explanation, only the parts related to the embodiments of this application are shown.

[0113] Reference Figure 6 The device 6 includes: The acquisition unit 61 is used to acquire system status data at time t during the wireless screen sharing process; wherein, the system status data includes feature data of the first communication layer and feature data of the second communication layer between the two devices performing wireless screen sharing; the first communication layer includes the data link layer and the physical layer, and the second communication layer is the upper communication layer of the first communication layer.

[0114] The decision unit 62 is used to make a decision on the system state data at time t based on the pre-trained decision model, and obtain the control parameters at time t.

[0115] Control unit 63 is used to adjust the bit rate at time t during the wireless screen transmission process according to the control parameters corresponding to time t.

[0116] Optionally, decision unit 62 is also used for: Obtain a pre-trained decision model; The system state data at time t is input into the decision model, and the control parameters at time t are output.

[0117] Optionally, the device 6 also includes a fine-tuning unit 64 for: Calculate the reward value corresponding to time t; wherein, the reward value corresponding to time t is used to characterize the system latency after adjusting the bitrate at time t; Acquire the system status data at time t+1 during the wireless screen sharing process; generate the training sample at time t; wherein, the training sample at time t includes the system status data at time t, the control parameters at time t, the reward value at time t, and the system status data at time t+1; The network parameters of the second network are adjusted according to the training samples corresponding to time t to obtain the adjusted second network. The adjusted decision model is determined based on the first network and the adjusted second network.

[0118] Optionally, the fine-tuning unit 64 is also used for: Calculate the first score corresponding to the time t; wherein the first score corresponding to the time t is used to characterize the wireless screen sharing image quality corresponding to the time t. Calculate the second score corresponding to time t; wherein the second score corresponding to time t is used to characterize the system delay at time t. The reward value corresponding to time t is calculated based on the first score and the second score.

[0119] Optionally, the fine-tuning unit 64 is also used for: Obtain the system delay corresponding to time t; If the system delay is less than or equal to a preset threshold, the reward value at time t is calculated based on the first penalty weight. If the system delay is greater than a preset threshold, the reward value at time t is calculated according to the second penalty weight; wherein the first penalty weight is less than the second penalty weight.

[0120] Optionally, the control unit 63 is also used for: Map the control parameters corresponding to time t to adjustment coefficients; Calculate the target bitrate at time t based on the adjustment coefficient and the bitrate at time t-1. The bitrate at time t during the wireless screen sharing process is adjusted to the target bitrate corresponding to time t.

[0121] Optionally, the control unit 63 is also used for: If the target bitrate at time t is lower than the preset bitrate, then the frame rate and resolution at time t are reduced.

[0122] It should be noted that the information interaction and execution process between the above-mentioned devices / units are based on the same concept as the method embodiments of this application. For details on their specific functions and technical effects, please refer to the method embodiments section, and they will not be repeated here.

[0123] in addition, Figure 6 The wireless screen transmission bitrate control device shown can be built into existing terminal equipment (such as...) Figure 5 The software units, hardware units, or combined software and hardware units within the transmitting end device (as shown) can also be integrated into the terminal device as independent components, or they can exist as independent terminal devices.

[0124] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0125] Figure 7 This is a schematic diagram of the structure of the transmitting device provided in an embodiment of this application. For example... Figure 7 As shown, the transmitting device 7 in this embodiment includes: at least one processor 70 ( Figure 7 (Only one is shown in the image) a processor, a memory 71, and a computer program 72 stored in the memory 71 and executable on the at least one processor 70, wherein the processor 70 executes the computer program 72 to implement the steps in any of the above embodiments of the wireless screen transmission rate control method.

[0126] The transmitting device can be a desktop computer, laptop, handheld computer, or cloud server, etc. This transmitting device may include, but is not limited to, a processor and memory. Those skilled in the art will understand that... Figure 7The example shown is merely of the transmitting device 7 and does not constitute a limitation on the transmitting device 7. It may include more or fewer components than shown, or combine certain components, or different components, such as input / output devices, network access devices, etc.

[0127] The processor 70 can be a Central Processing Unit (CPU), or it can be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor.

[0128] In some embodiments, the memory 71 may be an internal storage unit of the transmitting device 7, such as a hard disk or memory of the transmitting device 7. In other embodiments, the memory 71 may be an external storage device of the transmitting device 7, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc., equipped on the transmitting device 7. Further, the memory 71 may include both internal and external storage units of the transmitting device 7. The memory 71 is used to store the operating system, applications, boot loader, data, and other programs, such as the program code of the computer program. The memory 71 can also be used to temporarily store data that has been output or will be output.

[0129] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, can implement the steps in the above-described method embodiments.

[0130] This application provides a computer program product that, when run on a terminal device, enables the terminal device to implement the steps described in the various method embodiments.

[0131] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments of this application can be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include at least: any entity or device capable of carrying computer program code to a device / terminal equipment, a recording medium, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium. Examples include USB flash drives, portable hard drives, magnetic disks, or optical disks. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electrical carrier signals or telecommunication signals.

[0132] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0133] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0134] In the embodiments provided in this application, it should be understood that the disclosed devices / terminal equipment and methods can be implemented in other ways. For example, the device / terminal equipment embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual coupling or direct coupling or communication connection may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0135] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0136] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A wireless screen transmission bitrate control method, characterized in that, include: Acquire system status data at time t during wireless screen sharing; wherein, the system status data includes feature data of the first communication layer and feature data of the second communication layer between the two devices performing wireless screen sharing; the first communication layer includes the data link layer and the physical layer, and the second communication layer is the upper communication layer of the first communication layer; The system state data at time t is used to make decisions based on the pre-trained decision model to obtain the control parameters at time t. Adjust the bitrate at time t during the wireless screen sharing process according to the control parameters corresponding to time t.

2. The wireless screen transmission bitrate control method as described in claim 1, characterized in that, The decision model includes a first network and a second network; wherein the first network is used to extract feature information from the input data; and the second network is used to make decisions based on the feature information output by the first network. After adjusting the bitrate at time t during wireless screen sharing according to the control parameters corresponding to time t, the method further includes: Calculate the reward value corresponding to time t; wherein, the reward value corresponding to time t is used to characterize the system latency after adjusting the bitrate at time t; Obtain the system status data at time t+1 during the wireless screen sharing process; Generate the training sample corresponding to time t; wherein, the training sample corresponding to time t includes the system state data corresponding to time t, the control parameters corresponding to time t, the reward value corresponding to time t, and the system state data corresponding to time t+1; The network parameters of the second network are adjusted according to the training samples corresponding to time t to obtain the adjusted second network. The adjusted decision model is determined based on the first network and the adjusted second network.

3. The wireless screen transmission bitrate control method as described in claim 2, characterized in that, The calculation of the reward value corresponding to time t includes: Calculate the first score corresponding to the time t; wherein the first score corresponding to the time t is used to characterize the wireless screen sharing image quality corresponding to the time t. Calculate the second score corresponding to time t; wherein the second score corresponding to time t is used to characterize the system delay at time t. The reward value corresponding to time t is calculated based on the first score and the second score.

4. The wireless screen transmission bitrate control method as described in claim 3, characterized in that, The calculation of the second score corresponding to time t includes: Obtain the system delay corresponding to time t; If the system delay is less than or equal to a preset threshold, the reward value at time t is calculated based on the first penalty weight. If the system delay is greater than a preset threshold, the reward value at time t is calculated according to the second penalty weight; wherein the first penalty weight is less than the second penalty weight.

5. The wireless screen transmission bitrate control method according to any one of claims 1 to 4, characterized in that, The step of adjusting the bitrate at time t during wireless screen sharing based on the control parameters corresponding to time t includes: Map the control parameters corresponding to time t to adjustment coefficients; Calculate the target bitrate at time t based on the adjustment coefficient and the bitrate at time t-1. The bitrate at time t during the wireless screen sharing process is adjusted to the target bitrate corresponding to time t.

6. The wireless screen transmission bitrate control method as described in claim 5, characterized in that, The method further includes: If the target bitrate at time t is lower than the preset bitrate, then the frame rate and resolution at time t are reduced.

7. The wireless screen transmission bitrate control method according to any one of claims 1 to 6, characterized in that, The characteristic data of the first communication layer includes signal fluctuation gradient, MAC layer retransmission rate, and physical layer negotiation rate; wherein, the signal fluctuation gradient is used to characterize the attenuation trend of the wireless signal; the MAC layer retransmission rate is used to characterize the probability of retransmission at the MAC layer; and the physical layer negotiation rate is used to characterize the theoretical rate of wireless signal negotiation at the physical layer. The characteristic data of the second communication layer includes round-trip time, one-way delay gradient, and encoder backlog depth; wherein, the round-trip time is used to characterize the data round-trip time of the transport layer; the one-way delay gradient is used to characterize the data backlog of the wireless screen sharing network queue; and the encoder backlog depth is used to characterize the congestion level of the application layer's sending queue.

8. A wireless screen transmission bitrate control device, characterized in that, include: The acquisition unit is used to acquire system status data at time t during the wireless screen sharing process; wherein, the system status data includes feature data of the first communication layer and feature data of the second communication layer between the two devices performing wireless screen sharing; the first communication layer includes the data link layer and the physical layer, and the second communication layer is the upper communication layer of the first communication layer; The decision unit is used to make decisions based on the system state data at time t based on the pre-trained decision model, and obtain the control parameters at time t. The control unit is used to adjust the bit rate at time t during the wireless screen transmission process according to the control parameters corresponding to time t.

9. A wireless screen sharing system, characterized in that, It includes a transmitting device and a receiving device; the transmitting device and the receiving device are connected wirelessly. The transmitting device is configured to determine the wireless screen transmission bitrate according to the method described in any one of claims 1 to 7, and to transmit data to the receiving device based on the wireless screen transmission bitrate.

10. A transmitting device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the method as described in any one of claims 1 to 7.

Images