Screen projection data transmission method and device and electronic equipment

Abstract

The invention discloses a projection screen data transmission method and device and electronic equipment, and belongs to the field of data transmission. The method comprises the following steps: generating a screen projection service message corresponding to a screen projection data frame according to the frame type of the to-be-transmitted screen projection data frame in the screen projection data; the screen projection service message comprises frame data of the screen projection data frame and a frame type identifier for indicating a frame type; transmitting the screen projection service message through the first queue, and determining a retransmission strategy of the first screen projection service message in the transmission process according to the transmission link quality information of the first queue; the first screen projection service message is a screen projection service message which fails to be transmitted.

Description

Technical Field

[0001] The present application belongs to the field of data transmission, and specifically relates to a screen projection data transmission method, device and electronic device. Background Art

[0002] As the ecological layout of smartphone manufacturers gradually improves, users' demand for screen projection services continues to increase, and their requirements for screen projection experience are also getting higher and higher. The screen projection process can usually be divided into the encoding stage, encapsulation stage, transmission stage, decapsulation and display stage. Specifically, the projection data is encoded at the sending end; the encoded data is encapsulated through the User Datagram Protocol (UDP) or the Transmission Control Protocol (TCP); the projection data is transmitted to the receiving end through the wireless air interface; the receiving end network layer completes the data decapsulation, and generates and displays the projection interface based on the decapsulated data.

[0003] In order to improve the real-time performance of screen projection, related technologies often focus on wireless air interface optimization, transmission layer protocol optimization or codec-related optimization, and the overall screen projection effect has not been significantly improved, especially in multi-device and multi-screen collaborative scenarios such as TVs, tablets, computers and mobile phones. Data transmission is very sensitive to latency. Unsatisfactory latency conditions will directly affect the real-time performance of screen projection, resulting in abnormal problems such as unsynchronized audio and video, screen distortion, and image freeze. Summary of the invention

[0004] The purpose of the embodiments of the present application is to provide a screen projection data transmission method and electronic device, which can reduce the problems of asynchrony between audio and video, screen distortion, and screen freeze in screen projection, and improve the fluency of the user's screen projection experience.

[0005] In a first aspect, an embodiment of the present application provides a screen projection data transmission method, the method comprising: generating a screen projection service message corresponding to the screen projection data frame according to the frame type of the screen projection data frame to be transmitted in the screen projection data; the screen projection service message comprises the frame data of the screen projection data frame and a frame type identifier for indicating the frame type; the screen projection service message is transmitted through a first queue, and during the transmission process, a retransmission strategy of the first screen projection service message is determined according to the transmission link quality information of the first queue; the first screen projection service message is a screen projection service message that fails to transmit.

[0006] In the second aspect, an embodiment of the present application provides a screen projection data transmission device, which includes: a determination module, used to generate a screen projection service message corresponding to the screen projection data frame according to the frame type of the screen projection data frame to be transmitted in the screen projection data; the screen projection service message includes the frame data of the screen projection data frame and a frame type identifier used to indicate the frame type; a retransmission module, used to transmit the screen projection service message through a first queue, and during the transmission process, determine the retransmission strategy of the first screen projection service message according to the transmission link quality information of the first queue; the first screen projection service message is a screen projection service message that fails to transmit.

[0007] In a third aspect, an embodiment of the present application provides an electronic device, which includes a processor and a memory, wherein the memory stores programs or instructions that can be run on the processor, and when the program or instructions are executed by the processor, the steps of the method described in the first aspect are implemented.

[0008] In a fourth aspect, an embodiment of the present application provides a readable storage medium, on which a program or instruction is stored, and when the program or instruction is executed by a processor, the steps of the method described in the first aspect are implemented.

[0009] In a fifth aspect, an embodiment of the present application provides a chip, comprising a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run a program or instruction to implement the method described in the first aspect.

[0010] In a sixth aspect, an embodiment of the present application provides a computer program product, which is stored in a storage medium and is executed by at least one processor to implement the method described in the first aspect.

[0011] In an embodiment of the present application, a screen projection service message corresponding to the screen projection data frame is generated according to the frame type of the screen projection data frame to be transmitted in the screen projection data; the screen projection service message includes the frame data of the screen projection data frame and a frame type identifier for indicating the frame type; the screen projection service message is transmitted through the first queue, and during the transmission process, the retransmission strategy of the first screen projection service message is determined according to the transmission link quality information of the first queue; the first screen projection service message is a screen projection service message that fails to transmit. In this way, by considering multiple technical aspects in the screen projection process, including the management of frame types, the monitoring of transmission link quality information, and the optimization of retransmission strategies, the retransmission delay of the screen projection service message can be reduced in an interference environment, and the screen projection data frames of specific frame types, such as key frames, can be more accurately identified to ensure the reliable transmission of screen projection data frames of specific frame types, reduce problems such as asynchrony of audio and video, screen distortion, and screen freeze in screen projection, thereby improving the fluency of the user's screen projection experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 It is a flowchart of a screen projection data transmission method provided by some embodiments of the present application;

[0013] Figure 2 is a schematic diagram of priority queue transmission scheduling in a data transmission process provided by some embodiments of the present application;

[0014] Figure 3 is an example diagram of priority queue scheduling provided by some embodiments of the present application;

[0015] Figure 4 It is a flowchart of a screen projection data transmission method provided by some embodiments of the present application;

[0016] Figure 5 It is a flowchart of a screen projection data transmission method provided by some embodiments of the present application;

[0017] Figure 6 It is a flowchart of a screen projection data transmission method in a home entertainment scenario provided by some embodiments of the present application;

[0018] Figure 7 A flowchart of a method for transmitting screen projection data in an online conference scenario provided by some embodiments of the present application;

[0019] Figure 8 is a structural schematic diagram of a screen projection data transmission device provided in some embodiments of the present application;

[0020] Fig. 9 is a schematic diagram of the structure of an electronic device provided by some embodiments of the present application;

[0021] Fig.10 It is a schematic diagram of the hardware structure of an electronic device provided in some embodiments of the present application. DETAILED DESCRIPTION

[0022] The following will be combined with the drawings in the embodiments of the present application to clearly describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments in the present application belong to the scope of protection of this application.

[0023] The terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described here, and the objects distinguished by "first", "second", etc. are generally of one type, and the number of objects is not limited. For example, the first object can be one or more. In addition, "and / or" in the specification and claims represents at least one of the connected objects, and the character " / " generally indicates that the objects associated with each other are in an "or" relationship.

[0024] The following, in conjunction with the accompanying drawings, describes in detail the screen projection data transmission method provided in the embodiment of the present application through specific embodiments and their application scenarios.

[0025] It should be noted that the screen projection data transmission method provided in the embodiments of the present application can be executed by electronic devices such as mobile phones, tablet computers, laptop computers, PDAs, and vehicle-mounted electronic devices. In some embodiments of the present application, the screen projection data transmission method provided in the embodiments of the present application is illustrated by taking an electronic device as the execution subject to execute the screen projection data transmission method.

[0026] The screen projection data transmission method provided in the embodiment of the present application can be applied to a variety of scenarios that require high-quality data transmission. Among them, one specific application scenario is a home entertainment scenario, for example, wirelessly projecting video content on a mobile phone, tablet or computer to a TV. The mobile phone, tablet or computer can transmit video data frames to the TV through the screen projection data transmission method to ensure the reliable transmission of key frames and avoid screen freezes or frame drops during video playback. Another specific application scenario is an online meeting scenario, where participants share presentations or videos with all participants through screen projection. The first terminal used by the participant can use the screen projection data transmission method to transmit the presentation content data frame to the second terminal used by the participant to ensure that the participant can see the presentation content clearly.

[0027] See also Figure 1 , Figure 1 1 is a flow chart of a screen projection data transmission method provided by some embodiments of the present application. As shown in the figure, the method 100 includes:

[0028] Step 101: Generate a screen projection service message corresponding to the screen projection data frame according to the frame type of the screen projection data frame to be transmitted in the screen projection data; the screen projection service message includes the frame data of the screen projection data frame and a frame type identifier used to indicate the frame type.

[0029] The frame types include key frames (I frames), predicted frames (P frames), and bidirectional predicted frames (B frames).

[0030] In a specific implementation, the screen projection data may include multiple screen projection data frames to be transmitted. For each frame type of the screen projection data frame, a screen projection service message corresponding to the screen projection data frame is generated. The screen projection service message includes the frame data of the screen projection data frame and a frame type identifier for indicating the frame type, and then the screen projection service message is transmitted. The sending end can send the screen projection data frame to the receiving end in units of screen projection data frames in the time sequence of each screen projection data frame. Here, the frame type identifier can be an identifier for distinguishing screen projection data frames of different frame types. For example, identifier 0 is used to represent an I frame, identifier 1 is used to represent a P frame, and identifier 2 is used to represent a B frame.

[0031] Taking the home entertainment scenario as an example, the mobile phone generates a screen projection service message corresponding to the video data frame according to the frame type of the video data frame to be transmitted in the screen projection data, and sends the screen projection service message to the TV end according to the time sequence of each video data frame; taking the online meeting scenario as an example, the first terminal held by the participant generates a screen projection service message corresponding to the demonstration content data frame according to the frame type of the demonstration data frame to be transmitted in the screen projection data, and sends the screen projection service message to the second terminal held by the participant according to the time sequence of each demonstration content data frame.

[0032] In a possible implementation, in the above step 101, generating a screen projection service message corresponding to the screen projection data frame according to the frame type of the screen projection data frame to be transmitted in the screen projection data includes:

[0033] The frame data and frame type identifier of the screen projection data frame to be transmitted in the screen projection data are encapsulated into a message to obtain a screen projection service message corresponding to the screen projection data frame.

[0034] In some embodiments of the present application, the UDP protocol can be used to encapsulate the frame data and frame type identifier of the projection data frame to be transmitted in the projection data, and obtain the projection service message corresponding to the projection data frame. The projection service message includes a field for indicating the frame type identifier. For example, in the data part of the UDP protocol, the first byte is used to indicate the frame type identifier. In this way, during the data processing of the kernel, the projection data frames of different frame types can be identified by the frame type identifier.

[0035] Taking the home entertainment scenario as an example, the mobile phone can use the UDP protocol to encapsulate the frame data and frame type identifier of the video data frame to be transmitted in the projection data to obtain the projection service message; taking the online meeting scenario as an example, the first terminal can use the UDP protocol to encapsulate the frame data and frame type identifier of the presentation content data frame to be transmitted in the projection data to obtain the projection service message.

[0036] In some other possible implementations of the present application, in the above step 101, generating a screen projection service message corresponding to the screen projection data frame according to the frame type of the screen projection data frame to be transmitted in the screen projection data includes:

[0037] By sending a message function, the frame data and frame type identifier of the projection data frame to be transmitted in the projection data are encapsulated in a structure;

[0038] According to the frame type in the structure, the frame data is encapsulated into a message to obtain the screen projection service message corresponding to the screen projection data frame.

[0039] In an embodiment of the present application, the frame type identifier corresponding to the frame data of the projection data frame can be attached to the message through the message sending function. For example, the sendmsg message sending function allows the msg_control or msg_flags field to carry auxiliary information such as the frame type identifier, so that different types of projection data frames can be identified when the kernel processes the message.

[0040] Taking the home entertainment scenario as an example, the mobile phone can also encapsulate the frame data and frame type identifier of the video data frame to be transmitted in the msghdr structure of sendmsg, and encapsulate the frame data into a message according to the frame type in the msghdr structure to obtain a screen projection service message; taking the online conference scenario as an example, the first terminal can also encapsulate the frame data and frame type identifier of the presentation content data frame to be transmitted in the msghdr structure of sendmsg, and encapsulate the frame data into a message according to the frame type in the msghdr structure to obtain a screen projection service message.

[0041] Step 102: Transmit the screen projection service message through the first queue. During the transmission process, determine the retransmission strategy of the first screen projection service message based on the transmission link quality information of the first queue; the first screen projection service message is a screen projection service message that fails to transmit.

[0042] In an exemplary embodiment, if Figure 2As shown, in the 802.1D standard, the MAC Service Data Unit (MSDU) of the Media Access Control (MAC) layer with a label can be used to identify different service types or data flows, and the data frame is mapped to the corresponding access category according to its priority, so that it can be properly queued and processed according to the priority during the transmission process; the data frames to be sent are temporarily stored in the sending queue, and different priority queues are used to distinguish data with different service quality requirements. For example, the high priority queue may be used to store voice and video data frames to ensure that they are sent first before ordinary data frames; wherein, the sending queue includes the VO (Voice) queue, the VI (Video) queue, the Best Efforts queue and the Background queue; an internal conflict resolution mechanism is used to ensure that conflicts can be effectively avoided or resolved when sending data frames on the shared medium; and further, during the sending process, if a conflict is detected or the sending fails, multiple sending attempts may be made until the data frame is successfully sent or the maximum number of attempts is reached. In an embodiment of the present application, a first queue, such as a VI queue, can be used to transmit screen projection service messages, and the clear channel assessment (CCA) parameters, received signal strength (RSSI), link rate (LinkSpeed), queue length and other link quality information of the VI queue can be monitored to evaluate the transmission status of the frame data in real time through the link quality information.

[0043] Furthermore, during the transmission process, the retransmission strategy of the first screen projection service message is determined according to the transmission link quality information of the first queue, and the first screen projection service message is a screen projection service message that fails to be transmitted. For example, when it is determined based on the transmission link quality information that the transmission link of the first queue is better, all the screen projection service messages that fail to be sent can be retransmitted; when it is determined based on the transmission link quality information that the transmission link of the first queue is not ideal, the screen projection service messages with a specific frame type that fail to be sent can be retransmitted to ensure the timeliness and integrity of the screen projection service messages. In this way, when the transmission link is not ideal, the number of unnecessary retransmissions can be reduced and the retransmission delay of the screen projection service message can be reduced.

[0044] Taking the home entertainment scenario as an example, the mobile phone transmits the screen projection service message to the TV end through the VI queue. During the transmission process, the mobile phone determines the retransmission strategy of the first screen projection service message that fails to be transmitted based on the transmission link quality information of the VI queue. Since the frame type of the first screen projection service message can be identified by the frame type identifier included in the screen projection service message, the first screen projection service message of the specific frame type that fails to be transmitted can be retransmitted according to the transmission link quality information of the VI queue to ensure the reliable transmission of key frames and avoid screen freeze or frame loss during video playback; taking the online conference scenario as an example, the first terminal transmits the screen projection service message to the second terminal through the VI queue. During the transmission process, the first terminal determines the retransmission strategy of the first screen projection service message that fails to be transmitted based on the transmission link quality information of the VI queue. Since the frame type of the first screen projection service message can be identified by the frame type identifier included in the screen projection service message, the first screen projection service message of the specific frame type that fails to be transmitted can be retransmitted according to the transmission link quality information of the VI queue to ensure that the participants can see clear presentation content through the second terminal.

[0045] In a possible implementation, in the above step 102, determining a retransmission strategy for the first projection service message according to the transmission link quality information of the first queue, includes:

[0046] Determining a link quality metric value according to the measurement strategy for transmitting the link quality information, wherein the link quality metric value is used to indicate the degree of influence of the link quality information on the link quality;

[0047] Determine a link quality evaluation value based on the link quality metric value, where the link quality evaluation value is used to indicate the link quality corresponding to the link quality information;

[0048] Based on the link quality evaluation value, determine the retransmission strategy of the first screen projection service message.

[0049] Among them, the link quality metric value can be determined according to different requirements of different user scenarios. In a possible implementation, the above-mentioned link quality information includes: a clear channel assessment parameter CCA, a received signal strength RSSI and a link rate LinkSpeed;

[0050] The determining of a link quality evaluation value based on the link quality metric value comprises:

[0051] Determining the link quality assessment value based on the first link quality metric value, the second link quality metric value, and the third link quality metric value;

[0052] The first link quality metric value is a metric value of the idle channel assessment parameter, the second link quality metric value is a metric value of the received signal strength, and the third link quality metric value is a metric value of the link rate.

[0053] In some embodiments of the present application, the link quality evaluation value LQ may be determined according to the following formula:

[0054] LQ=100-f(CCA)-f(RSSI)-f(LinkSpeed);

[0055] Among them, the first link quality metric value f(CCA) represents the metric value of the influence of different CCA parameters on the link quality, the second link quality metric value f(RSSI) represents the metric value of the influence of different RSSI parameters on the link quality, and the third link quality metric value f(LinkSpeed) represents the metric value of the influence of different LinkSpeed ​​parameters on the link quality; according to the preset mapping strategy, the link quality metric value of each link quality information is determined, and the mapping strategy is shown in Table 1 to Table 3:

[0056] Table 1. Mapping relationship between clear channel assessment CCA parameters and metric values

[0057] CCA Metrics 0～100 0 100～150 10 150～200 20 200～255 40

[0058] Table 2. Mapping relationship between received signal strength RSSI and measurement value

[0059] RSSI Metrics 0~-55dbm 0 -55~-65dbm 10 -65~-75dbm 20 Less than -75dbm 40

[0060] Table 3. Mapping relationship between link rate LinkSpeed ​​and metric value

[0061] LinkSpeed Metrics More than 144Mbps 0 54Mbps~144Mbps 10 24Mbps~54Mbps 20 11Mbps~24Mbps 30 1Mbps~11Mbps 40

[0062] In a possible implementation manner, the determining of the retransmission strategy of the first screen projection service message based on the link quality evaluation value includes:

[0063] When the link quality evaluation value is greater than or equal to the first evaluation value threshold, retransmitting the first screen projection service message through the second queue;

[0064] The transmission priority of the second queue is higher than the transmission priority of the first queue.

[0065] Here, the first evaluation value threshold can be selected according to an empirical value, for example, the first evaluation value threshold is 60; taking the first queue as a VI queue as an example, the transmission priority of the second queue is higher than the transmission priority of the first queue, and the second queue can be a VO queue.

[0066] In an exemplary embodiment, the link quality metric LQ of the VI queue is determined relative to the first evaluation value threshold. If LQ ≥ 60, the priority queue retransmission strategy is triggered, and the first projection service message that failed to be transmitted is retransmitted through the VO queue.

[0067] Taking the home entertainment scenario as an example, the mobile phone transmits the screen projection service message to the TV end through the VI queue. During the transmission process, if the transmission link quality information LQ ≥ 60 of the VI queue, the first screen projection service message that failed to be sent is retransmitted to avoid the loss of video data frames; taking the online conference scenario as an example, the first terminal transmits the screen projection service message to the second terminal through the VI queue. During the transmission process, if the transmission link quality information LQ ≥ 60 of the VI queue, the first screen projection service message that failed to be sent is retransmitted to avoid the loss of presentation content data frames.

[0068] like Figure 3 As shown, when the VI queue is preparing to send the screen projection service message, it will first detect whether the channel is idle, which can be understood as whether other devices are transmitting data. If the channel is idle, a random backoff time is selected for countdown. During this period, if the channel is detected to be busy, the countdown is stopped and the channel is waited for to become idle again. When the screen projection service message fails to be sent, the VO queue with a higher transmission priority is used to retransmit the failed screen projection service message DATA. After the retransmission is successful, all subsequent screen projection service messages can be transmitted using the VO queue until the screen projection message fails to be sent again and needs to be retransmitted or the VO queue is too long, and then it is switched to transmission through the VI queue.

[0069] In a possible implementation, the frame type includes a key frame I frame; and the determining of the retransmission strategy of the first screen projection service message based on the link quality evaluation value includes:

[0070] When the link quality evaluation value is less than the first evaluation value threshold, the link quality evaluation value is greater than or equal to the second evaluation value threshold, and the frame type indicated by the frame type identifier included in the first screen projection service message is an I frame, the first screen projection service message is retransmitted through the second queue;

[0071] The transmission priority of the second queue is higher than the transmission priority of the first queue, and the second evaluation value threshold is lower than the first evaluation value threshold.

[0072] Continuing with the above example, the second evaluation value threshold can be selected based on an empirical value, for example, the second evaluation value threshold is 30. When the link quality evaluation value is less than the first evaluation value threshold, and the link quality evaluation value is greater than or equal to the second evaluation value threshold, that is, 30≤LQ<60, if the I frame in the screen projection service message fails to be sent, the second queue is used to retransmit the failed I frame, and the failed screen projection service messages of other frame types except the I frame are not processed, where the other frame types can be P frames or B frames.

[0073] Taking the home entertainment scenario as an example, the mobile phone transmits the screen projection service message to the TV through the VI queue. During the transmission process, if the transmission link quality information of the VI queue is 30≤LQ<60, only the I frame that failed to be sent is retransmitted to ensure the reliable transmission of key video data frames when the network condition is not ideal, thereby improving the smoothness of the user's screen projection experience; taking the online meeting scenario as an example, the first terminal transmits the screen projection service message to the second terminal through the VI queue. During the transmission process, if the transmission link quality information of the VI queue is 30≤LQ<60, only the I frame that failed to be sent is retransmitted to ensure the reliable transmission of key content data frames when the network condition is not ideal, thereby avoiding the loss of presentation content data frames.

[0074] Here, I frame is a key frame in the video sequence. It uses intra-frame coding and only uses the data in the frame for coding. I frame contains the data of the complete picture, so it is also called key frame or independent frame. When decoding, I frame can independently decode the complete picture without referring to other frames. Since I frame contains complete data, its data volume is relatively large.

[0075] P frames use inter-frame coding, which uses the previous I frame or P frame for predictive coding. P frames only contain the difference data between the previous frame and the previous frame, so the amount of data is relatively small. When decoding, P frames need to refer to the previous I frame or P frame to decode the complete picture. Since P frames only contain difference data, their compression efficiency is higher.

[0076] B frames also use inter-frame coding, but they refer not only to the previous I or P frame, but also to the next I or P frame for predictive coding. B frames contain the difference data between the previous and next frames, so their data volume is also small. When decoding, B frames need to refer to the previous and next frames to decode a complete picture. Since B frames refer to both the previous and next frames at the same time, their compression efficiency is higher.

[0077] Based on the characteristics of different types of video data frames, it can be seen that different frame types have different importance to the screen projection experience. Among them, the I frame contains complete data and can independently decode the complete picture without referring to other frames. The importance of the I frame is higher than that of the P frame and the B frame. Therefore, when 30≤LQ<60, the priority queue retransmission strategy can be triggered only when the I frame fails to send, and the P frame and B frame are not processed.

[0078] In a possible implementation, the frame types include key frames I frames, predicted frames P frames, and bidirectional predicted frames B frames; and the above-mentioned determining the retransmission strategy of the first screen projection service message based on the link quality evaluation value includes:

[0079] When the link quality evaluation value is less than the second evaluation value threshold, the data transmission length in the first queue exceeds the preset length threshold, and the frame type indicated by the frame type identifier included in the first screen projection service message is an I frame, the first screen projection service message is retransmitted through the second queue, and the second screen projection service message in the first queue is discarded;

[0080] Among them, the transmission priority of the second queue is higher than the transmission priority of the first queue, and the frame type indicated by the frame type identifier included in the second screen projection service message is a P frame or a B frame.

[0081] Continuing with the above example, when the link quality assessment value is less than the second assessment value threshold, that is, LQ < 30, and the data transmission length in the first queue exceeds the preset length threshold, if the I frame in the projection service message fails to be sent, the second queue is used to retransmit the failed I frame, and the failed P frame and B frame are discarded. The length threshold can be selected based on an empirical value, for example, the length threshold value is 100.

[0082] Taking the home entertainment scenario as an example, the mobile phone transmits the screen projection service message to the TV through the VI queue. During the transmission process, if the transmission link quality information LQ of the VI queue is less than 30, and the data transmission length in the VI queue is less than 100, only the I frame that failed to be sent is retransmitted, and the P frame and B frame that failed to be sent are discarded, so as to ensure the reliable transmission of key video data frames under unsatisfactory network conditions and improve the smoothness of the user's screen projection experience; taking the online meeting scenario as an example, the first terminal transmits the screen projection service message to the second terminal through the VI queue. During the transmission process, if the transmission link quality information LQ of the VI queue is less than 30, and the data transmission length in the VI queue is less than 100, only the I frame that failed to be sent is retransmitted, and the P frame and B frame that failed to be sent are discarded, so as to ensure the reliable transmission of key content data frames under unsatisfactory network conditions and avoid the loss of presentation content data frames.

[0083] In a possible implementation manner, after discarding the second screen projection service message in the first queue, the method further includes:

[0084] The encoding bit rate of the projection data is reduced from a first bit rate to a second bit rate.

[0085] In an exemplary embodiment, when LQ is less than 30 and the data transmission length is greater than 100, the encoding bit rate of the projection data is adjusted from a high bit rate to a low bit rate. For example, the current default encoding rate for projection is generally 60fps. In this case, the encoding bit rate can be switched to 30fps.

[0086] Figure 4 It is a flowchart of a screen projection data transmission method provided by some embodiments of the present application. As shown in the figure, the screen projection data transmission method includes the following steps:

[0087] Step 401: Generate a screen projection service message corresponding to the screen projection data frame according to the frame type of the screen projection data frame to be transmitted in the screen projection data, for example, mark the I frame, P frame and B frame in the screen projection data to obtain the screen projection service message; the screen projection service message includes the frame data of the screen projection data frame and a frame type identifier for indicating the frame type;

[0088] Among them, the frame data and frame type identifier of the video data frame to be transmitted in the projection data can be encapsulated in a message to obtain the projection service message corresponding to the projection data frame; the frame data and frame type identifier of the video data frame to be transmitted in the projection data can also be encapsulated in a structure through the sending message function; the structure is encapsulated in a message to obtain the projection service message corresponding to the video data frame.

[0089] Step 402: before transmitting the screen projection service message through the first queue, it is also possible to determine whether the message to be transmitted is a screen projection service message according to the frame type identifier. If the message to be transmitted is a screen projection service message, proceed to step 403; if the message to be transmitted is not a screen projection service message, the process ends;

[0090] Step 403: Transmit the screen projection service message through the first queue. For example, the first queue is a VI queue, and the screen projection service message is placed in the VI queue for transmission;

[0091] Step 404: during the transmission of the screen projection service message, determining the link quality assessment value of the VI queue based on the transmission link quality information of the VI queue; wherein the transmission link quality information includes an idle channel assessment parameter, a received signal strength, and a link rate;

[0092] Among them, a link quality measurement value can be determined according to the measurement strategy of transmitting link quality information, and the link quality measurement value is used to indicate the degree of influence of the link quality information on the link quality; based on the link quality measurement value, a link quality evaluation value is determined, and the link quality evaluation value is used to indicate the link quality corresponding to the link quality information.

[0093] Step 405: Determine whether a first condition is met, the first condition being that the link quality evaluation value is greater than or equal to a first evaluation value threshold, for example, determine whether the link quality evaluation value LQ ≥ 60 is met, if so, proceed to step 406, otherwise proceed to step 407;

[0094] Step 406: When the screen projection service message fails to be sent, the priority queue retransmission strategy is triggered, wherein the priority queue retransmission strategy is to retransmit the first screen projection service message that failed to be sent through the second queue, wherein the second queue is the VO queue, and the frame type of the screen projection data frame is not distinguished at this time;

[0095] Step 407: Determine whether the second condition is met, the second condition being that the link quality evaluation value is less than the first evaluation value threshold, and the link quality evaluation value is greater than or equal to the second evaluation value threshold, for example, determine whether the evaluation value of the link quality information 30≤LQ<60 is met, if so, proceed to step 408; otherwise, proceed to step 409;

[0096] Step 408: When the frame type indicated by the frame type identifier included in the first screen projection service message that failed to be sent is an I frame, the priority queue retransmission strategy is triggered; wherein the priority queue retransmission strategy is to retransmit the first screen projection service message that failed to be sent through the VO queue, and at this time, only when the I frame fails to be sent, the priority retransmission strategy is triggered;

[0097] Step 409: Determine whether the third condition is met, the third condition is that the link quality evaluation value is less than the second evaluation value threshold, and the data transmission length in the VI queue exceeds the preset length threshold, for example, determine whether the link quality information evaluation value LQ is less than 30, and the data transmission length is greater than 100. If so, proceed to step 410; otherwise, end the process;

[0098] Step 410: When the frame type indicated by the frame type identifier included in the first screen projection service message is an I frame, the priority queue retransmission strategy is triggered, and the P frames or B frames in the queue are discarded. At this time, when only the I frame fails to be sent, the priority retransmission strategy is triggered, and the P frames or B frames in the queue are discarded. At the same time, the encoding bit rate of the screen projection data is reduced from the first bit rate to the second bit rate.

[0099] In a possible implementation, in the above step 102, after determining the retransmission strategy of the first projection service message according to the transmission link quality information of the first queue, the step further includes:

[0100] When the number of retransmissions of the first screen projection service message is greater than the retransmission number threshold, generating a third screen projection service message in the screen projection data;

[0101] Among them, the third screen projection service message is a screen projection service message corresponding to a screen projection data frame which is adjacent to the screen projection data frame corresponding to the first screen projection service message and whose data frame generation time is later than that of the screen projection data frame.

[0102] In the embodiment of the present application, the number of retransmissions can be limited. If a certain number of retransmissions are unsuccessful, the upper-layer application is notified to generate the next frame of the projection service message adjacent to the corresponding frame of the projection service message, so that the smoothness of the service will not be greatly affected. Specifically, when the data packet of the projection service message is lost, the kernel puts the skb containing the lost information into the error queue of the socket and sets the MSG_ERRQUEUE flag; the sending application parses the message with the MSG_ERRQUEUE flag to obtain the packet loss information, and then generates the next frame adjacent to the frame corresponding to the current packet loss information when the number of retransmissions exceeds the preset number threshold.

[0103] In an exemplary embodiment, if Figure 5 As shown, the above-mentioned screen projection data transmission method also includes:

[0104] Step 411: Determine whether the number of retransmissions of the first projection service message is greater than the retransmission threshold. The retransmission threshold can be selected based on an empirical value. For example, the retransmission threshold is 1. Determine whether the retransmitted I frame is sent successfully. If the retransmitted I frame fails to be sent, proceed to step 612, otherwise the process ends.

[0105] Step 412: Generate a third screen projection service message in the screen projection data; wherein the third screen projection service message is a screen projection service message corresponding to a screen projection data frame that is adjacent to the screen projection data frame corresponding to the first screen projection service message and whose data frame generation time is later than that of the screen projection data frame, for example, generating the next I frame adjacent to the retransmitted I frame.

[0106] The embodiment of the present application provides a method for transmitting screen projection data, which generates a screen projection service message corresponding to the screen projection data frame according to the frame type of the screen projection data frame to be transmitted in the screen projection data; the screen projection service message includes the frame data of the screen projection data frame and a frame type identifier for indicating the frame type; the screen projection service message is transmitted through the first queue, and during the transmission process, the retransmission strategy of the first screen projection service message is determined according to the transmission link quality information of the first queue; the first screen projection service message is a screen projection service message that fails to transmit. In this way, by considering multiple technical aspects in the screen projection process, including the management of frame types, the monitoring of transmission link quality information, and the optimization of retransmission strategies, the retransmission delay of the screen projection service message can be reduced in an interference environment, and the screen projection data frames of specific frame types, such as key frames, can be more accurately identified, to ensure the reliable transmission of screen projection data frames of specific frame types, reduce the problems of screen projection audio and video asynchrony, screen distortion, and screen freeze, thereby improving the fluency of the user's screen projection experience.

[0107] In some embodiments of the present application, taking the home entertainment scenario as an example, the execution subject of the above-mentioned screen projection data transmission method can be a mobile terminal such as a mobile phone, tablet or computer, and the mobile terminal projects the video content to the TV. The following takes the sending end as a mobile phone and the receiving end as a TV as an example to illustrate the screen projection data transmission method provided in the embodiment of the present application. Figure 6 As shown, the screen projection data transmission method may specifically include the following steps:

[0108] Step 601: The mobile phone generates a screen projection service message corresponding to the video data frame according to the frame type of the video data frame to be transmitted in the screen projection data; the screen projection service message includes the frame data of the video data frame and a frame type identifier for indicating the frame type;

[0109] Among them, the frame data and frame type identifier of the video data frame to be transmitted in the projection data can be encapsulated in a message to obtain the projection service message corresponding to the projection data frame; the frame data and frame type identifier of the video data frame to be transmitted in the projection data can also be encapsulated in a structure through the sending message function; the structure is encapsulated in a message to obtain the projection service message corresponding to the video data frame.

[0110] Step 602: Transmitting the screen projection service message through the first queue; for example, transmitting the screen projection service message to the TV end through the VI queue;

[0111] Step 603: Acquire transmission link quality information of the VI queue; wherein the transmission link quality information includes idle channel assessment parameters, received signal strength and link rate;

[0112] Step 604: Determine a link quality evaluation value LQ based on the transmission link quality information;

[0113] Among them, a link quality metric value can be determined according to a metric strategy for transmitting link quality information, and the link quality metric value is used to indicate the degree of influence of the link quality information on the link quality; based on the link quality metric value, a link quality evaluation value is determined, and the link quality evaluation value is used to indicate the link quality corresponding to the link quality information;

[0114] For example, the idle channel assessment parameter is 120, the received signal strength is -30dbm, and the link rate is 144Mbps; according to the mapping relationship between the link quality information and the metric value in Tables 1 to 3 above, the metric value of the idle channel assessment parameter is 10, the metric value of the received signal strength is 0, and the metric value of the link rate is 10, and then the link quality assessment value LQ=100-10-0-10=80;

[0115] In this way, by analyzing the transmission link quality information, such as idle channel assessment parameters, received signal strength, link rate, etc., the link quality of the transmission link can be accurately evaluated, thereby providing a theoretical basis for the subsequent determination of the retransmission strategy of the first projection service message;

[0116] Step 605: Determine whether the link quality evaluation value is greater than or equal to a first evaluation value threshold. The first evaluation value threshold can be selected based on an empirical value, such as 60, that is, whether LQ≥60 is established; if established, proceed to step 606, otherwise proceed to step 607; for example, the link quality evaluation value LQ obtained in the above step 604 is LQ=80, LQ≥60 is established, and proceed to step 606;

[0117] Step 606: retransmitting the screen projection service message that failed to be transmitted through the second queue, for example, retransmitting the screen projection service message that failed to be transmitted through the VO queue; proceeding to step 611;

[0118] Step 607: Continue to determine whether the link quality evaluation value is greater than or equal to the second evaluation value threshold. The second evaluation value threshold can be selected based on an empirical value, such as 30, that is, whether 30≤LQ<60 is established; if so, proceed to step 608, otherwise proceed to step 609;

[0119] Step 608: When the first screen projection service message whose frame type indicated by the frame type identifier is an I frame fails to be sent, the first screen projection service message is retransmitted through the VO queue, and the process proceeds to step 611;

[0120] In this way, reliable transmission of key video data frames can be ensured even when network conditions are not ideal, improving the smoothness of the user's screen projection experience;

[0121] Step 609: Continue to determine whether the data transmission length in the VI queue exceeds a preset length threshold. The preset length threshold can be selected based on an empirical value. For example, the preset length threshold is 100, that is, LQ < 30, and the data transmission length > 100. If so, proceed to step 610; otherwise, end the process.

[0122] Step 610: When the first screen projection service message whose frame type indicated by the frame type identifier is an I frame fails to be sent, the first screen projection service message is retransmitted through the VO queue, and the second screen projection service message in the VI queue is discarded, the frame type indicated by the frame type identifier included in the second screen projection service message is a P frame or a B frame, and the encoding bit rate of the screen projection data is reduced from the first bit rate to the second bit rate;

[0123] In this way, when the network condition is poor and the amount of data transmitted is large, by discarding P frames and B frames, the reliable transmission of I frames can be ensured, reducing problems such as asynchrony of audio and video, screen distortion, and image freezes.

[0124] Step 611: Determine whether the number of retransmissions of the first projection service message is greater than the retransmission threshold. The retransmission threshold can be selected based on an empirical value. For example, the retransmission threshold is 1. Determine whether the retransmitted I frame is sent successfully. If the retransmitted I frame fails to be sent, proceed to step 612, otherwise the process ends.

[0125] Step 612: Generate a third screen projection service message in the screen projection data; wherein the third screen projection service message is a screen projection service message corresponding to a screen projection data frame that is adjacent to the video data frame corresponding to the first screen projection service message and whose data frame generation time is later than that of the video data frame, for example, generating the next I frame adjacent to the retransmitted I frame;

[0126] In this way, when the first screen projection service message is retransmitted many times, the next frame can be generated in time, which can reduce the waiting delay caused by the failure of the previous frame transmission and improve the data transmission efficiency.

[0127] The embodiments of the present application take into account multiple technical aspects of the screen projection process, including frame type management, monitoring of transmission link quality information, and optimization of retransmission strategies. It can determine whether to retransmit certain screen projection service messages, especially screen projection service messages containing key frames, in an interference environment, especially when the wireless network environment is poor, to ensure the reliable transmission of key frames, reduce problems such as asynchrony of audio and video in screen projection, screen distortion, and screen freeze, thereby improving the smoothness of the user's screen projection experience.

[0128] In some embodiments of the present application, taking an online conference scenario as an example, the execution subject of the above-mentioned screen projection data transmission method can be a first terminal used by a participant, and the first terminal projects the presentation content to a second terminal used by the participant. The following takes the sending end as the first terminal and the receiving end as the second terminal as an example to illustrate the screen projection data transmission method provided in the embodiment of the present application. Figure 7 As shown, the screen projection data transmission method may specifically include the following steps:

[0129] Step 701: The first terminal generates a screen projection service message corresponding to the video data frame according to the frame type of the presentation content data frame to be transmitted in the screen projection data; the screen projection service message includes the frame data of the presentation content data frame and a frame type identifier for indicating the frame type;

[0130] Among them, the frame data and frame type identifier of the demonstration content data frame to be transmitted in the projection data can be encapsulated into a message to obtain the projection service message corresponding to the projection data frame; the frame data and frame type identifier of the demonstration content data frame to be transmitted in the projection data can also be encapsulated in a structure through the sending message function; the structure is encapsulated into a message to obtain the projection service message corresponding to the demonstration content data frame.

[0131] Step 702: Transmitting the screen projection service message through the first queue; for example, transmitting the screen projection service message to the second terminal through the VI queue;

[0132] Step 703: Acquire transmission link quality information of the VI queue; wherein the transmission link quality information includes idle channel assessment parameters, received signal strength and link rate;

[0133] Step 704: determining a link quality evaluation value LQ based on the transmission link quality information; wherein the link quality metric value may be determined according to a metric strategy for the transmission link quality information, and the link quality metric value is used to indicate the degree of influence of the link quality information on the link quality; and determining a link quality evaluation value based on the link quality metric value, and the link quality evaluation value is used to indicate the link quality corresponding to the link quality information;

[0134] For example, the idle channel assessment parameter is 180, the received signal strength is -70dbm, and the link rate is 144Mbps; according to the mapping relationship between the link quality information and the metric value in Tables 1 to 3 above, the metric value of the idle channel assessment parameter is 20, the metric value of the received signal strength is 20, and the metric value of the link rate is 10, and then the link quality assessment value LQ=100-20-20-10=50;

[0135] In this way, by analyzing the transmission link quality information, such as idle channel assessment parameters, received signal strength, link rate, etc., the link quality of the transmission link can be accurately evaluated, thereby providing a theoretical basis for the subsequent determination of the retransmission strategy of the first projection service message;

[0136] Step 705: Determine whether the link quality evaluation value is greater than or equal to a first evaluation value threshold. The first evaluation value threshold can be selected based on an empirical value, such as 60, that is, whether LQ≥60 is established; if established, proceed to step 606, otherwise proceed to step 607; for example, the link quality evaluation value LQ=50 obtained in the above step 704, LQ≥60 is not established, and proceed to step 707;

[0137] Step 706: retransmit the screen projection service message that failed to be transmitted through the second queue, for example, retransmit the screen projection service message that failed to be transmitted through the VO queue; proceed to step 711;

[0138] Step 707: Continue to determine whether the link quality evaluation value is greater than or equal to the second evaluation value threshold. The second evaluation value threshold can be selected based on an empirical value, such as 30, that is, whether 30≤LQ<60 is established; if established, proceed to step 708, otherwise proceed to step 709; for example, the link quality evaluation value LQ obtained in the above step 704 is LQ=50, 30≤LQ<60 is established, and proceed to step 708;

[0139] Step 708: When the first screen projection service message whose frame type indicated by the frame type identifier is an I frame fails to be sent, the first screen projection service message is retransmitted through the VO queue, and the process proceeds to step 711;

[0140] In this way, reliable transmission of key frames of the presentation content data can be ensured when the network condition is not ideal, thereby improving the fluency of the presentation content of the second terminal;

[0141] Step 709: Continue to determine whether the data transmission length in the VI queue exceeds the preset length threshold. The preset length threshold can be selected based on an empirical value. For example, the preset length threshold is 100, that is, LQ < 30, and the data transmission length > 100. If so, proceed to step 610; otherwise, end the process.

[0142] Step 710: When the first screen projection service message whose frame type indicated by the frame type identifier is an I frame fails to be sent, the first screen projection service message is retransmitted through the VO queue, and the second screen projection service message in the VI queue is discarded, the frame type indicated by the frame type identifier included in the second screen projection service message is a P frame or a B frame, and the encoding bit rate of the screen projection data is reduced from the first bit rate to the second bit rate;

[0143] In this way, when the network condition is poor and the amount of data transmitted is large, by discarding P frames and B frames, the reliable transmission of I frames can be ensured, and the problems of presentation content freeze and presentation content loss on the second terminal can be reduced;

[0144] Step 711: Determine whether the number of retransmissions of the first projection service message is greater than the retransmission threshold. The retransmission threshold can be selected based on an empirical value. For example, the retransmission threshold is 1. Determine whether the retransmitted I frame is sent successfully. If the retransmitted I frame fails to be sent, proceed to step 612, otherwise the process ends.

[0145] Step 712: Generate a third screen projection service message in the screen projection data; wherein the third screen projection service message is a screen projection service message corresponding to a screen projection data frame that is adjacent to the presentation content data frame corresponding to the first screen projection service message and whose data frame generation time is later than that of the presentation content data frame, for example, generating the next I frame adjacent to the retransmitted I frame;

[0146] In this way, when the first screen projection service message is retransmitted many times, the next frame can be generated in time, which can reduce the waiting delay caused by the failure of the previous frame transmission and improve the data transmission efficiency.

[0147] The embodiments of the present application take into account multiple technical aspects of the screen projection process, including frame type management, monitoring of transmission link quality information, and optimization of retransmission strategies. It can determine whether to retransmit certain screen projection service messages, especially those containing key frames, in an interference environment, especially in a poor wireless network environment, to ensure that participants can see clear presentation content.

[0148] The screen projection data transmission method provided in the embodiment of the present application can be executed by a screen projection data transmission device. In the embodiment of the present application, the method for executing the screen projection data transmission by the screen projection data transmission device is taken as an example to illustrate the device of the screen projection data transmission method provided in the embodiment of the present application.

[0149] Figure 8 1 is a schematic diagram of a screen projection data transmission device provided in some embodiments of the present application. As shown in the figure, the screen projection data transmission device 800 includes:

[0150] The determination module 810 is used to generate a screen projection service message corresponding to the screen projection data frame according to the frame type of the screen projection data frame to be transmitted in the screen projection data; the screen projection service message includes the frame data of the screen projection data frame and a frame type identifier for indicating the frame type;

[0151] The retransmission module 820 is used to transmit the screen projection service message through the first queue. During the transmission process, the retransmission strategy of the first screen projection service message is determined according to the transmission link quality information of the first queue; the first screen projection service message is a screen projection service message that fails to transmit.

[0152] In a possible implementation, the determining module 810 is further configured to:

[0153] The frame data and frame type identifier of the screen projection data frame to be transmitted in the screen projection data are encapsulated into a message to obtain a screen projection service message corresponding to the screen projection data frame.

[0154] In a possible implementation, the determining module 810 is further configured to:

[0155] By sending a message function, the frame data and frame type identifier of the projection data frame to be transmitted in the projection data are encapsulated in a structure;

[0156] According to the frame type in the structure, the frame data is encapsulated into a message to obtain the screen projection service message corresponding to the screen projection data frame.

[0157] In a possible implementation, the retransmission module 820 is further configured to:

[0158] Determining a link quality metric value according to the measurement strategy for transmitting the link quality information, wherein the link quality metric value is used to indicate the degree of influence of the link quality information on the link quality;

[0159] Determine a link quality evaluation value based on the link quality metric value, where the link quality evaluation value is used to indicate the link quality corresponding to the link quality information;

[0160] Based on the link quality evaluation value, determine the retransmission strategy of the first screen projection service message.

[0161] The link quality information includes: idle channel assessment parameter, received signal strength indicator and link rate;

[0162] The retransmission module 820 is further configured to:

[0163] Determining the link quality assessment value based on the first link quality metric value, the second link quality metric value, and the third link quality metric value;

[0164] The first link quality metric value is a metric value of the idle channel assessment parameter, the second link quality metric value is a metric value of the received signal strength, and the third link quality metric value is a metric value of the link rate.

[0165] In a possible implementation, the retransmission module 820 is further configured to:

[0166] When the link quality evaluation value is greater than or equal to the first evaluation value threshold, retransmitting the first screen projection service message through the second queue;

[0167] The transmission priority of the second queue is higher than the transmission priority of the first queue.

[0168] In a possible implementation, the frame type includes a key frame I frame; the retransmission module 620 is further configured to:

[0169] When the link quality evaluation value is less than the first evaluation value threshold, the link quality evaluation value is greater than or equal to the second evaluation value threshold, and the frame type indicated by the frame type identifier included in the first screen projection service message is an I frame, the first screen projection service message is retransmitted through the second queue;

[0170] The transmission priority of the second queue is higher than the transmission priority of the first queue, and the second evaluation value threshold is lower than the first evaluation value threshold.

[0171] In a possible implementation, the frame types include key frames I frames, predicted frames P frames, and bidirectional predicted frames B frames; the retransmission module 820 is further configured to:

[0172] When the link quality evaluation value is less than the second evaluation value threshold, the data transmission length in the first queue exceeds the preset length threshold, and the frame type indicated by the frame type identifier included in the first screen projection service message is an I frame, the first screen projection service message is retransmitted through the second queue, and the second screen projection service message in the first queue is discarded;

[0173] Among them, the transmission priority of the second queue is higher than the transmission priority of the first queue, and the frame type indicated by the frame type identifier included in the second screen projection service message is a P frame or a B frame.

[0174] In a possible implementation, the retransmission module 820 is further configured to:

[0175] The encoding bit rate of the projection data is reduced from a first bit rate to a second bit rate.

[0176] In a possible implementation, the method further includes:

[0177] A message generation module, used to generate a third screen projection service message in the screen projection data when the number of retransmissions of the first screen projection service message is greater than the retransmission number threshold;

[0178] Among them, the third screen projection service message is a screen projection service message corresponding to a screen projection data frame which is adjacent to the screen projection data frame corresponding to the first screen projection service message and whose data frame generation time is later than that of the screen projection data frame.

[0179] The embodiment of the present application provides a screen projection data transmission device, including a determination module and a retransmission module; the determination module generates a screen projection service message corresponding to the screen projection data frame according to the frame type of the screen projection data frame to be transmitted in the screen projection data; the screen projection service message includes the frame data of the screen projection data frame and the frame type identifier for indicating the frame type; the retransmission module transmits the screen projection service message through a first queue, and during the transmission process, determines the retransmission strategy of the first screen projection service message according to the transmission link quality information of the first queue; the first screen projection service message is a screen projection service message that fails to be transmitted. In this way, by considering multiple technical aspects in the screen projection process, including the management of frame types, the monitoring of transmission link quality information, and the optimization of retransmission strategies, the retransmission delay of the screen projection service message can be reduced in an interference environment, and the screen projection data frame of a specific frame type can be more accurately identified, such as a key frame, to ensure the reliable transmission of the screen projection data frame of a specific frame type, reduce the problems of screen projection audio and video asynchronization, screen distortion, and screen freeze, thereby improving the fluency of the user's screen projection experience.

[0180] The screen projection data transmission device in the embodiment of the present application can be an electronic device or a component in the electronic device, such as an integrated circuit or a chip. The electronic device can be a terminal or other devices other than a terminal. Exemplarily, the electronic device can be a mobile phone, a tablet computer, a laptop computer, a PDA, a vehicle-mounted electronic device, a mobile Internet device (Mobile Internet Device, MID), an augmented reality (augmented reality, AR) / virtual reality (virtual reality, VR) device, a robot, a wearable device, an ultra-mobile personal computer (ultra-mobile personal computer, UMPC), a netbook or a personal digital assistant (personal digital assistant, PDA), etc. It can also be a server, a network attached storage (Network Attached Storage, NAS), a personal computer (personal computer, PC), a television (television, TV), a teller machine or a self-service machine, etc., which is not specifically limited in the embodiment of the present application.

[0181] The screen projection data transmission device in the embodiment of the present application may be a device having an operating system. The operating system may be an Android operating system, an iOS operating system, or other possible operating systems, which are not specifically limited in the embodiment of the present application.

[0182] The screen projection data transmission device provided in the embodiment of the present application can achieve Figures 1 to 7 To avoid repetition, the various processes implemented by the method embodiment are not described here.

[0183] Alternatively, if Fig. 9 As shown, an embodiment of the present application also provides an electronic device 900, including a processor 901 and a memory 902, wherein the memory 902 stores a program or instruction that can be executed on the processor 901, and when the program or instruction is executed by the processor 901, the various steps of the above-mentioned screen projection data transmission method embodiment are implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0184] It should be noted that the electronic devices in the embodiments of the present application include the mobile electronic devices and non-mobile electronic devices mentioned above.

[0185] Fig.10 It is a schematic diagram of the hardware structure of an electronic device provided in some embodiments of the present application.

[0186] The electronic device 1000 includes but is not limited to: a radio frequency unit 1001, a network module 1002, an audio output unit 1003, an input unit 1004, a sensor 1005, a display unit 1006, a user input unit 1007, an interface unit 1008, a memory 1009, and a processor 1010 and other components.

[0187] Those skilled in the art will appreciate that the electronic device 1000 may also include a power source (such as a battery) for supplying power to each component, and the power source may be logically connected to the processor 1010 through a power management system, thereby implementing functions such as managing charging, discharging, and power consumption management through the power management system. Fig.10 The electronic device structure shown in the figure does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently, which will not be described in detail here.

[0188] Among them, the processor 1010 is used to generate a screen projection service message corresponding to the screen projection data frame according to the frame type of the screen projection data frame to be transmitted in the screen projection data; the screen projection service message includes the frame data of the screen projection data frame and a frame type identifier for indicating the frame type;

[0189] The screen projection service message is transmitted through the first queue. During the transmission process, the retransmission strategy of the first screen projection service message is determined according to the transmission link quality information of the first queue; the first screen projection service message is a screen projection service message that fails to transmit.

[0190] In an embodiment of the present application, by considering multiple technical aspects of the screen projection process, including frame type management, monitoring of transmission link quality information, and optimization of retransmission strategies, the retransmission delay of screen projection service messages can be reduced in an interference environment, and screen projection data frames of specific frame types, such as key frames, can be more accurately identified to ensure reliable transmission of screen projection data frames of specific frame types, reduce problems such as asynchrony of audio and video in screen projection, screen distortion, and screen freeze, thereby improving the smoothness of the user's screen projection experience.

[0191] Optionally, the processor 1010, when used to generate a screen projection service message corresponding to the screen projection data frame according to a frame type of the screen projection data frame to be transmitted in the screen projection data, is further used to:

[0192] The frame data and frame type identifier of the screen projection data frame to be transmitted in the screen projection data are encapsulated into a message to obtain a screen projection service message corresponding to the screen projection data frame.

[0193] Optionally, the processor 1010, when used to generate a screen projection service message corresponding to the screen projection data frame according to a frame type of the screen projection data frame to be transmitted in the screen projection data, is further used to:

[0194] By sending a message function, the frame data and frame type identifier of the projection data frame to be transmitted in the projection data are encapsulated in a structure;

[0195] According to the frame type in the structure, the frame data is encapsulated into a message to obtain the screen projection service message corresponding to the screen projection data frame.

[0196] Optionally, the processor 1010 is used to determine a retransmission strategy for the first screen projection service message according to the transmission link quality information of the first queue, and is further used to:

[0197] Determining a link quality metric value according to the measurement strategy for transmitting the link quality information, wherein the link quality metric value is used to indicate the degree of influence of the link quality information on the link quality;

[0198] Determine a link quality evaluation value based on the link quality metric value, where the link quality evaluation value is used to indicate the link quality corresponding to the link quality information;

[0199] Based on the link quality evaluation value, determine the retransmission strategy of the first screen projection service message.

[0200] Optionally, the link quality information includes: an idle channel assessment parameter, a received signal strength, and a link rate; the processor 1010, when used to determine a link quality assessment value based on the link quality metric value, is further used to:

[0201] Determining the link quality assessment value based on the first link quality metric value, the second link quality metric value, and the third link quality metric value;

[0202] The first link quality metric value is a metric value of the idle channel assessment parameter, the second link quality metric value is a metric value of the received signal strength, and the third link quality metric value is a metric value of the link rate.

[0203] Optionally, the processor 1010, when used to determine a retransmission strategy for the first screen projection service message based on the link quality evaluation value, is further used to:

[0204] When the link quality evaluation value is greater than or equal to the first evaluation value threshold, retransmitting the first screen projection service message through the second queue;

[0205] The transmission priority of the second queue is higher than the transmission priority of the first queue.

[0206] Optionally, the frame type includes a key frame I frame; the processor 1010, when used to determine a retransmission strategy of the first screen projection service message based on the link quality evaluation value, is further used to:

[0207] When the link quality evaluation value is less than the first evaluation value threshold, the link quality evaluation value is greater than or equal to the second evaluation value threshold, and the frame type indicated by the frame type identifier included in the first screen projection service message is an I frame, the first screen projection service message is retransmitted through the second queue;

[0208] The transmission priority of the second queue is higher than the transmission priority of the first queue, and the second evaluation value threshold is lower than the first evaluation value threshold.

[0209] Optionally, the frame type includes a key frame I frame, a predicted frame P frame, and a bidirectional predicted frame B frame; the processor 810, when used to determine the retransmission strategy of the first projection service message based on the link quality evaluation value, is further used to:

[0210] When the link quality evaluation value is less than the second evaluation value threshold, the data transmission length in the first queue exceeds the preset length threshold, and the frame type indicated by the frame type identifier included in the first screen projection service message is an I frame, the first screen projection service message is retransmitted through the second queue, and the second screen projection service message in the first queue is discarded;

[0211] Among them, the transmission priority of the second queue is higher than the transmission priority of the first queue, and the frame type indicated by the frame type identifier included in the second screen projection service message is a P frame or a B frame.

[0212] Optionally, after being used to discard the second screen projection service message in the first queue, the processor 1010 is further used to:

[0213] The encoding bit rate of the projection data is reduced from a first bit rate to a second bit rate.

[0214] Optionally, after being used to determine a retransmission strategy for the first projection service message according to the transmission link quality information of the first queue, the processor 1010 is further used to:

[0215] When the number of retransmissions of the first screen projection service message is greater than the retransmission number threshold, generating a third screen projection service message in the screen projection data;

[0216] Among them, the third screen projection service message is a screen projection service message corresponding to a screen projection data frame which is adjacent to the screen projection data frame corresponding to the first screen projection service message and whose data frame generation time is later than that of the screen projection data frame.

[0217] It should be understood that in the embodiment of the present application, the input unit 1004 may include a graphics processor (Graphics Processing Unit, GPU) 10041 and a microphone 10042, and the graphics processor 10041 processes the image data of the static picture or video obtained by the image capture device (such as a camera) in the video capture mode or the image capture mode. The display unit 1006 may include a display panel 10061, and the display panel 10061 may be configured in the form of a liquid crystal display, an organic light emitting diode, etc. The user input unit 1007 includes a touch panel 10071 and at least one of other input devices 10072. The touch panel 10071 is also called a touch screen. The touch panel 10071 may include two parts: a touch detection device and a touch controller. Other input devices 10072 may include, but are not limited to, a physical keyboard, function keys (such as volume control keys, switch keys, etc.), a trackball, a mouse, and a joystick, which will not be repeated here.

[0218] The memory 1009 can be used to store software programs and various data. The memory 1009 may mainly include a first storage area for storing programs or instructions and a second storage area for storing data, wherein the first storage area may store an operating system, an application program or instructions required for at least one function (such as a sound playback function, an image playback function, etc.), etc. In addition, the memory 1009 may include a volatile memory or a non-volatile memory, or the memory 1009 may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), a static random access memory (SRAM), a dynamic random access memory (DRAM), a synchronous dynamic random access memory (SDRAM), a double data rate synchronous dynamic random access memory (DDRSDRAM), an enhanced synchronous dynamic random access memory (ESDRAM), a synchronous link dynamic random access memory (SLDRAM) and a direct memory bus random access memory (DRRAM). The memory 809 in the embodiment of the present application includes but is not limited to these and any other suitable types of memory.

[0219] The processor 1010 may include one or more processing units; optionally, the processor 1010 integrates an application processor and a modem processor, wherein the application processor mainly processes operations related to an operating system, a user interface, and application programs, and the modem processor mainly processes wireless communication signals, such as a baseband processor. It is understandable that the modem processor may not be integrated into the processor 1010.

[0220] An embodiment of the present application also provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, each process of the above-mentioned screen projection data transmission method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0221] The processor is the processor in the electronic device described in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory ROM, a random access memory RAM, a magnetic disk or an optical disk.

[0222] An embodiment of the present application further provides a chip, which includes a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run programs or instructions to implement the various processes of the above-mentioned screen projection data transmission method embodiment, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0223] It should be understood that the chip mentioned in the embodiments of the present application can also be called a system-level chip, a system chip, a chip system or a system-on-chip chip, etc.

[0224] An embodiment of the present application provides a computer program product, which is stored in a storage medium. The program product is executed by at least one processor to implement the various processes of the above-mentioned screen projection data transmission method embodiment, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0225] It should be noted that, in this article, the terms "comprise", "include" or any other variant thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise one..." do not exclude the presence of other identical elements in the process, method, article or device including the element. In addition, it should be pointed out that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in reverse order according to the functions involved, for example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0226] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a computer software product, which is stored in a storage medium (such as ROM / RAM, a disk, or an optical disk), and includes a number of instructions for a terminal (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0227] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.

Claims

1. A screen projection data transmission method, characterized in that: include: Generate a screen projection service message corresponding to the screen projection data frame according to the frame type of the screen projection data frame to be transmitted in the screen projection data; the screen projection service message includes the frame data of the screen projection data frame and a frame type identifier for indicating the frame type; The screen projection service message is transmitted through the first queue. During the transmission process, the retransmission strategy of the first screen projection service message is determined according to the transmission link quality information of the first queue; the first screen projection service message is a screen projection service message that fails to transmit.

2. The method according to claim 1, characterized in that The generating, according to the frame type of the screen projection data frame to be transmitted in the screen projection data, a screen projection service message corresponding to the screen projection data frame comprises: The frame data and frame type identifier of the screen projection data frame to be transmitted in the screen projection data are encapsulated into a message to obtain a screen projection service message corresponding to the screen projection data frame.

3. The method according to claim 1, characterized in that: The generating, according to the frame type of the screen projection data frame to be transmitted in the screen projection data, a screen projection service message corresponding to the screen projection data frame comprises: By sending a message function, the frame data and frame type identifier of the projection data frame to be transmitted in the projection data are encapsulated in a structure; According to the frame type in the structure, the frame data is encapsulated into a message to obtain the screen projection service message corresponding to the screen projection data frame.

4. The method according to claim 1, characterized in that: The determining, according to the transmission link quality information of the first queue, a retransmission strategy of the first projection service message includes: Determining a link quality metric value according to the measurement strategy for transmitting the link quality information, wherein the link quality metric value is used to indicate the degree of influence of the link quality information on the link quality; Determine a link quality evaluation value based on the link quality metric value, where the link quality evaluation value is used to indicate the link quality corresponding to the link quality information; Based on the link quality evaluation value, determine the retransmission strategy of the first screen projection service message.

5. The method according to claim 4, characterized in that The link quality information includes: idle channel assessment parameters, received signal strength and link rate; The determining of a link quality evaluation value based on the link quality metric value comprises: Determining the link quality assessment value based on the first link quality metric value, the second link quality metric value, and the third link quality metric value; The first link quality metric value is a metric value of the idle channel assessment parameter, the second link quality metric value is a metric value of the received signal strength, and the third link quality metric value is a metric value of the link rate.

6. The method according to claim 4, characterized in that The determining, based on the link quality evaluation value, a retransmission strategy for the first screen projection service message includes: When the link quality evaluation value is greater than or equal to the first evaluation value threshold, retransmitting the first screen projection service message through the second queue; The transmission priority of the second queue is higher than the transmission priority of the first queue.

7. The method according to claim 4, characterized in that The frame type includes a key frame I frame; and determining a retransmission strategy of the first screen projection service message based on the link quality evaluation value includes: When the link quality evaluation value is less than the first evaluation value threshold, the link quality evaluation value is greater than or equal to the second evaluation value threshold, and the frame type indicated by the frame type identifier included in the first screen projection service message is an I frame, the first screen projection service message is retransmitted through the second queue; The transmission priority of the second queue is higher than the transmission priority of the first queue, and the second evaluation value threshold is lower than the first evaluation value threshold.

8. The method according to claim 4, characterized in that The frame types include key frames I frames, predicted frames P frames and bidirectional predicted frames B frames; The determining, based on the link quality evaluation value, a retransmission strategy for the first screen projection service message includes: When the link quality evaluation value is less than the second evaluation value threshold, the data transmission length in the first queue exceeds the preset length threshold, and the frame type indicated by the frame type identifier included in the first screen projection service message is an I frame, the first screen projection service message is retransmitted through the second queue, and the second screen projection service message in the first queue is discarded; Among them, the transmission priority of the second queue is higher than the transmission priority of the first queue, and the frame type indicated by the frame type identifier included in the second screen projection service message is a P frame or a B frame.

9. The method according to claim 8, characterized in that After discarding the second screen projection service message in the first queue, the method further includes: The encoding bit rate of the projection data is reduced from a first bit rate to a second bit rate.

10. The method according to any one of claims 1 to 9, characterized in that: After determining a retransmission strategy for the first projection service message according to the transmission link quality information of the first queue, the method further includes: When the number of retransmissions of the first screen projection service message is greater than the retransmission number threshold, generating a third screen projection service message in the screen projection data; Among them, the third screen projection service message is a screen projection service message corresponding to a screen projection data frame which is adjacent to the screen projection data frame corresponding to the first screen projection service message and whose data frame generation time is later than that of the screen projection data frame.

11. A screen projection data transmission device, characterized in that: include: A determination module, used to generate a screen projection service message corresponding to the screen projection data frame according to the frame type of the screen projection data frame to be transmitted in the screen projection data; the screen projection service message includes the frame data of the screen projection data frame and a frame type identifier for indicating the frame type; The retransmission module is used to transmit the screen projection service message through the first queue. During the transmission process, the retransmission strategy of the first screen projection service message is determined according to the transmission link quality information of the first queue; the first screen projection service message is a screen projection service message that fails to transmit.

12. The device according to claim 11, characterized in that The determining module is further used for: The frame data and frame type identifier of the screen projection data frame to be transmitted in the screen projection data are encapsulated into a message to obtain a screen projection service message corresponding to the screen projection data frame.

13. The device according to claim 11, characterized in that The determining module is further used for: By sending a message function, the frame data and frame type identifier of the projection data frame to be transmitted in the projection data are encapsulated in a structure; According to the frame type in the structure, the frame data is encapsulated into a message to obtain the screen projection service message corresponding to the screen projection data frame.

14. The device according to claim 11, characterized in that The retransmission module is further used for: Determining a link quality metric value according to the measurement strategy for transmitting the link quality information, wherein the link quality metric value is used to indicate the degree of influence of the link quality information on the link quality; Determine a link quality evaluation value based on the link quality metric value, where the link quality evaluation value is used to indicate the link quality corresponding to the link quality information; Based on the link quality evaluation value, determine the retransmission strategy of the first screen projection service message.

15. The device according to claim 14, characterized in that The link quality information includes: idle channel assessment parameters, received signal strength indication and link rate; The retransmission module is further used for: Determining the link quality assessment value based on the first link quality metric value, the second link quality metric value, and the third link quality metric value; The first link quality metric value is a metric value of the idle channel assessment parameter, the second link quality metric value is a metric value of the received signal strength, and the third link quality metric value is a metric value of the link rate.

16. The device according to claim 14, characterized in that The retransmission module is further used for: When the link quality evaluation value is greater than or equal to the first evaluation value threshold, retransmitting the first screen projection service message through the second queue; The transmission priority of the second queue is higher than the transmission priority of the first queue.

17. The device according to claim 14, characterized in that The frame type includes a key frame I frame; the retransmission module is further used for: When the link quality evaluation value is less than the first evaluation value threshold, the link quality evaluation value is greater than or equal to the second evaluation value threshold, and the frame type indicated by the frame type identifier included in the first screen projection service message is an I frame, the first screen projection service message is retransmitted through the second queue; The transmission priority of the second queue is higher than the transmission priority of the first queue, and the second evaluation value threshold is lower than the first evaluation value threshold.

18. The device according to claim 14, characterized in that The frame types include key frames I frames, predicted frames P frames and bidirectional predicted frames B frames; the retransmission module is further used for: When the link quality evaluation value is less than the second evaluation value threshold, the data transmission length in the first queue exceeds the preset length threshold, and the frame type indicated by the frame type identifier included in the first screen projection service message is an I frame, the first screen projection service message is retransmitted through the second queue, and the second screen projection service message in the first queue is discarded; Among them, the transmission priority of the second queue is higher than the transmission priority of the first queue, and the frame type indicated by the frame type identifier included in the second screen projection service message is a P frame or a B frame.

19. The device according to claim 18, characterized in that The retransmission module is further used for: The encoding bit rate of the projection data is reduced from a first bit rate to a second bit rate.

20. The device according to any one of claims 10 to 19, characterized in that Also includes: A message generation module, used to generate a third screen projection service message in the screen projection data when the number of retransmissions of the first screen projection service message is greater than the retransmission number threshold; Among them, the third screen projection service message is a screen projection service message corresponding to a screen projection data frame which is adjacent to the screen projection data frame corresponding to the first screen projection service message and whose data frame generation time is later than that of the screen projection data frame.

21. An electronic device, characterized in that: It includes a processor and a memory, the memory stores programs or instructions that can be run on the processor, and when the program or instructions are executed by the processor, the steps of the screen projection data transmission method as described in any one of claims 1-10 are implemented.

Images