Information transmission method and device, display equipment and computer readable storage medium

Abstract

The embodiment of the invention discloses an information transmission method, and the method comprises the steps: initiating a session establishment request to a source device under the condition that the source device and a display device establish a device connection through a first protocol; wherein the source equipment and the display equipment are accessed to the same wireless network; aiming at the session establishment request, establishing session connection between the source equipment and the display equipment based on types and fields of various source equipment added in a second protocol; after the session connection is established, monitoring packet loss data of media data transmitted to the display equipment by the source equipment to obtain a monitoring result, and adjusting a to-be-adjusted transmission parameter for transmitting the media data based on the monitoring result to obtain a target transmission parameter; and receiving the media data based on the target transmission parameter. The embodiment of the invention further discloses an information transmission device, display equipment and a computer readable storage medium.

Description

Technical Field

[0001] The present application relates to the technical field of Internet of Things, and in particular to an information transmission method, apparatus, display device and computer-readable storage medium. Background Art

[0002] With the development of science and technology, screen projection technology has also developed accordingly, and screen projection technology refers to projecting small screens such as smartphones, tablets and laptops onto the large screen of a TV; however, in the related technologies, for the screen projection technology of the wireless display standard (Miracast) protocol, many screen projection products can only achieve one-to-one screen projection, and cannot support multi-channel projection to the same large screen, which limits the diversity and flexibility of screen projection applications, and screen projection products have screen distortion and freeze problems during use, thereby reducing the screen projection effect and user experience. Summary of the invention

[0003] In order to solve the above technical problems, the embodiments of the present application hope to provide an information transmission method, apparatus, display device and computer-readable storage medium, which can solve the problem that the related technology cannot support multi-channel screen projection to the same large screen and there are screen distortion and jamming during use.

[0004] The technical solution of this application is implemented as follows:

[0005] An information transmission method, applied to a display device, comprising:

[0006] Initiating a session establishment request to the source device when the source device and the display device establish a device connection through a first protocol; wherein the source device and the display device are connected to the same wireless network;

[0007] In response to the session establishment request, based on the types and fields of multiple source devices added in the second protocol, establish a session connection between the source device and the display device;

[0008] After the session connection is established, packet loss data of the media data transmitted from the source device to the display device is monitored to obtain a monitoring result, and based on the monitoring result, a transmission parameter to be adjusted for transmitting the media data is adjusted to obtain a target transmission parameter;

[0009] The media data is received based on the target transmission parameters.

[0010] In the above scheme, the packet loss data of the media data transmitted from the source device to the display device is monitored to obtain a monitoring result, and the transmission parameter to be adjusted for transmitting the media data is adjusted based on the monitoring result to obtain a target transmission parameter, including:

[0011] Determine a lost data packet in the media data based on a first sequence number of a current data packet in the media data and a second sequence number of a target data packet; wherein the media data includes a plurality of data packets; and the target data packet is a data packet preceding the current data packet;

[0012] For each of the lost data packets, if the number of retransmissions of the lost data packet meets a quantity threshold, the lost data packet is deleted; wherein the packet loss data includes the number of retransmissions of the lost data packet, and the monitoring result includes that the number of retransmissions meets the quantity threshold;

[0013] Adjusting the transmission parameter to be adjusted to obtain the target transmission parameter;

[0014] For each of the lost data packets, if the number of retransmissions of the lost data packet does not meet a quantity threshold, the lost data packet is sent to a buffer management channel; wherein the buffer management channel is a channel established between the source device and the display device.

[0015] In the above solution, the step of determining the lost data packet in the media data based on the first sequence number of the current data packet in the media data and the second sequence number of the target data packet includes:

[0016] Calculate the first sequence number and the second sequence number to obtain a sequence interval between the current data packet and the target data packet;

[0017] The lost data packet is determined based on the sequence interval and the storage quantity of the data packets stored in the buffer management channel.

[0018] In the above solution, the determining the lost data packet based on the sequence interval and the storage quantity of the data packets stored in the buffer management channel includes:

[0019] Comparing the sequence interval with the stored quantity to obtain a comparison result;

[0020] If the comparison result is that the sequence interval is less than or equal to the stored number, and the sequence interval is greater than a first value, it is determined that the lost data packets are all data packets between the first sequence number and the second sequence number.

[0021] In the above scheme, the information transmission method also includes:

[0022] If the comparison result is that the sequence interval is greater than the storage quantity, deleting the data packet before the target data packet from the buffer management channel;

[0023] If the sequence interval is equal to the first value, sending the current data packet to the buffer management channel;

[0024] If the sequence interval is smaller than a second value, the current data packet is deleted from the buffer management channel; wherein the second value is smaller than the first value.

[0025] In the above scheme, the information transmission method also includes:

[0026] receiving a data packet transmitted by a source device to the buffer management channel through a transmission channel based on the target transmission parameter;

[0027] The data packets are rearranged according to the sequence numbers of the data packets through the buffer management channel to obtain target media data.

[0028] In the above scheme, the information transmission method also includes:

[0029] Establishing a mouse acquisition channel for the mouse content of the source device;

[0030] The mouse content in the media data is collected through the mouse collection channel; wherein the mouse content is the changing content of the mouse when the mouse is operated.

[0031] In the above scheme, the information transmission method also includes:

[0032] Establish a reverse control channel for user input;

[0033] If the display device receives a manipulation operation instruction, the manipulation operation instruction is sent to the source device through the user input reverse control channel, so that the source device manipulates the media data based on the manipulation operation instruction.

[0034] An information transmission device, comprising:

[0035] an initiating unit, configured to initiate a session establishment request to the source device when the source device and the display device establish a device connection through a first protocol; wherein the source device and the display device are connected to the same wireless network;

[0036] A session unit, configured to establish a session connection between the source device and the display device based on the types and fields of the multiple source devices added in the second protocol in response to the session establishment request;

[0037] a processing unit, configured to, after establishing the session connection, monitor packet loss data of the media data transmitted from the source device to the display device to obtain a monitoring result, and adjust a transmission parameter to be adjusted for transmitting the media data based on the monitoring result to obtain a target transmission parameter;

[0038] A receiving unit is used to receive the media data based on the target transmission parameter.

[0039] A display device, the device comprising: a processor, a memory and a communication bus;

[0040] The communication bus is used to realize the communication connection between the processor and the memory;

[0041] The processor is used to execute the information transmission program in the memory to implement the steps of the above-mentioned information transmission method.

[0042] A computer-readable storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the steps of the above-mentioned information transmission method.

[0043] The information transmission method, apparatus, display device and computer-readable storage medium provided in the embodiments of the present application first initiate a session establishment request to the source device when the source device and the display device establish a device connection through a first protocol; and the source device and the display device are connected to the same wireless network, and then, in response to the session establishment request, a session connection between the source device and the display device is established based on the types and fields of multiple source devices added in a second protocol, and then after the session connection is established, the packet loss data of the media data transmitted from the source device to the display device is monitored to obtain a monitoring result, and the transmission parameters to be adjusted for the transmission of the media data are adjusted based on the monitoring result to obtain the target transmission parameters, and then the media data is received based on the target transmission parameters, so that by A first protocol is established between the source device and the display device, so that the source device can be connected to the display device to realize the simultaneous sharing of multi-channel / multi-screen / multi-content information, and by adding multiple source device types and fields in the second protocol, the display device can support more source devices, thereby supporting more source devices to successfully establish session connections with the display device, solving Miracast compatibility issues, and at the same time, by monitoring packet loss data and adjusting the transmission parameters to be adjusted for the transmitted media data based on the monitoring results, the video freeze and screen distortion caused by data packet loss during the transmission of media data are greatly reduced, ensuring the transmission stability and smoothness of media data, thereby not only improving the video projection effect but also improving the user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 A flowchart of an information transmission method provided in an embodiment of the present application;

[0045] Figure 2 A schematic diagram of the collaborative cooperation of multiple channels in an information transmission method provided in an embodiment of the present application;

[0046] Figure 3 A flowchart of another information transmission method provided in an embodiment of the present application;

[0047] Figure 4 A schematic diagram of determining lost data packets in an information transmission method provided in an embodiment of the present application;

[0048] Figure 5 A schematic diagram of adding a delay in an information transmission method provided in an embodiment of the present application;

[0049] Figure 6 A schematic diagram of the working principle of a buffer management channel in an information transmission method provided in an embodiment of the present application;

[0050] Figure 7 A schematic diagram of the structure of an information transmission device provided in an embodiment of the present application;

[0051] Figure 8 A schematic diagram of the structure of an information transmission device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0052] The technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application.

[0053] It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0054] Before further describing the embodiments of the present application in detail, the nouns and terms involved in the embodiments of the present application are described first. The nouns and terms involved in the embodiments of the present application are subject to the following interpretations:

[0055] 1. Wireless Display Standard (Miracast): It is a technical standard (protocol) for wireless video and audio transmission between different devices. It allows users to share content between devices that support Miracast through Wi-Fi Direct (Wi-Fi Direct), such as screen mirroring or video streaming between mobile phones, tablets and TVs.

[0056] 2. Real-time Transport Protocol (RTP): It is a standardized protocol for transmitting audio and video data on IP networks. RTP is usually used together with Real-time Transport Control Protocol (RTCP), where RTP is responsible for transmitting media data and RTCP is responsible for transmitting related control information and feedback. RTP is often used in scenarios where applications need to transmit audio, video or multimedia data in real time, such as Internet telephony, video conferencing and streaming media. It can provide information such as timestamp, sequence number (seq num), source endpoint identifier and destination endpoint identifier to ensure orderly playback and synchronization of real-time streams.

[0057] 3. RTCP: It is a control protocol used in conjunction with RTP, and RTCP is designed to provide control, monitoring, and feedback information related to RTP to ensure the quality and stability of real-time multimedia sessions.

[0058] 4. Jitter buffer: It is a mechanism used in real-time transport protocols (such as RTP) to address the impact of delay changes (jitter) introduced in the network on audio or video quality.

[0059] 5. User Input Back Channel (UIBC): It is a function used in the Wi-Fi Alliance Miracast standard that allows users to control the content display on the source device (such as a smartphone, tablet) through a display device connected to Wi-Fi Direct.

[0060] 6. Peer-to-Peer (P2P): It is a computer network communication model in which participants can communicate and share resources directly without going through a centralized server or intermediate node.

[0061] 7. GO: Go groups are usually used in Wi-Fi Direct technology. In Wi-Fi Direct, devices (including source devices and display devices) can communicate directly through P2P connections, and the P2P Go group is a special P2P network in which one device acts as the owner of the group to manage and control the distribution of data traffic and connections.

[0062] It should be noted that the following problems exist in the existing technology based on Miracast wireless screen projection: 1. One-to-one screen projection limitation: Many screen projection products can only realize one-to-one screen projection, and cannot support multi-channel screen projection to the same large screen, which limits the diversity and flexibility of screen projection applications. 2. Connection stability: The screen projection product has unstable connection during use, which may cause problems such as screen projection interruption and picture delay, affecting the user experience. 3. Reverse control operation limitation: Some screen projection products cannot realize reverse control operations on the projected content, such as being unable to use the screen projection device for content control or interactive operations, which limits the user's operating experience. 4. Standard uniformity: There are differences in the implementation methods of screen projection products between different manufacturers, and the lack of unified standards may lead to compatibility issues, making the screen projection connection between different devices unstable or impossible to achieve.

[0063] Based on this, the embodiment of the present application provides an information transmission method, which can be applied to a display device, referring to Figure 1 As shown, the method comprises the following steps:

[0064] Step 101: When a source device and a display device establish a device connection through a first protocol, a session establishment request is initiated to the source device.

[0065] The source device and the display device are connected to the same wireless network.

[0066] In an embodiment of the present application, the source device, also known as the transmitting end, may specifically include a smart phone, a tablet computer and a laptop computer, and the source device may be an Android system or a Windows system; the display device, also known as the receiving end, may specifically include a television, and both the source device and the display device are devices that support the Miracast protocol; the first protocol may be a distributed network protocol, and may specifically refer to a P2P protocol, in which participants (i.e., the source device and the display device) may communicate and share resources directly without going through a centralized server or intermediate node, that is, the content in the source device may be projected to the display device through the P2P protocol; through the P2P protocol, a display device may establish device connections with multiple different source devices at the same time, that is, a smart phone, a tablet computer and a laptop computer may be connected to the TV at the same time, so that multi-channel / multi-screen / multi-content information may be shared simultaneously, and native protocol and private protocol hybrid connections may be supported, and the same P2P interface may be shared at the network link layer, thereby reducing the instability caused by wireless network card channel switching and time-division multiplexing.

[0067] In the embodiment of the present application, the session establishment request may be used to request to establish a session connection between a source device and a display device; Figure 2As shown, after the source device establishes a P2P connection with the display device, the display device may initiate a session establishment request to the source device to establish a session connection with the source device.

[0068] Step 102: In response to the session establishment request, a session connection is established between the source device and the display device based on the types and fields of the multiple source devices added in the second protocol.

[0069] In the embodiment of the present application, the second protocol is a protocol that can realize real-time streaming media transmission between devices, and specifically refers to the Real Time Streaming Protocol (RTSP); Figure 2 As shown, multiple source device types and fields can be added to the device support list during the RTSP session negotiation between the source device and the display device, so that more source devices can be identified during the RTSP session negotiation, that is, by optimizing the RTSP session negotiation process in the Miracast protocol, it supports the transmission of more device types and more field information, thereby improving compatibility and connectivity, ensuring stable and reliable connections and functional interoperability between different devices, thereby ensuring that a session connection can be successfully established between the source device and the display device, and solving the compatibility problem between the source device and the display device.

[0070] Step 103: After the session connection is established, the packet loss data of the media data transmitted from the source device to the display device is monitored to obtain a monitoring result, and based on the monitoring result, the transmission parameters to be adjusted for transmitting the media data are adjusted to obtain target transmission parameters.

[0071] Step 104: Receive media data based on the target transmission parameters.

[0072] In an embodiment of the present application, media data may include video streams; packet loss data may include the number of retransmissions of each lost data packet in the media data; the transmission parameters to be adjusted may include the transmission rate of the transmitted media data, and the target transmission parameters may refer to the transmission parameters after the transmission parameters to be adjusted are adjusted based on the monitoring results, and the target transmission parameters may also include the transmission rate; the monitoring results may include the number of retransmissions satisfying a quantity threshold and the number of retransmissions not satisfying a quantity threshold, and the number of retransmissions satisfying the quantity threshold indicates that the quality of the transmission parameters to be adjusted for the transmitted media data is not high, and therefore the transmission parameters to be adjusted may be adjusted.

[0073] In an embodiment of the present application, after a session connection is established between the source device and the display device, an RTCPNACK link can be established, and the number of retransmissions of each lost data packet in the media data transmitted from the source device to the display device can be monitored through the RTCP NACK link. In this way, the data quality of the screen projection transmission is monitored by using the RTCP channel for real-time control protocol, and the transmission parameters to be adjusted are adjusted in time to improve the video screen projection effect and ensure stable and smooth transmission. That is, the NACK data packet statistics of RTCP are used to greatly reduce the impact of video freezes and screen distortion caused by packet loss during data transmission.

[0074] The information transmission method provided in the embodiment of the present application establishes a first protocol between the source device and the display device so that the source device can be connected to the display device to realize simultaneous sharing of multi-channel / multi-screen / multi-content information, and by adding multiple source device types and fields in the second protocol, the display device can support more source devices, thereby supporting more source devices to successfully establish session connections with the display device, solving Miracast compatibility issues, and at the same time, by monitoring packet loss data and adjusting the transmission parameters to be adjusted for transmitting media data based on the monitoring results, video freezes and screen distortion caused by data packet loss during the transmission of media data are greatly reduced, ensuring the transmission stability and smoothness of media data, thereby not only improving the video projection effect but also improving the user experience.

[0075] Based on the above embodiments, the present application provides another information transmission method. Figure 3 As shown, the method may include the following steps:

[0076] Step 201: When a source device and the display device establish a device connection through a first protocol, the display device initiates a session establishment request to the source device.

[0077] The source device and the display device are connected to the same wireless network.

[0078] Step 202: The display device establishes a session connection between the source device and the display device in response to the session establishment request and based on the types and fields of the multiple source devices added in the second protocol.

[0079] Step 203: After establishing the session connection, the display device determines the lost data packet in the media data based on the first sequence number of the current data packet in the media data and the second sequence number of the target data packet.

[0080] The media data includes multiple data packets; the target data packet is the previous data packet of the current data packet.

[0081] In an embodiment of the present application, the sequence number (i.e., sequence number, seq num) of each data packet in the media data can be obtained from the RTP protocol, and the data packet can specifically refer to the RTP data packet, the sequence number can refer to the RTP sequence number, the first sequence number can refer to the sequence number of the current data packet, the second sequence number can refer to the sequence number of the target data packet, and the target data packet can refer to the data packet adjacent to the current data packet and before the current data packet. Since the sequence numbers of the data packets can be sorted in ascending order according to the order in which they are received, the second sequence number can refer to the largest sequence number before the current data packet is received, and the target data packet refers to the data packet corresponding to the largest sequence number.

[0082] In an embodiment of the present application, the first sequence number and the second sequence number may be operated, and the lost data packet in the media data may be determined according to the operation result.

[0083] It should be noted that step 203 can be implemented by step 203A and step 203B. Figure 4 As shown:

[0084] Step 203A: The display device calculates the first sequence number and the second sequence number to obtain a sequence interval between the current data packet and the target data packet.

[0085] In an embodiment of the present application, a subtraction operation can be performed on the first sequence number and the second sequence number to calculate the sequence interval between the current data packet and the target data packet, and then the lost data packet can be determined based on the sequence interval and the number of data packets stored in the buffer management channel (i.e., Jitterbuf); it should be noted that the sequence interval can be represented by diff.

[0086] Step 203B: The display device determines the lost data packets based on the sequence interval and the storage quantity of the data packets stored in the buffer management channel.

[0087] In an embodiment of the present application, the storage quantity of the data packets stored in the buffer management channel can be determined first, and then the lost data packets can be determined based on the sequence interval and the storage quantity of the buffer management channel. In a feasible implementation, the storage quantity can be 100, and of course, the storage quantity can also be other values.

[0088] It should be noted that step 203B can be implemented in the following ways:

[0089] Step 203b1: The display device compares the sequence interval with the stored quantity to obtain a comparison result.

[0090] In the embodiment of the present application, the sequence interval may be compared with the storage quantity to obtain that the sequence interval is greater than the storage quantity, or the sequence interval is less than the storage quantity, or the sequence interval is equal to the storage quantity.

[0091] Step 203b2: If the comparison result of the information transmission device is that the sequence interval is less than or equal to the storage quantity and the sequence interval is greater than the first value, the information transmission device determines that the lost data packets are all data packets between the first sequence number and the second sequence number.

[0092] In an embodiment of the present application, when the sequence interval is greater than the first value and less than or equal to the stored number, it is determined that diff data packets are lost in the middle, and then all data packets between the second sequence number and the first sequence number can be added to the map for storing lost packets, and the second sequence number can be reset to the first sequence number.

[0093] In a feasible implementation, the first value may be 1. Then, when the first value is 1 and the storage quantity is 100, the sequence interval is less than or equal to the storage quantity and the sequence interval is greater than the first value, which can be represented as 1. <diff<=100。

[0094] It should be noted that after step 203A, steps 203C to 203E may also be performed. Figure 4 As shown:

[0095] Step 203C: If the comparison result is that the sequence interval is greater than the storage quantity, the display device deletes the data packet before the target data packet from the buffer management channel.

[0096] Step 203D: If the sequence interval is equal to the first value, the display device sends the current data packet to the buffer management channel.

[0097] Step 203E: If the sequence interval is less than the second value, the display device deletes the current data packet from the buffer management channel.

[0098] The second value is smaller than the first value.

[0099] In an embodiment of the present application, if the comparison result is that the sequence interval is greater than the storage number (i.e., diff>100), it means that the sequence interval of the lost data packet is too large. At this time, there is no need to retransmit the current data packet, and all data packets before the target data packet can be deleted from the map storing lost packets in the buffer management channel, and the second sequence number can be reset to the first sequence number; if the sequence interval is equal to the first value, it means that the data packet is transmitted normally and all data packets can arrive at the buffer management channel in an orderly manner. At this time, the current data packet can be sent normally and the second sequence number can be reset to the first sequence number; if the sequence interval is less than the second value (the second value is 0), it means that the data packet that should have been received earlier was received late, so the current data packet stored in the map storing lost packets in the buffer management channel can be directly deleted.

[0100] It should be noted that steps 204 to 205 may be performed after step 203, and step 206 may also be performed after step 203;

[0101] Step 204: The display device deletes each lost data packet if the number of retransmissions of the lost data packet meets the quantity threshold.

[0102] The packet loss data includes the number of retransmissions of the lost data packets, and the monitoring result includes whether the number of retransmissions meets the quantity threshold.

[0103] In the embodiment of the present application, the quantity threshold may be a value determined based on historical experimental data; the number of retransmissions satisfying the quantity threshold may mean that the number of retransmissions is greater than or equal to the quantity threshold; Figure 4 As shown, after the lost data packet is determined according to the size relationship between the sequence interval and the storage quantity, for each lost data packet in the entire media data, a counter can be used to count the number of retransmissions of each lost data packet to obtain a counting result (i.e., the number of retransmissions), and then the number of retransmissions and the number threshold can be compared. If the number of retransmissions of a lost data packet is greater than or equal to the number threshold, the lost data packet can be deleted from the map storing lost packets in the buffer management channel. That is to say, when the number of retransmissions is greater than or equal to the number threshold, the lost data packet will not be retransmitted.

[0104] In a feasible implementation, the quantity threshold may be set to 10.

[0105] Step 205: The display device adjusts the transmission parameter to be adjusted to obtain the target transmission parameter.

[0106] Step 206: The display device adjusts the transmission parameters to be adjusted of the transmission media data based on the monitoring result to obtain the target transmission parameters.

[0107] Step 207: The display device receives media data based on the target transmission parameters.

[0108] Step 208: For each lost data packet, if the number of retransmissions of the lost data packet does not meet the quantity threshold, the display device sends the lost data packet to the buffer management channel.

[0109] The buffer management channel is a channel established between the source device and the display device.

[0110] In an embodiment of the present application, the number of retransmissions not satisfying the quantity threshold may refer to the number of retransmissions being less than the quantity threshold, that is, if the number of retransmissions of the lost data packet is less than the quantity threshold, the lost data packet is sent to the buffer management channel, that is, when the number of retransmissions is less than the quantity threshold, the retransmission of the lost data packet is triggered; specifically, after the lost data packet is determined, the RTP sequence number of the lost data packet may be obtained, and multiple NACK messages may be created according to the RTP sequence number, and when the number of retransmissions of the lost data packet is less than the quantity threshold, the NACK message containing the lost data packet is sent to the buffer management channel.

[0111] It should be noted that if the number of retransmissions of a lost data packet is greater than or equal to the quantity threshold, the lost data packet will not be encapsulated into the created NACK message.

[0112] It should be noted that the above embodiment may further include the following steps:

[0113] Step 209: The display device receives the data packet transmitted by the source device to the buffer management channel through the transmission channel based on the target transmission parameter.

[0114] Step 210: The display device rearranges the data packets according to the sequence numbers of the data packets through the buffer management channel to obtain target media data.

[0115] In the embodiment of the present application, the target media data may refer to data composed of data packets obtained by rearranging successive data packets using a buffer management channel, that is, the data packets in the target media data are data packets arranged in a certain order of sequence numbers; the transmission channel may refer to a user datagram protocol (User Datagram Protocol, UDP); Figure 2As shown in the figure, after receiving the data packet based on the target transmission parameters, the data packet is arranged in a disordered order and can be rearranged in a disordered order through the buffer management channel. In this way, the inter-frame delay (jitterdelay) is increased by rearranging the continuous data packets (i.e., adding a delay due to jitter), thereby providing weak network resistance capability. Through the Jitter buf buffer management mechanism, the system can better cope with network fluctuations and packet loss, and keep the projection picture smooth when the packet loss rate is <30%, the Miracast connection is uninterrupted for a long time, and there will be no flowery screen or black screen phenomenon.

[0116] It should be noted that there will be different delays when sending media data from the source device to the display device. Such delay changes are called jitter, and jitter will make the audio and video playback unstable, such as audio tremors and video speedups. Figure 5 As shown, the inter-frame delay can be increased to combat jitter, that is, a delay caused by jitter, namely, jitter delay, can be added.

[0117] It should be noted that the above embodiment may further include the following steps:

[0118] Step 211: The display device establishes a mouse acquisition channel for the mouse content of the source device.

[0119] Step 212: The display device collects mouse content in the media data through the mouse collection channel.

[0120] The mouse content refers to the changing content of the mouse when the mouse is operated.

[0121] In the embodiment of the present application, the mouse content may include the size of the mouse, the shape of the mouse when performing a click event, the coordinates of the mouse movement and the mouse click event, etc. Microsoft officially supports the mouse independent acquisition channel in the win10 version, but some wireless screen projection manufacturers did not follow up in time, which reduced the transmission efficiency of the mouse and also reduced the user experience. Therefore, Figure 2 As shown, by establishing an independent mouse acquisition channel, which is only used to separately acquire and transmit mouse content, the way users operate the device is simplified, the convenience of operation is improved, and thus the user experience is enhanced. In addition, by establishing an independent mouse acquisition channel, conflicts and delays that may occur during multi-channel data transmission can be avoided, thereby ensuring the accuracy and real-time nature of user operations.

[0122] It should be noted that the above embodiment may further include the following steps:

[0123] Step 213: The display device establishes a user input reverse control channel.

[0124] Step 214: If the display device receives the manipulation operation instruction, the display device sends the manipulation operation instruction to the source device through the user input reverse control channel, so that the source device manipulates the media data based on the manipulation operation instruction.

[0125] In the embodiments of the present application, some screen projection products cannot realize reverse control operations on the projected content, such as being unable to use the screen projection device to control the content or perform interactive operations, which limits the user's operating experience. Figure 2 As shown, by establishing a user input reverse control channel (UIBC), that is, introducing UIBC, users are allowed to control and interact with the playback content through supported devices during the screen projection process, thereby improving the user experience and increasing the diversity of screen projection functions.

[0126] In other embodiments of the present application, the process of Jitterbuf processing the received RTP data packet is as follows: Figure 6As shown, the details are as follows: 1. When a video packet is received for the first time, an empty frame block is popped from the freeframes queue to place the packet. 2. After that, each time an RTP packet is received, it is searched in incompleteframes and decodableframes according to the timestamp to see if a packet with the same timestamp has been received. If found, the frame block is popped from incompleteframes or decodableframes, otherwise, an empty frame is popped from freeframes. 3. According to the sequence number of the packet, the position where the frame should be inserted is found and the state is updated. The state includes empty, incomplete, decodable and complete. Empty is a state with no data, incomplete is a state with at least one packet, decodable is a decodable state, and complete is a state where all data of this frame has arrived. decodable will have different rules according to decode_error_mode. Different QOS strategies will set different decode_error_mode, including kNoErrors, kSelectiveErrors and kWithErrors. decode_error_mode determines whether the frame taken out of the buffer by the decoding thread contains errors, that is, whether the current frame has packet loss. 4. Push the frame back to the queue according to the different states. When the state is incomplete, push it to the incompleteframes queue, and push the frames in decodable and complete states back to the decodableframes queue. 5. The freeframes queue has an initial size. When the freeframes queue is empty, the queue size will increase, but there is a maximum value. Some outdated frames will also be periodically cleared from the incompleteframes and decodableframes queues and pushed to the freeframes queue. 6. The decoding thread takes out the frame, and after decoding is completed, pushes it back to the freeframes queue.

[0127] It should be noted that, for the description of the same steps and the same contents in this embodiment as those in other embodiments, reference can be made to the description in other embodiments and will not be repeated here.

[0128] The information transmission method provided in the embodiment of the present application establishes a first protocol between the source device and the display device so that the source device can be connected to the display device to realize simultaneous sharing of multi-channel / multi-screen / multi-content information, and by adding multiple source device types and fields in the second protocol, the display device can support more source devices, thereby supporting more source devices to successfully establish session connections with the display device, solving Miracast compatibility issues, and at the same time, by monitoring packet loss data and adjusting the transmission parameters to be adjusted for transmitting media data based on the monitoring results, video freezes and screen distortion caused by data packet loss during the transmission of media data are greatly reduced, ensuring the transmission stability and smoothness of media data, thereby not only improving the video projection effect but also improving the user experience.

[0129] Based on the above embodiments, the present invention provides an information transmission device, which can be applied to Figure 1 and Figure 3 In the information transmission method provided in the corresponding embodiment, refer to Figure 7 As shown, the information transmission device 3 may include: an initiating unit 31, a conversation unit 32, a processing unit 33 and a receiving unit 34, wherein:

[0130] The initiating unit 31 is used to initiate a session establishment request to the source device when the source device and the display device establish a device connection through the first protocol; wherein the source device and the display device are connected to the same wireless network;

[0131] The session unit 32 is used to establish a session connection between the source device and the display device based on the types and fields of the multiple source devices added in the second protocol in response to the session establishment request;

[0132] The processing unit 33 is used to monitor the packet loss data of the media data transmitted from the source device to the display device to obtain a monitoring result after the session connection is established, and adjust the transmission parameters to be adjusted of the transmitted media data based on the monitoring result to obtain the target transmission parameters;

[0133] The receiving unit 34 is configured to receive media data based on the target transmission parameters.

[0134] In other embodiments of the present application, the processing unit 33 is further configured to perform the following steps:

[0135] Determine a lost data packet in the media data based on a first sequence number of a current data packet in the media data and a second sequence number of a target data packet; wherein the media data includes a plurality of data packets; and the target data packet is a data packet preceding the current data packet;

[0136] For each lost data packet, if the number of retransmissions of the lost data packet meets the quantity threshold, the lost data packet is deleted; wherein the packet loss data includes the number of retransmissions of the lost data packet, and the monitoring result includes the number of retransmissions meeting the quantity threshold;

[0137] Adjust the transmission parameters to be adjusted to obtain target transmission parameters;

[0138] For each lost data packet, if the number of retransmissions of the lost data packet does not meet the quantity threshold, the lost data packet is sent to a buffer management channel; wherein the buffer management channel is a channel established between the source device and the display device.

[0139] In other embodiments of the present application, the processing unit 33 is further configured to perform the following steps:

[0140] Calculate the first sequence number and the second sequence number to obtain the sequence interval between the current data packet and the target data packet;

[0141] Lost packets are determined based on the sequence interval and the number of packets stored in the buffer management channel.

[0142] In other embodiments of the present application, the processing unit 33 is further configured to perform the following steps:

[0143] Compare the sequence interval with the stored quantity to obtain a comparison result;

[0144] If the comparison result is that the sequence interval is less than or equal to the stored number, and the sequence interval is greater than the first value, it is determined that the lost data packets are all data packets between the first sequence number and the second sequence number.

[0145] In other embodiments of the present application, the processing unit 33 is further configured to perform the following steps:

[0146] If the comparison result is that the sequence interval is greater than the storage quantity, the data packet before the target data packet is deleted from the buffer management channel;

[0147] If the sequence interval is equal to the first value, sending the current data packet to the buffer management channel;

[0148] If the sequence interval is smaller than a second value, the current data packet is deleted from the buffer management channel; wherein the second value is smaller than the first value.

[0149] In other embodiments of the present application, the processing unit 33 is further configured to perform the following steps:

[0150] receiving a data packet transmitted by a source device to a buffer management channel through a transmission channel based on a target transmission parameter;

[0151] The data packets are rearranged according to the sequence numbers of the data packets through the buffer management channel to obtain the target media data.

[0152] In other embodiments of the present application, the processing unit 33 is further configured to perform the following steps:

[0153] Establish a mouse acquisition channel for the mouse content of the source device;

[0154] The mouse content in the media data is collected through the mouse collection channel; wherein the mouse content refers to the changes in the mouse when the mouse is operated.

[0155] In other embodiments of the present application, the processing unit 33 is further configured to perform the following steps:

[0156] Establish a reverse control channel for user input;

[0157] If the display device receives the manipulation operation instruction, it sends the manipulation operation instruction to the source device through the user input reverse control channel, so that the source device manipulates the media data based on the manipulation operation instruction.

[0158] It should be noted that the specific implementation process of the steps performed by each unit in the embodiment of the present application can be referred to Figure 1 and Figure 3 The implementation process of the information transmission method provided in the corresponding embodiment will not be repeated here.

[0159] The information transmission device provided in the embodiment of the present application establishes a first protocol between the source device and the display device, so that the source device can be connected to the display device to realize simultaneous sharing of multi-channel / multi-screen / multi-content information, and by adding multiple source device types and fields in the second protocol, the display device can support more source devices, thereby supporting more source devices to successfully establish session connections with the display device, solving Miracast compatibility issues, and at the same time, by monitoring packet loss data and adjusting the transmission parameters to be adjusted for transmitting media data based on the monitoring results, video freezes and screen distortion caused by data packet loss during the transmission of media data are greatly reduced, ensuring the transmission stability and smoothness of media data, thereby not only improving the video projection effect but also improving the user experience.

[0160] Based on the above embodiments, the embodiments of the present application provide a display device, which can be applied to Figure 1 and Figure 3 In the information transmission method provided in the corresponding embodiment, refer to Figure 8 As shown, the display device 4 may include: a processor 41, a memory 42 and a communication bus 43, wherein:

[0161] The communication bus 43 is used to realize the communication connection between the processor 41 and the memory 42;

[0162] The processor 41 is used to execute the information transmission program in the memory 42 to implement the following steps:

[0163] Initiating a session establishment request to the source device when the source device and the display device establish a device connection through the first protocol; wherein the source device and the display device are connected to the same wireless network;

[0164] In response to the session establishment request, based on the types and fields of the multiple source devices added in the second protocol, establish a session connection between the source device and the display device;

[0165] After the session connection is established, the packet loss data of the media data transmitted from the source device to the display device is monitored to obtain a monitoring result, and the transmission parameters to be adjusted for the transmitted media data are adjusted based on the monitoring result to obtain a target transmission parameter;

[0166] Media data is received based on the target transmission parameters.

[0167] In other embodiments of the present application, the processor 41 is used to execute the information transmission program in the memory 42 to monitor the packet loss data of the media data transmitted from the source device to the display device to obtain a monitoring result, and adjust the transmission parameters to be adjusted of the transmitted media data based on the monitoring result to obtain the target transmission parameters, so as to implement the following steps:

[0168] Determine a lost data packet in the media data based on a first sequence number of a current data packet in the media data and a second sequence number of a target data packet; wherein the media data includes a plurality of data packets; and the target data packet is a data packet preceding the current data packet;

[0169] For each lost data packet, if the number of retransmissions of the lost data packet meets the quantity threshold, the lost data packet is deleted; wherein the packet loss data includes the number of retransmissions of the lost data packet, and the monitoring result includes the number of retransmissions meeting the quantity threshold;

[0170] Adjust the transmission parameters to be adjusted to obtain target transmission parameters;

[0171] For each lost data packet, if the number of retransmissions of the lost data packet does not meet the quantity threshold, the lost data packet is sent to a buffer management channel; wherein the buffer management channel is a channel established between the source device and the display device.

[0172] In other embodiments of the present application, the processor 41 is used to execute the information transmission program in the memory 42 to determine the lost data packet in the media data based on the first sequence number of the current data packet in the media data and the second sequence number of the target data packet, so as to implement the following steps:

[0173] Calculate the first sequence number and the second sequence number to obtain the sequence interval between the current data packet and the target data packet;

[0174] Lost packets are determined based on the sequence interval and the number of packets stored in the buffer management channel.

[0175] In other embodiments of the present application, the processor 41 is used to execute the information transmission program in the memory 42 to determine the lost data packets based on the sequence interval and the storage number of the data packets stored in the buffer management channel, so as to implement the following steps:

[0176] Compare the sequence interval with the stored quantity to obtain a comparison result;

[0177] If the comparison result is that the sequence interval is less than or equal to the stored number, and the sequence interval is greater than the first value, it is determined that the lost data packets are all data packets between the first sequence number and the second sequence number.

[0178] In other embodiments of the present application, the processor 41 is used to execute the information transmission method of the information transmission program in the memory 42 to implement the following steps:

[0179] If the comparison result is that the sequence interval is greater than the storage quantity, the data packet before the target data packet is deleted from the buffer management channel;

[0180] If the sequence interval is equal to the first value, sending the current data packet to the buffer management channel;

[0181] If the sequence interval is smaller than a second value, the current data packet is deleted from the buffer management channel; wherein the second value is smaller than the first value.

[0182] In other embodiments of the present application, the processor 41 is used to execute the information transmission method of the information transmission program in the memory 42 to implement the following steps:

[0183] receiving a data packet transmitted by a source device to a buffer management channel through a transmission channel based on a target transmission parameter;

[0184] The data packets are rearranged according to the sequence numbers of the data packets through the buffer management channel to obtain the target media data.

[0185] In other embodiments of the present application, the processor 41 is used to execute the information transmission method of the information transmission program in the memory 42 to implement the following steps:

[0186] Establish a mouse acquisition channel for the mouse content of the source device;

[0187] The mouse content in the media data is collected through the mouse collection channel; wherein the mouse content refers to the changes in the mouse when the mouse is operated.

[0188] In other embodiments of the present application, the processor 41 is used to execute the information transmission method of the information transmission program in the memory 42 to implement the following steps:

[0189] Establish a reverse control channel for user input;

[0190] If the display device receives the manipulation operation instruction, it sends the manipulation operation instruction to the source device through the user input reverse control channel, so that the source device manipulates the media data based on the manipulation operation instruction.

[0191] It should be noted that the specific description of the steps performed by the processor can be referred to Figure 1 and Figure 3 The implementation process of the information transmission method provided in the corresponding embodiment will not be repeated here.

[0192] The display device provided in the embodiment of the present application establishes a first protocol between the source device and the display device, so that the source device can be connected to the display device to realize simultaneous sharing of multi-channel / multi-screen / multi-content information, and by adding multiple source device types and fields in the second protocol, the display device can support more source devices, thereby supporting more source devices to successfully establish session connections with the display device, solving Miracast compatibility issues, and at the same time, by monitoring packet loss data and adjusting the transmission parameters to be adjusted for transmitting media data based on the monitoring results, video freezes and screen distortion caused by data packet loss during the transmission of media data are greatly reduced, ensuring the transmission stability and smoothness of media data, thereby not only improving the video projection effect but also improving the user experience.

[0193] Based on the foregoing embodiments, the embodiments of the present application provide a computer-readable storage medium, which stores one or more programs, and the one or more programs can be executed by one or more processors to implement Figure 1 and Figure 3The steps in the information transmission method provided in the corresponding embodiment. It should be noted that the above-mentioned computer-readable storage medium can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a magnetic random access memory (FRAM), a flash memory (Flash Memory), a magnetic surface memory, an optical disk, or a compact disc read-only memory (CD-ROM) and other memories; it can also be various electronic devices including one or any combination of the above memories, such as mobile phones, computers, tablet devices, personal digital assistants, etc.

[0194] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0195] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0196] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, a magnetic disk, or an optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0197] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0198] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0199] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0200] The above are only preferred embodiments of the present application, and are not intended to limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. An information transmission method, characterized in that: Applied to a display device, the method comprises: Initiating a session establishment request to the source device when the source device and the display device establish a device connection through a first protocol; wherein the source device and the display device are connected to the same wireless network; In response to the session establishment request, based on the types and fields of multiple source devices added in the second protocol, establish a session connection between the source device and the display device; After the session connection is established, packet loss data of the media data transmitted from the source device to the display device is monitored to obtain a monitoring result, and based on the monitoring result, a transmission parameter to be adjusted for transmitting the media data is adjusted to obtain a target transmission parameter; The media data is received based on the target transmission parameters.

2. The method according to claim 1, characterized in that Monitoring packet loss data of media data transmitted from the source device to the display device to obtain a monitoring result, and adjusting a transmission parameter to be adjusted for transmitting the media data based on the monitoring result to obtain a target transmission parameter, including: Determine a lost data packet in the media data based on a first sequence number of a current data packet in the media data and a second sequence number of a target data packet; wherein the media data includes a plurality of data packets; and the target data packet is a data packet preceding the current data packet; For each of the lost data packets, if the number of retransmissions of the lost data packet meets the quantity threshold, delete the lost data packet; wherein the packet loss data includes the number of retransmissions of the lost data packet, and the monitoring result includes that the number of retransmissions meets the quantity threshold; Adjusting the transmission parameter to be adjusted to obtain the target transmission parameter; For each of the lost data packets, if the number of retransmissions of the lost data packet does not meet a quantity threshold, the lost data packet is sent to a buffer management channel; wherein the buffer management channel is a channel established between the source device and the display device.

3. The method according to claim 2, characterized in that The determining, based on the first sequence number of the current data packet in the media data and the second sequence number of the target data packet, a lost data packet in the media data comprises: Calculate the first sequence number and the second sequence number to obtain a sequence interval between the current data packet and the target data packet; The lost data packet is determined based on the sequence interval and the storage quantity of the data packets stored in the buffer management channel.

4. The method according to claim 3, characterized in that The determining the lost data packet based on the sequence interval and the storage quantity of the data packets stored in the buffer management channel comprises: Comparing the sequence interval with the stored quantity to obtain a comparison result; If the comparison result is that the sequence interval is less than or equal to the stored number, and the sequence interval is greater than a first value, it is determined that the lost data packets are all data packets between the first sequence number and the second sequence number.

5. The method according to claim 4, characterized in that The method further comprises: If the comparison result is that the sequence interval is greater than the storage quantity, deleting the data packet before the target data packet from the buffer management channel; If the sequence interval is equal to the first value, sending the current data packet to the buffer management channel; If the sequence interval is smaller than a second value, the current data packet is deleted from the buffer management channel; wherein the second value is smaller than the first value.

6. The method according to claim 5, characterized in that The method further comprises: receiving a data packet transmitted by a source device to the buffer management channel through a transmission channel based on the target transmission parameter; The data packets are rearranged according to the sequence numbers of the data packets through the buffer management channel to obtain target media data.

7. The method according to claim 1, characterized in that The method further comprises: Establishing a mouse acquisition channel for the mouse content of the source device; The mouse content in the media data is collected through the mouse collection channel; wherein the mouse content is the changing content of the mouse when the mouse is operated.

8. The method according to claim 1, characterized in that The method further comprises: Establish a reverse control channel for user input; If the display device receives a manipulation operation instruction, the manipulation operation instruction is sent to the source device through the user input reverse control channel, so that the source device manipulates the media data based on the manipulation operation instruction.

9. An information transmission device, characterized in that: The device comprises: an initiating unit, configured to initiate a session establishment request to the source device when the source device and the display device establish a device connection through a first protocol; wherein the source device and the display device are connected to the same wireless network; A session unit, configured to establish a session connection between the source device and the display device based on the types and fields of the multiple source devices added in the second protocol in response to the session establishment request; a processing unit, configured to, after establishing the session connection, monitor packet loss data of the media data transmitted from the source device to the display device to obtain a monitoring result, and adjust a transmission parameter to be adjusted for transmitting the media data based on the monitoring result to obtain a target transmission parameter; A receiving unit is used to receive the media data based on the target transmission parameter.

10. A display device, characterized in that: The device comprises: a processor, a memory and a communication bus; The communication bus is used to realize the communication connection between the processor and the memory; The processor is used to execute the information transmission program in the memory to implement the steps of the information transmission method according to any one of claims 1 to 8.

11. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the steps of the information transmission method according to any one of claims 1 to 8.

Images