Wireless screen projection method and device

Abstract

Embodiments of the present application provide a wireless screen projection method and device, which relate to the field of communications and can reduce screen tearing, freezes, and image quality degradation in wireless screen projection, thereby improving user experience. The wireless screen projection method, applied to a Wi-Fi communication system, includes: an electronic device transmitting a layered video coding segment of a first video frame to a display device in a first video transmission cycle; obtaining channel state information of the first video frame transmitted from the display device; and selectively initiating a retry mechanism based on the channel state information, wherein the retry mechanism is: transmitting the first layered video coding segment of the second video frame to the display device in a first time segment in a second video transmission cycle, and transmitting the discarded portion of the first layered video coding segment in the first time segment to the display device in at least one other time segment after the first time segment.

Description

Technical Field

[0001] The present application relates to the field of communications, and in particular to a wireless screen projection method and device. Background Art

[0002] Wireless screen projection is a technology that encodes and compresses the screen information of terminals such as mobile phones or tablets, and then wirelessly projects it to large-screen devices such as TVs or virtual reality (VR) devices through Wi-Fi technology (the Wi-Fi Alliance created the wireless local area network technology based on the Institute of Electrical and Electronics Engineers (IEEE) 802.11 standard) for decoding and display output.

[0003] Most of the existing wireless screen projection compression and transmission technologies are based on video frames, and generally have disadvantages such as large end-to-end delay and weak anti-interference ability, which greatly limits its application in scenarios such as office, gaming, and VR. The low-latency wireless screen projection solution that combines source coding and wireless channel transmission technology can solve the above problems to a certain extent. In this solution, the terminal continuously divides each frame of video data into multiple data segments (slices) based on source coding, and uses layered video coding technology for each slice to obtain multiple layered code streams (layers, for example: a base layer and multiple enhancement layers) with quality (or resolution) layering. The terminal transmits the layered code stream of the slice through Wi-Fi within the configured video service period (VSP). At this time, the terminal needs to transmit a slice within a fixed time segment within the VSP. If it is not fully transmitted, it needs to be discarded so as not to affect the normal transmission of subsequent slices. Because each slice is layered video encoded, for each slice, when Wi-Fi successfully transmits the base layer but fails to transmit the enhancement layer, a slightly lower-quality bitstream can still be decoded, thereby achieving low-latency transmission and the ability to adapt to fluctuations in wireless channel transmission.

[0004] However, wireless channel interference is often sudden, and the aforementioned solution only uses layered coding to mitigate fluctuations in channel capacity, so its capabilities are limited. This is especially true in environments with high Wi-Fi interference, where even the base layer often fails to transmit correctly within a certain timeframe. This can cause screen tearing, lags, and image quality degradation during wireless screen projection, impacting the user experience. Summary of the Invention

[0005] The present application provides a wireless screen projection method and device, which can reduce the screen tearing, freeze and image quality degradation that occur in wireless screen projection, and improve user experience.

[0006] In a first aspect, a wireless screen projection method is provided. The wireless screen projection method is applied to a Wi-Fi communication system, and the electronic device and the display device are connected via wireless fidelity Wi-Fi. The wireless screen projection method includes: the electronic device transmits a layered video coding segment of a first video frame to the display device in a first video transmission cycle; the electronic device obtains channel state information of any layered video coding segment of the first video frame fed back by the display device; the electronic device selectively starts a retry mechanism according to the channel state information, wherein the retry mechanism is: the electronic device transmits the first layered video coding segment of the second video frame to the display device in a first time segment in a second video transmission cycle, and transmits the discarded portion of the first layered video coding segment in the first time segment to the display device in at least one other time segment after the first time segment, wherein the at least one other time segment respectively transmits one other layered video coding segment of the second video frame. In this way, the electronic device can first transmit the layered video coding segments of the video frame in the video transmission cycle, for example, transmitting a layered video coding segment in a time segment of the initial state video transmission cycle. Of course, if the layered video coding segment is not fully transmitted within the corresponding time segment, it is directly discarded. Then, the electronic device obtains the channel state information of any layered video coding segment of the first video frame transmitted from the display device, for example, after each time segment, the channel state information of the layered video coding segment transmitted in the time segment is obtained. It can be understood that the channel state information can reflect the discarded portion of the layered video coding segment in the corresponding time segment. For example, the channel state can be the total number of data packets of the layered video coding segment transmitted in a time segment and the number of successfully transmitted data packets. In this way, the electronic device can selectively activate a retry mechanism based on the channel state information. For example, if the electronic device determines that the channel quality is poor based on the channel state information, it activates the retry mechanism; or if the channel quality is good, it does not activate the retry mechanism, or if the retry mechanism has been activated, it disables the retry mechanism when it is determined that the channel quality is good. The retry mechanism is as follows: the electronic device uses multiple time segments in a subsequent video transmission cycle to transmit a layered video coding segment. For example, in a first time segment in a subsequent video transmission cycle, the electronic device transmits a first layered video coding segment of another video frame to the display device, and transmits the discarded portion of the first layered video coding segment in the first time segment to the display device in at least one other time segment after the first time segment, wherein at least one other time segment respectively transmits another layered video coding segment of the second video frame. In this way, when a layered video coding segment has a discarded portion in the corresponding time segment, the discarded portion can be transmitted in another time segment.For example, if the base layer and at least one enhancement layer of a slice's layered video coding segment are not fully transmitted in their corresponding time segments, they may be transmitted in the next time segment. In this way, the display device can decode the first layered video coding segment transmitted in the two time segments. This ensures the successful transmission of the layered video coding segment while sacrificing some latency, thereby reducing screen tearing, freezes, and image quality degradation in wireless screen projection and improving user experience.

[0007] In one possible implementation, the layered video coding segment includes a base layer and at least one enhancement layer; the electronic device transmits the first layered video coding segment of the second video frame in the first time segment of the second video transmission cycle, comprising: the electronic device transmits the base layer and at least one enhancement layer of the first layered video coding segment of the second video frame to the display device in the first time segment of the second video transmission cycle according to the transmission priority of the base layer and at least one enhancement layer of the first layered video coding segment of the second video frame. Typically, layered video coding is performed on each slice of the first video frame to obtain a layered video coding segment; therefore, for each layered video coding segment, when the base layer is successfully transmitted according to the transmission priority but the enhancement layer fails to be transmitted, a code stream with slightly lower quality can still be decoded, thereby achieving low-latency transmission and the ability to adapt to fluctuations in wireless channel transmission.

[0008] In a possible implementation, the layered video coding segment includes a base layer and at least one enhancement layer; the at least one other time segment includes a second time segment of the second video transmission period; the other layered video coding segment includes a second layered video coding segment of the second video frame;

[0009] The electronic device transmits the discarded part of the first layered video coding segment in the first time segment to the display device in at least one other time segment after the first time segment, including: the electronic device transmits the discarded part of the first layered video coding segment and the second layered video coding segment to the display device in the second time segment according to the transmission priority of the basic layer and at least one enhancement layer of the first layered video coding segment, and the transmission priority of the basic layer and at least one enhancement layer of the second layered video coding segment, wherein the transmission priority of the basic layer of the first layered video coding segment is higher than the transmission priority of the basic layer of the second layered video coding segment, the transmission priority of the basic layer of the second layered video coding segment is higher than the transmission priority of the enhancement layer of the first layered video coding segment, and the transmission priority of the enhancement layer of the first layered video coding segment is higher than the transmission priority of the enhancement layer of the second layered video coding segment. In this way, for the data transmitted in the second time segment, it can be transmitted in the manner of prioritizing the transmission of the basic layer of the first layered video coding segment, then the basic layer of the second layered video coding segment, then the enhancement layer of the first layered video coding segment, and finally the enhancement layer of the second layered video coding segment. In this way, it is prioritized to ensure that the basic layer of each layered video coding segment is successfully transmitted in sequence in the corresponding time segment, and then it is ensured that the enhancement layer of each layered video coding segment is successfully transmitted in sequence in the time segment.

[0010] In one possible implementation, the discarded portion of the first layered video coding segment includes the base layer and at least one enhancement layer of the first layered video coding segment; the electronic device transmits the discarded portion of the first layered video coding segment in the first time segment to the display device in at least one other time segment after the first time segment, including: the electronic device transmits the base layer and at least one enhancement layer of the first layered video coding segment to the display device in at least one other time segment after the first time segment. In this solution, if the base layer and at least one enhancement layer of the first layered video coding segment are both discarded (transmission is not completed) within the second video transmission period, the base layer and at least one enhancement layer of the first layered video coding segment can be retransmitted in at least one other time segment after the first time segment.

[0011] In one possible implementation, the discarded portion of the first layered video coding segment includes at least one enhancement layer of the first layered video coding segment; the electronic device transmits the discarded portion of the first layered video coding segment in the first time segment to the display device in at least one other time segment after the first time segment, including: the electronic device transmits at least one enhancement layer of the first layered video coding segment to the display device in at least one other time segment after the first time segment. In this solution, if only at least one enhancement layer of the first layered video coding segment is discarded (not transmitted) in the first time segment, then only the at least one enhancement layer of the first layered video coding segment can be retransmitted in at least one other time segment after the first time segment.

[0012] In one possible implementation, the method further includes: reconstructing a reference frame based on the successfully transmitted portion of the first layered video coding segment after the at least one other time segment; and generating the first layered video coding of the third video frame based on the reference frame. Thus, when the electronic device transmits the third video frame, it can refer to the successfully transmitted portion of the first layered video coding segment of the second video frame in the first time segment and at least one other time segment after the first time segment. The inter-frame predictive coding (P-frame) encoding thus formed can reduce the amount of encoded data and ensure that the third video frame of the electronic device can be decoded according to the actually successfully transmitted portion after being transmitted to the display device.

[0013] In a second aspect, a wireless screen projection method is provided. Applied to a Wi-Fi communication system, an electronic device and a display device are connected via wireless fidelity Wi-Fi. The wireless screen projection method includes: the display device receives a layered video coding segment of a first video frame transmitted by the electronic device to the display device in a first video transmission cycle; the display device feeds back channel state information of the first video frame to the electronic device, wherein the electronic device selectively initiates a retry mechanism based on the channel state information; the display device decodes the layered video coding segment transmitted by the electronic device in the first video transmission cycle; the display device receives a first layered video coding segment of a second video frame transmitted by the electronic device in a first time segment in a second video transmission cycle under the retry mechanism; the display device receives a portion of the first layered video coding segment discarded by the electronic device in the first time segment transmitted by the electronic device in at least one other time segment after the first time segment, wherein the at least one other time segment respectively transmits one other layered video coding segment of the second video frame; and the display device decodes the first layered video coding segment of the second video frame transmitted by the electronic device in the first time segment and the portion of the first layered video coding segment discarded in the first time segment transmitted in at least one other time segment after the first time segment.

[0014] In one possible implementation, the layered video coding segment includes a base layer and at least one enhancement layer; the display device receives the first layered video coding segment of the second video frame transmitted by the electronic device on the first time segment in the second video transmission cycle, including: the display device receives the base layer and at least one enhancement layer of the first layered video coding segment of the second video frame transmitted by the electronic device on the first time segment in the second video transmission cycle according to the transmission priority of the base layer and at least one enhancement layer of the first layered video coding segment.

[0015] In one possible implementation, the layered video coding segment includes a base layer and at least one enhancement layer; the display device receives the part of the first layered video coding segment discarded on the first time segment transmitted by the electronic device on at least one other time segment after the first time segment, including: the display device receives the base layer and at least one enhancement layer of the first layered video coding segment transmitted by the electronic device on at least one other time segment after the first time segment.

[0016] In one possible implementation, the layered video coding segment includes a base layer and at least one enhancement layer; the display device receives the part of the first layered video coding segment discarded on the first time segment transmitted by the electronic device on at least one other time segment after the first time segment, including: the display device receives at least one enhancement layer of the first layered video coding segment transmitted by the electronic device on at least one other time segment after the first time segment.

[0017] In a third aspect, a wireless screen projection device is provided, which is applied to a Wi-Fi communication system. The wireless screen projection device can be an electronic device, a module or a chip in the electronic device, or a chip or a system on chip, including: a transmitter, used to transmit a layered video coding segment of a first video frame to a display device in a first video transmission cycle; a receiver, used to obtain channel state information of transmitting the first video frame fed back by the display device; a processor, used to selectively start a retry mechanism according to the channel state information obtained by the receiver, wherein the retry mechanism is: the transmitter is also used to transmit the first layered video coding segment of the second video frame to the display device in a first time segment in a second video transmission cycle, and transmit the part of the first layered video coding segment discarded in the first time segment to the display device in at least one other time segment after the first time segment, wherein the at least one other time segment respectively transmits one other layered video coding segment of the second video frame.

[0018] In one possible implementation, the layered video coding segment includes a base layer and at least one enhancement layer; the transmitter is specifically used to transmit the base layer and at least one enhancement layer of the first layered video coding segment of the second video frame to the display device in the first time segment of the second video transmission period according to the transmission priority of the base layer and at least one enhancement layer of the first layered video coding segment of the second video frame.

[0019] In one possible implementation, the layered video coding segment includes a base layer and at least one enhancement layer; the at least one other time segment includes the second time segment of the second video transmission period; the other layered video coding segments include the second layered video coding segment of the second video frame; the transmitter is specifically used to transmit the discarded part of the first layered video coding segment and the second layered video coding segment to the display device in the second time segment according to the transmission priority of the base layer and at least one enhancement layer of the first layered video coding segment, and the transmission priority of the base layer and at least one enhancement layer of the second layered video coding segment, wherein the transmission priority of the base layer of the first layered video coding segment is higher than the transmission priority of the base layer of the second layered video coding segment, the transmission priority of the base layer of the second layered video coding segment is higher than the transmission priority of the enhancement layer of the first layered video coding segment, and the transmission priority of the enhancement layer of the first layered video coding segment is higher than the transmission priority of the enhancement layer of the second layered video coding segment.

[0020] In one possible implementation, the discarded portion of the first layered video coding segment includes the base layer and at least one enhancement layer of the first layered video coding segment; the transmitter is specifically used to transmit the base layer and at least one enhancement layer of the first layered video coding segment to the display device in at least one other time segment after the first time segment.

[0021] In one possible implementation, the discarded portion of the first layered video coding segment includes at least one enhancement layer of the first layered video coding segment; the transmitter is specifically used to transmit at least one enhancement layer of the first layered video coding segment to the display device on at least one other time segment after the first time segment.

[0022] In a possible implementation, the processor is further configured to, after the at least one other time segment, reconstruct a reference frame based on a portion of the first layered video coding segment that is successfully transmitted; and generate a first layered video coding of a third video frame based on the reference frame.

[0023] In a fourth aspect, a wireless screen projection device is provided, which is applied to a Wi-Fi communication system. The wireless screen projection device can be a display device, or a module or chip in a display device. The display device can also be a chip or a system on chip, including: a receiver for receiving a layered video coding segment of a first video frame transmitted by an electronic device to a display device in a first video transmission cycle; a transmitter for feeding back channel state information of transmitting the first video frame to the electronic device, wherein the electronic device selectively starts a retry mechanism according to the channel state information; a processor for decoding the layered video coding segment transmitted by the electronic device in the first video transmission cycle; and a receiver for receiving a signal from the electronic device during retry. A test mechanism is provided for decoding the first layered video coding segment of the second video frame transmitted on the first time segment in the second video transmission cycle; receiving the part of the first layered video coding segment discarded on the first time segment transmitted by the electronic device on at least one other time segment after the first time segment, wherein the at least one other time segment respectively transmits one other layered video coding segment of the second video frame; the processor is further used to decode the first layered video coding segment of the second video frame transmitted by the electronic device on the first time segment and the part of the first layered video coding segment discarded on the first time segment transmitted on at least one other time segment after the first time segment.

[0024] In one possible implementation, the layered video coding segment includes a base layer and at least one enhancement layer; the receiver is specifically used to receive the base layer and at least one enhancement layer of the first layered video coding segment of the second video frame transmitted by the electronic device in the first time segment in the second video transmission cycle according to the transmission priority of the base layer and at least one enhancement layer of the first layered video coding segment.

[0025] In one possible implementation, the layered video coding segment includes a base layer and at least one enhancement layer; the receiver is specifically used to receive the base layer and at least one enhancement layer of the first layered video coding segment transmitted by the electronic device on at least one other time segment after the first time segment.

[0026] In one possible implementation, the layered video coding segment includes a base layer and at least one enhancement layer; the receiver is specifically used to receive at least one enhancement layer of the first layered video coding segment transmitted by the electronic device in at least one other time segment after the first time segment.

[0027] In a fifth aspect, a computer-readable storage medium is provided, in which a computer program is stored. When the computer-readable storage medium is run on a computer, the computer can execute the method described in any one of the above aspects.

[0028] In a sixth aspect, a computer program product comprising instructions is provided, the computer program product comprising: computer program code, which enables the computer to execute the method described in any one of the above aspects when the computer program code is run on the computer.

[0029] In a seventh aspect, a communication system is provided, comprising the wireless screen projection device described in the third aspect and the wireless screen projection device described in the fourth aspect. In one example, the wireless screen projection device described in the third aspect may be an electronic device, such as a mobile phone; and the wireless screen projection device described in the fourth aspect may be a display device, such as a television.

[0030] Among them, the technical effects brought about by any design method in the second to seventh aspects can refer to the technical effects brought about by the different design methods in the above-mentioned first aspect, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 A schematic diagram of the architecture of a communication system provided in an embodiment of the present application;

[0032] Figure 2 A schematic diagram of the functional architecture of an electronic device provided in an embodiment of the present application;

[0033] Figure 3 A schematic diagram of the pipeline structure of a wireless screen projection method provided in an embodiment of the present application;

[0034] Figure 4 A schematic structural diagram of a wireless screen projection device provided in an embodiment of the present application;

[0035] Figure 5 A flowchart of a wireless screen projection method provided in an embodiment of the present application;

[0036] Figure 6 A schematic diagram of a pipeline structure of a wireless screen projection method provided in another embodiment of the present application;

[0037] Figure 7 A schematic diagram of a pipeline structure of a wireless screen projection method provided in yet another embodiment of the present application;

[0038] Figure 8 A schematic structural diagram of a wireless screen projection device provided in another embodiment of the present application;

[0039] Figure 9A structural schematic diagram of a wireless screen projection device is provided for yet another embodiment of the present application. DETAILED DESCRIPTION

[0040] In the following, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of the embodiments of this application, unless otherwise specified, "plurality" means two or more.

[0041] Currently, terminals can encode and compress the terminal's screen information through wireless screen projection technology, and then wirelessly project it to large-screen devices such as TVs or VR devices through Wi-Fi technology for decoding and display output. The low-latency wireless screen projection solution that combines source coding and wireless channel transmission technology can reduce the shortcomings of large end-to-end delay and weak anti-interference ability to a certain extent. The main principle of the low-latency wireless screen projection solution that combines source coding and wireless channel transmission technology is: the terminal continuously divides each frame of video data into multiple slices based on source coding, and uses layered video coding technology for each slice to obtain multiple layers with quality (or resolution) layering. The terminal transmits the slice's layered code stream through Wi-Fi within the configured VSP. At this time, the terminal needs to transmit a slice in a time segment within the VSP. If it is not fully transmitted, it needs to be discarded so as not to affect the normal transmission of subsequent slices. Because each slice is layered video encoded, even if Wi-Fi successfully transmits the base layer but fails to transmit the enhancement layer, a slightly lower-quality code stream can still be decoded for each slice, thereby achieving low-latency transmission and the ability to adapt to fluctuations in wireless channel transmission. However, wireless channel interference is often sudden, and the above solution only uses layered coding to resist fluctuations in channel capacity, so its capabilities are limited. Especially when the Wi-Fi environment is highly interfering, resulting in the base layer often being unable to be correctly transmitted within the time segment, wireless screen projection will experience screen tearing, freezes, and image quality degradation, affecting the user experience.

[0042] An embodiment of the present application provides a wireless screen projection method. The wireless screen projection method can be applied to a Wi-Fi communication system. The electronic device of the Wi-Fi communication system is connected to the display device via wireless fidelity Wi-Fi, and the electronic device wirelessly projects its screen information to the display device for decoding and display output. Using the wireless screen projection method provided by this embodiment, the electronic device transmits a layered video coding segment of a first video frame to the display device in a first video transmission cycle, wherein a time segment of the first video transmission cycle corresponds to the transmission of a layered video coding segment of the first video frame; for example: the electronic device performs layered video coding on multiple data segments of the first video frame in sequence to generate multiple layered video coding segments. For example, the electronic device encodes multiple data segments of the first video frame into multiple layered video coding segments, where one layered video coding segment may include a base layer and at least one enhancement layer; then the electronic device transmits the layered video coding segments of the video frame in each time segment of the video transmission period, where one layered video coding segment is transmitted in a time segment of the initial state video transmission period. Of course, if the layered video coding segment is not transmitted in the corresponding time segment, it is directly discarded; then, the electronic device obtains the channel state information of any layered video coding segment of the first video frame transmitted as fed back by the display device, for example, after each time segment, the channel state information of the layered video coding segment transmitted in the time segment is obtained. It can be understood that the channel state information can reflect the discarded part of the layered video coding segment in the corresponding time segment, for example, the channel state information can be the total number of data packets of the layered video coding segment transmitted in a time segment and the number of data packets successfully transmitted. In this way, the electronic device can selectively start the retry mechanism according to the channel status information. For example, if the electronic device determines that the channel quality is poor according to the channel status information, it starts the retry mechanism; or when the channel quality is good, it does not start the retry mechanism, or when the retry mechanism has been started, it turns off the retry mechanism when it is determined that the channel quality is good. The retry mechanism is that the electronic device uses multiple time segments to transmit a layered video coding segment in a subsequent video transmission cycle. For example, in a subsequent video transmission cycle, the electronic device transmits a first layered video coding segment of another video frame to the display device in a first time segment, and transmits a discarded portion of the first layered video coding segment in the first time segment to the display device in at least one other time segment after the first time segment, wherein at least one other time segment transmits another layered video coding segment of the second video frame. In this way, when a layered video coding segment has a discarded portion in the corresponding time segment, the discarded portion can be transmitted in another time segment.For example, when one or more of the base layer and at least one enhancement layer of a slice of layered video coding is discarded (not transmitted) in the corresponding time segment, it has the opportunity to be transmitted in the next time segment. In this way, the display device can decode the first layered video coding segment transmitted in the two time segments, so as to ensure the successful transmission of the layered video coding segment as much as possible while sacrificing some delay, thereby reducing the screen tearing and image quality degradation in wireless screen projection and improving user experience. In addition, after at least one other time segment, the reference frame is reconstructed based on the successfully transmitted part of the first layered video coding segment; the first layered video coding of the third video frame is generated based on the reference frame. In this way, when the electronic device transmits the third video frame, for the first layered video coding segment of the second video frame, it can refer to the successfully transmitted part in the first time segment and at least one other time segment after the first time segment. The inter-frame predictive coding frame (P frame) coding formed in this way can reduce the amount of encoded data and help ensure that the third video frame of the electronic device can be decoded according to the actually successfully transmitted part after being transmitted to the display device.

[0043] The following is a detailed description of a wireless screen projection method and device provided in an embodiment of the present application in conjunction with the accompanying drawings.

[0044] Figure 1 A schematic diagram of a communication system for a wireless screen projection method provided in an embodiment of the present application. Figure 1 As shown, the communication system includes an electronic device 100 and a display device 200. The electronic device 100 and the display device 200 can be connected via a wireless network, for example, via a Wi-Fi network.

[0045] In some embodiments, the electronic device 100 sends the content displayed on the screen to the display device 200 via the Wi-Fi network, and the display device 200 displays it. That is, during the wireless screen projection process, the display content of the electronic device 100 and the display device 200 is the same. Figure 2 The principle diagram of wireless screen projection is shown in FIG. Among them, the electronic device 100 can have the following functions. First, the electronic device 100 can display the video frame as a screen image, and can capture the displayed screen image as a video frame for frame caching. In addition, the electronic device 100 can provide frame level timing, that is, the video frame is divided into multiple slices in sequence according to the VSP, and each time segment in the VSP corresponds to a slice. The electronic device 100 can perform scaled video coding (SVC) on the slice (such as Figure 3As shown, encoding 1-8), for example, a slice is divided into multiple layers of resolution, quality, and frame rate. The electronic device 100 can adjust the encoding bit rate of the slice based on the channel status. That is, the slice is divided in time, space, and quality, and a multi-layer stream (including a base layer and an enhancement layer) is output. The base layer data allows the display device 200 to completely decode the basic video content. However, the video image obtained from the base layer data may have a lower frame rate, lower resolution, or lower quality. When the channel is limited or the channel environment is complex, it can ensure that the decoder can receive a smooth and watchable video image. When the channel environment is good or the channel resources are abundant, the enhancement layer data can be transmitted to improve the frame rate, resolution, or video quality. The enhancement layer can be encoded in multiple layers, which means that within the total bit rate of the video stream, the higher the received bit rate, the better the video quality. The electronic device 100 can also negotiate the transmission period with the display device 200, for example, to negotiate the VSP. Specifically, the electronic device 100 sends the negotiation parameters carrying the VSP to the display device 200 via a Wi-Fi unicast frame. In addition, the electronic device 100 can also synchronize transmission and reception with the display device 200 at regular intervals to ensure that the VSPs at both ends are synchronized. For example, a timer synchronization function (TSF) mechanism is used to ensure time synchronization between the electronic device 100 and the display device 200. Specifically, the electronic device 100 needs to periodically send beacon frames, and the display device 200 needs to wake up periodically to receive the beacon frames. For example, the display device 200 initializes the TSF timer, and uses the beacon frame to inform other electronic devices 100 of its local time, and sets the timestamp for sending the beacon frame to achieve transmission and reception synchronization. The electronic device 100 can also transmit the layered video coding data of the slice within the corresponding time segment in the VSP through transmission control (such as Figure 3 The time segments VSP1-VSP3 shown are respectively transmitted with coding 1-coding 3), and the electronic device 100 is guided to encode the slice with reference to the channel state. The functions of the display device 200 are as follows: the display device 200 is capable of negotiating the transmission cycle with the electronic device 100, and receiving the layered video coding data of the slice within the negotiated VSP through transmission control, and reordering the layered video coding data of the slice (for example, reordering the data packets according to the Wi-Fi data packet sequence number (seq), and discarding the empty messages). The display device 200 is capable of performing transmission and reception synchronization with the electronic device 100 according to the above-mentioned transmission and reception synchronization mechanism. The display device 200 is also capable of performing slice-level layered video decoding on the reordered data packets in combination with the channel state (such as Figure 3As shown, VSP2 starts decoding the received code), and merges the data, and finally displays the video frame on the screen. In an embodiment of the present application, the electronic device 100 can also negotiate with the display device 200 for at least one other time segment after the above-mentioned first time segment (VSP retry, or the video transmission period in the retry mechanism); in addition, the electronic device 100 can also reconstruct the reference frame based on the successful transmission of the layered video coding segment after at least one other time segment, wherein it can be understood that the reference frame refers to the reference object when P frame coding is used in the video frame transmitted subsequently by the second video frame. For example, usually when the first slice in the subsequent third video frame refers to the layered video coding of the first slice in the second video frame for P frame coding, one or more layers of the layered video coding of the first slice in the second video frame may not be fully transmitted within the corresponding time segment. Therefore, the layered video coding of the first slice in the third video frame is performed with reference to the layered video coding of the first slice in the second video frame, which may cause the display device 200 to fail to decode correctly. Therefore, in an embodiment of the present application, the electronic device 100 refers to the partially reconstructed reference frame that is successfully transmitted through layered video coding of the first slice in the second video frame as a reference object for layered video coding of the first slice in the third video frame.

[0046] The above-mentioned electronic device 100 includes, for example, a mobile phone, a tablet computer, a laptop computer, an ultra-mobile personal computer (UMPC), a netbook, a personal digital assistant (PDA), a vehicle-mounted device, a user terminal (UT), a terminal device (UD), a user equipment (UE), an artificial intelligence (AI) device, and other terminal devices with an image display function. The embodiment of the present application does not impose any restrictions on the specific type of the electronic device 100.

[0047] The above-mentioned display device 200 includes, for example, laptop computers, large-screen display devices (such as smart screens, etc.), projection devices, AI devices, tablet computers and other terminal devices that can realize large-screen display functions. The embodiment of this application does not impose any restrictions on the specific type of the display device 200.

[0048] Optionally, the electronic device 100 and the display device 200 in the embodiment of the present application can be implemented by different devices. For example, the electronic device 100 and the display device 200 in the embodiment of the present application can be implemented by different devices. Figure 4This is achieved by using the wireless screen projection device in the system. Figure 4 As shown, the wireless screen projection device 200 includes at least one processor 201, a communication line 202, a memory 203 and at least one communication interface 204. The memory 203 may also be included in the processor 201.

[0049] The processor 201 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor, etc.

[0050] The communication line 202 may be a circuit that connects the above components to each other and transmits information between the above components.

[0051] Communication interface 204 is used to communicate with other devices. In embodiments of the present application, communication interface 204 can be a module, circuit, bus, interface, transceiver, or other device capable of implementing communication functions, used to communicate with other devices. Optionally, when communication interface 204 is a transceiver, the transceiver can be an independently provided transmitter that can be used to send information to other devices, or the transceiver can be an independently provided receiver that is used to receive information from other devices. The transceiver can also be a component that integrates the functions of sending and receiving information. Embodiments of the present application do not limit the specific implementation of the transceiver.

[0052] The memory 203 may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. The non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM may be used, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct rambus RAM (DR RAM), or other magnetic storage devices, or any other medium that can be used to carry or store desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto. Memory 203 may be independent and connected to processor 201 via communication line 202. Memory 203 may also be integrated with processor 201.

[0053] The memory 203 is used to store computer-executable instructions for implementing the solution of the present application, and the execution is controlled by the processor 201. The processor 201 is used to execute the computer-executable instructions stored in the memory 203, thereby implementing the method for determining parameters provided in the following embodiments of the present application.

[0054] It should be noted that the memory described herein is intended to include, but not be limited to, these and any other suitable types of memory.

[0055] Optionally, the computer-executable instructions in the embodiments of the present application may also be referred to as application code, instructions, computer programs or other names, which are not specifically limited in the embodiments of the present application.

[0056] In a specific implementation, as an embodiment, the processor 201 may include one or more CPUs, such as Figure 4CPU0 and CPU1 in.

[0057] In a specific implementation, as an embodiment, the wireless screen projection device 200 may include multiple processors, such as Figure 4 201 and processor 207 in FIG. Each of these processors may be a single-CPU processor or a multi-CPU processor. A processor herein may refer to one or more devices, circuits, and / or processing cores for processing data (e.g., computer program instructions).

[0058] In a specific implementation, as an embodiment, the wireless projection device 200 may further include an output device 205 and an input device 206. The output device 205 communicates with the processor 201 and can display information in a variety of ways. For example, the output device 205 can be a liquid crystal display (LCD), a light emitting diode (LED) display device, a cathode ray tube (CRT) display device, or a projector. The input device 206 communicates with the processor 201 and can receive user input in a variety of ways. For example, the input device 206 can be a mouse, a keyboard, a touch screen device, or a sensing device.

[0059] It should be noted that the above-mentioned wireless screen projection device 200 can be a general device or a special device, and the embodiment of the present application does not limit the type of the wireless screen projection device. Figure 4 The structure of the wireless screen projection device 200 shown in the figure does not constitute a limitation of the wireless screen projection device. The actual wireless screen projection device may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently. The components shown in the figure can be implemented in hardware, software, or a combination of software and hardware.

[0060] The following takes the electronic device as a mobile phone and the display device as a television as an example, and combines the accompanying drawings to introduce in detail the wireless screen projection method provided in the embodiment of the present application. Figure 5 A flow chart of a wireless screen projection method provided in an embodiment of the present application. Figure 5 As shown, the method may include:

[0061] S501. The mobile phone establishes a Wi-Fi connection with the TV.

[0062] A mobile phone and a TV can establish a Wi-Fi connection using Wi-Fi's peer-to-peer (P2P) protocol. Establishing a Wi-Fi connection between terminals requires each terminal to have transmission capabilities and mutual knowledge of the other terminal's connection information. This connection information can include a terminal's device identifier, such as an Internet Protocol (IP) address, port number, or the terminal's login account. The login account can be an account provided by the operator. It can also be an application account. The terminal's transmission capabilities can include near-field communication (NFC) or long-range communication (LRC). In other words, the wireless communication protocol used to establish a connection between terminals, such as a mobile phone and a TV, can include a near-field communication protocol such as Wi-Fi, Bluetooth, or NFC, or a cellular network protocol. For example, a user can touch a mobile phone to an NFC tag on a TV, and the phone can read the connection information stored in the NFC tag, such as the TV's IP address. The mobile phone can then establish a connection to the TV using another protocol, such as Wi-Fi, based on the TV's IP address. For another example, both the mobile phone and the TV have Bluetooth and Wi-Fi enabled. A mobile phone can broadcast Bluetooth signals to discover nearby devices. For example, the phone can display a list of discovered devices, which includes the identifiers of the devices discovered by the phone, such as the identifier of a TV. During the device discovery process, the phone can also exchange connection information, such as IP addresses, with the discovered devices. After the phone receives an action from the user selecting the TV's identifier in the displayed device list, it can establish a connection with the TV using the Wi-Fi protocol based on the TV's IP address.

[0063] S502: The mobile phone negotiates screen projection parameters with the TV.

[0064] For example, a mobile phone can negotiate screen projection parameters by sending a Wi-Fi unicast frame to the TV. Specifically, the Wi-Fi unicast frame can carry VSP, VSP retry, and the delay time for screen projection startup. Among them, the time segment in VSP is mainly used for the first transmission of the layered video coding segment of the data segment in the video frame, and the time segment in VSP retry is mainly used for the retransmission of the part of the layered video coding segment of the data segment in the video frame that is discarded in the time segment of VSP, wherein a time segment in VSP retry is the same length as a time segment in VSP. The delay delay time for screen projection startup is used for both ends to agree on the start time of screen projection.

[0065] S503: The mobile phone transmits a layered video coding segment of a first video frame to the television in a first video transmission cycle.

[0066] The first video frame may be transmitted in a time segment of the first video transmission period in a layered video coding segment. First, the mobile phone needs to perform layered video coding on multiple data segments of the video frame in sequence to generate multiple layered video coding segments. Figure 3 As shown, the mobile phone performs layered video encoding on the first data segment slice of the first video frame to generate a first layered video encoding segment (such as encoding 1), and performs layered video encoding on the second data segment slice of the first video frame to generate a second layered video encoding segment (such as encoding 2). In this way, multiple data segments in the first video frame are sequentially encoded into multiple layered video encoding segments. The layered video encoding method for other video frames can refer to the description of the first video frame. It should be noted that the layered video encoding of any slice needs to be completed before the arrival of the time segment in the video transmission cycle in which it is transmitted. For example, the first video transmission cycle of the first video frame includes four time segments VSP1-VSP4. In this way, encoding 1 needs to be completed in VSP0 before VSP1, and encoding 2 needs to be completed in VSP1 before VSP2. Only in this way can encoding 1 be transmitted in VSP1 and encoding 2 be transmitted in VSP2. Specifically, for the first video frame transmitted by the mobile phone for the first time, since it can only be transmitted after encoding 1 is completed, no data transmission is performed before VSP1 (for example, VSP0). For the first video frame in the mobile phone transmission process, for example, there are other video frames before the first video frame, then in VSP0, the layered video coding fragments of the data fragments in the other video frames before the first video frame may be transmitted. In this embodiment, the emptiness in VSP0 only means that the data fragment of the first video frame is not transmitted. It can be understood that after the data fragment is layered video encoded, encoding 1 may include layers of different quality, resolution or frame rate, such as a base layer, a first enhancement layer, a second enhancement layer, etc. Figure 3 As shown, the mobile phone transmits the generated code 1 in VSP1 after VSP0. It should be noted that within VSP1, the mobile phone can perform QoS (quality of service) control. For example, because code 1 includes a base layer, a first enhancement layer, and a second enhancement layer, the base layer and at least one enhancement layer can be transmitted in sequence according to the transmission priority of each layer.

[0067] S504: The mobile phone obtains the channel state information of the first video frame transmitted from the television.

[0068] Mobile phone in Figure 3In the illustrated time segments VSP1, VSP2, ..., Code 1, Code 2, ... are sequentially transmitted. After each time segment, the TV can provide the mobile phone with channel state information for the corresponding layered video coding segment. For example, after VSP1, the TV provides the mobile phone with channel state information for Code 1. This channel state information may include the total number of data packets corresponding to each slice's code and the number of successfully transmitted data packets within the corresponding time segment, for example, the total number of Code 1 data packets and the number of successfully transmitted Code 1 data packets within VSP1 (excluding data packets successfully transmitted after holes). The mobile phone and the TV can use a block acknowledgement (ACK) mechanism to determine whether a data packet is successfully transmitted. For example, after receiving a series of data packets sent by the mobile phone, the TV provides a corresponding block ACK. The mobile phone determines the number of successfully transmitted data packets based on the bitmap in the block ACK. Alternatively, the TV can transmit slice-specific reception feedback information to the mobile phone, and the mobile phone can determine the channel state information based on the reception feedback information. The electronic device can selectively activate a retry mechanism based on the channel state information. For example, the electronic device can determine the channel quality based on the channel state information. When the channel quality is below a quality threshold, the channel quality is determined to be poor, and the retry mechanism is activated. Alternatively, when the channel quality is greater than or equal to the quality threshold, the channel state is determined to be good, and the retry mechanism is not activated. Alternatively, if the retry mechanism has already been activated, the retry mechanism is disabled when the channel state is determined to be good. The quality threshold can be a number threshold set for successfully transmitted data packets, or a reference threshold set for the proportion of the number of successfully transmitted data packets in the total number of data packets corresponding to the encoding. For example, when the channel state is poor, the number of successfully transmitted data packets is less than a certain number threshold, or the proportion of the number of successfully transmitted data packets is less than a reference threshold; when the channel state is good, the number of successfully transmitted data packets is greater than or equal to a certain number threshold, or the proportion of the number of successfully transmitted data packets is greater than or equal to the reference threshold. The retry mechanism is to execute step S506.

[0069] S505: The TV decodes the layered video encoding segment transmitted by the mobile phone in the first video transmission cycle.

[0070] Specifically, after Code 1 is transmitted to the TV set in VSP1, the TV set starts decoding the layered video coding segments received by VSP1 after VSP1 (eg, in VSP2).

[0071] S506. The mobile phone transmits the first layered video coding segment of the second video frame to the TV in the first time segment in the second video transmission cycle, and transmits the part of the first layered video coding segment discarded in the first time segment to the TV in at least one other time segment after the first time segment, wherein at least one other time segment respectively transmits another layered video coding segment of the second video frame.

[0072] Among them Figure 6 As shown, the second video frame is transmitted in the second video transmission period (including time segments VSP5, VSP6, VSP7, VSP8). Figure 6 In the illustrated time segments VSP5, VSP6, etc., Code 5, Code 6, etc. are sequentially transmitted. Within VSP5, the mobile phone can perform QoS (Quality of Service) control. For example, since Code 5 includes a base layer, a first enhancement layer, a second enhancement layer, etc., the base layer and at least one enhancement layer can be transmitted sequentially according to the transmission priority of each layer. Specifically, if the transmission of Code 5 is not completed within VSP5 in step S504, i.e., if a portion of Code 5 is discarded within VSP5, the incomplete portion of Code 5 is retransmitted in the time segment VSP1 re1 (retry1, the first retransmission) following VSP5. Specifically, as described above, Code 5 includes coded data of multiple layers, such as a base layer and at least one enhancement layer. Therefore, in step S506, if both the base layer and at least one enhancement layer of Code 5 are discarded within VSP5 (i.e., neither is completely transmitted), the base layer and at least one enhancement layer of Code 5 are transmitted within VSP1 re1. If the base layer of code 5 is transmitted in VSP5 and at least one enhancement layer is discarded in VSP5, then at least one enhancement layer of code 5 is transmitted in VSP1 re1. Before step S505, VSP retry can be disabled by default. That is, the mobile phone refers to the VSP retry for each video frame. Figure 3 As shown, code 1, code 2, ... are transmitted in sequence in time segments VSP1, VSP2, ...; when VSP retry is enabled according to the channel state information, for subsequent video frames, for example Figure 6The second video frame is transmitted in sequence in time segments VSP5, VSP6, ..., and the discarded parts of codes 5, codes 6, ... in their respective corresponding time segments VSP5, VSP6, ... are retransmitted in sequence in VSP1 re1, VSP2re2, ... In addition, according to the result of the negotiation in step S502, codes 5, codes 6, ... may also be retransmitted multiple times. For example, if there is still a discarded part (incomplete transmission) of code 5 after transmission of VSP1 and VSP1 re1, it may also be retransmitted in the next time segment VSP1 re2 of VSP1re1. Figure 6 As shown, two VSPs are included, wherein VSP (VSP5, VSP6...) are mainly used to sequentially transmit code 5, code 6..., and in VSP retry (VSP1 re1, VSP2 re2...), the parts of code 5, code 6... discarded in their respective corresponding time segments VSP5, VSP6... are sequentially retransmitted, and in the time segment after VSP1 re1 (VSP7 / VSP2 re1), the TV decodes the received code 5 into decoding 5; at this time, the channel state information feedback is delayed by one VSP (the channel state feedback is delayed to after VSP6). And the electronic device can re-determine whether to start the retry mechanism in the subsequent video transmission cycle based on the channel state information. The number of VSP retry times can also be configured to be more, for example, in Figure 7If VSP retry is configured as 2, it can be understood that the greater the number of VSP retries, the greater the chance of Code 5 being fully transmitted, the stronger its ability to resist transient interference, and the larger the coding buffer required. When VSP retry is enabled, the different priorities of different layers within a slice are jointly controlled for QoS flow control. For example, if VSP retry is set to 1, there will be two VSP transmission cycles (VSP and VSP retry) at a given moment, corresponding to the VSP retry cycle of the previous slice and the VSP cycle of the current slice. During the VSP6 cycle, there will be an opportunity to transmit the data of Code 5 that was not successfully transmitted within VSP5. In one example, within VSP6, the transmission priority of the base layer of the first layered video coding segment (Code 5) of the second video frame is higher than the transmission priority of the base layer of the second layered video coding segment (Code 6), the transmission priority of the base layer of the second layered video coding segment (Code 6) is higher than the transmission priority of the enhancement layer of the first layered video coding segment (Code 5), and the transmission priority of the enhancement layer of the first layered video coding segment (Code 5) is higher than the transmission priority of the enhancement layer of the second layered video coding segment (Code 6). Specifically, assuming that the basic layer 1 of coding 5 in VSP5 has been transmitted successfully, and the enhancement layer 2 and enhancement layer 3 of coding 5 have not been transmitted successfully, then in VSP6, the transmission priority of each layer from high to low is the transmission priority of basic layer 1 of coding 5, the transmission priority of basic layer 1 of coding 6, the transmission priority of enhancement layer 2 of coding 5, the transmission priority of enhancement layer 2 of coding 6, the transmission priority of enhancement layer 3 of coding 5, the transmission priority of enhancement layer 3 of coding 6...

[0073] In addition, after at least one other time segment, a reference frame is reconstructed based on the successfully transmitted portion of the first layered video coding segment; and the first layered video coding of the third video frame is generated based on the reference frame. Figure 6As shown, after VSP6 (VSP1 re1), the mobile phone reconstructs the reference frame 1 based on the part of the code 5 that has been successfully transmitted; and performs layered video encoding on the first data segment of the third video frame in VSP8 based on the reference frame 1 to generate code 9, wherein the third video frame is located after the second video frame. It can be understood that the reference frame 1 refers to the reference object when P frame encoding is used in the third video frame that is subsequently transmitted after the second video frame. For example, usually when code 9 in the subsequent third video frame refers to code 5 in the second video frame for P frame encoding, one or more layers of code 5 may not be fully transmitted in the corresponding VSP5 and VSP6. Therefore, referring to code 5 to generate code 9 in the third video frame may cause the TV to be unable to decode correctly. In an embodiment of the present application, dynamic reference frame reconstruction is designed, and the mobile phone refers to the part of the code 5 in the second video frame that has been successfully transmitted to reconstruct the reference frame 1 as the reference object for generating code 9 in the third video frame. For example, the reference frame is reconstructed with reference to only the successfully transmitted basic layer. As shown Figure 6 As shown in the figure, when the number of VSP retry is 1, the reconstruction of reference frame 1 is delayed by one VSP (after VSP6 / VSP1 re1), which does not affect the subsequent encoding 9. The dynamically reconstructed reference frame 5 is used for subsequent P frame encoding. Combined with the bit rate control of the mobile phone, the amount of encoded data can be significantly reduced and the transmission anti-interference ability can be enhanced. Figure 6 、 Figure 7 In the examples above, a video frame is divided into four slices. Figure 7 As shown, when the number of VSP retries is 2 or greater than 2, since the transmission of code 5 and the feedback of channel state information are completed only at VSP7, the mobile phone cannot generate a reference frame for code 9 in a timely manner. Therefore, an alternate frame reference can be performed. For example, reference frame 1 is generated at VSP11 to refer to reference frame 1 when generating code 13 in the next fourth video frame. The specific embodiments of the present application do not limit the number of slices divided into each video frame and the number of VSP retries. Specifically, the number of slices and the number of VSP retries can be dynamically adjusted based on the channel state information and the negotiation in step S502, taking into account both latency and anti-interference performance.

[0074] S507. The TV decodes the first layered video coding segment of the second video frame transmitted by the mobile phone in the first time segment and the discarded portion of the first layered video coding segment in the first time segment transmitted in at least one other time segment after the first time segment.

[0075] Prior to step S507, the television receives a first layered video coded segment of a second video frame transmitted by the mobile phone during a first time segment in a second video transmission cycle; and receives a portion of the first layered video coded segment discarded during a first time segment transmitted by the mobile phone during at least one other time segment after the first time segment, wherein the at least one other time segment respectively transmits another layered video coded segment of the second video frame. It should be noted that a layered video coded segment includes a base layer and at least one enhancement layer. Specifically, the television receives the base layer and at least one enhancement layer of the first layered video coded segment of the second video frame transmitted by the mobile phone during the first time segment in the second video transmission cycle according to the transmission priority of the base layer and at least one enhancement layer of the first layered video coded segment. Furthermore, if the base layer and at least one enhancement layer of the first layered video coded segment of the second video transmission are not fully transmitted within the first time segment, the mobile phone will receive the base layer and at least one enhancement layer of the first layered video coded segment transmitted by the television during at least one other time segment after the first time segment. If at least one enhancement layer of the first layered video coded segment of the second video transmission is not fully transmitted within the first time segment, the mobile phone will receive the at least one enhancement layer of the first layered video coded segment transmitted by the electronic device during at least one other time segment after the first time segment. Reference Figure 6 As shown, the TV receives the code 1 transmitted by the mobile phone in VSP5 and the part discarded by the mobile phone when VSP5 transmits code 5 in VSP6; and decodes the code 1 received in VSP5 and VSP6 and displays it on the screen.

[0076] In the above scheme, the mobile phone can first transmit the layered video coding segments of the video frame during the video transmission cycle, for example, transmitting a layered video coding segment during a time segment of the initial video transmission cycle. Of course, if the layered video coding segment is not fully transmitted within the corresponding time segment, it is directly discarded. Then, the mobile phone obtains the channel state information of any layered video coding segment of the first video frame fed back by the TV. For example, the channel state information of the layered video coding segment transmitted in each time segment can be obtained. It can be understood that the channel state information can reflect the discarded portion of the layered video coding segment in the corresponding time segment. For example, the channel state can be the total number of data packets of the layered video coding segment transmitted in a time segment and the number of data packets successfully transmitted. In this way, the mobile phone can selectively activate a retry mechanism based on the channel state information. For example, if the mobile phone determines that the channel quality is poor based on the channel state information, it can activate the retry mechanism; or if the channel quality is good, it can not activate the retry mechanism. Or, if the retry mechanism has been activated, it can be disabled when it is determined that the channel quality is good. The retry mechanism involves the phone using multiple time segments to transmit a layered video coded segment during a subsequent video transmission cycle. For example, during a first time segment in a subsequent video transmission cycle, the phone transmits a first layered video coded segment of another video frame to the display device, and during at least one other time segment following the first time segment, transmits the discarded portion of the first layered video coded segment during the first time segment to the display device. Each of the at least one other time segments transmits another layered video coded segment of the second video frame. This allows the phone to transmit a discarded portion of a layered video coded segment during a corresponding time segment in a subsequent video transmission cycle. For example, if one or more of the base layer and at least one enhancement layer of a layered video coded segment in a slice is not fully transmitted during its corresponding time segment, it may be transmitted during the next time segment. The TV can then decode the combined first layered video coded segments transmitted in both time segments, ensuring successful transmission of the layered video coded segments while sacrificing some latency. This reduces screen tearing, lag, and image quality degradation during wireless screen mirroring, improving the user experience. Furthermore, after at least one other time segment, a reference frame is reconstructed based on the successfully transmitted portion of the first layered video coding segment; and the first layered video coding of the third video frame is generated based on the reference frame. Thus, when the electronic device transmits the third video frame, it can refer to the successfully transmitted portion of the first layered video coding segment of the second video frame in the first time segment and at least one other time segment after the first time segment. The resulting inter-frame predictive coding (P-frame) encoding reduces the amount of encoded data and helps ensure that the third video frame of the electronic device can be decoded according to the actually successfully transmitted portion after being transmitted to the display device.

[0077] It is understandable that, in order to realize the above functions, the above-mentioned mobile phones and televisions include hardware structures and / or software modules corresponding to the execution of each function. Those skilled in the art should easily realize that, in combination with the units and algorithmic operations of the various examples described in the embodiments disclosed herein, the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0078] The embodiment of the present application can divide the functional modules of the mobile phone according to the above method example. For example, each functional module can be divided according to each function, or two or more functions can be integrated into one processing module. The above integrated modules can be implemented in the form of hardware or in the form of software functional modules. It should be noted that the division of modules in the embodiment of the present application is schematic and is only a logical function division. In actual implementation, there may be other division methods.

[0079] For example, when the functional modules are divided in an integrated manner, Figure 8 The figure shows a schematic diagram of the structure of a wireless screen projection device 80. The wireless screen projection device 80 can be a chip or system-on-chip in the above-mentioned mobile phone, or other combined devices, components, etc. that can realize the functions of the above-mentioned mobile phone. The wireless screen projection device 80 can be used to perform the functions of the mobile phone involved in the above-mentioned embodiments.

[0080] As a possible implementation, Figure 8 The wireless screen projection device 80 shown includes: a sending unit 801, a processing unit 802, and a receiving unit 803. The sending unit 801 is used to transmit a layered video coding segment of a first video frame to a display device in a first video transmission cycle; the receiving unit 803 is used to obtain channel state information of the transmission of the first video frame fed back by the display device; the processing unit 802 is used to selectively start a retry mechanism according to the channel state information obtained by the receiver, wherein the retry mechanism is: the sending unit 801 is also used to transmit a first layered video coding segment of a second video frame to the display device in a first time segment in a second video transmission cycle, and transmit to the display device in at least one other time segment after the first time segment a portion of the first layered video coding segment discarded in the first time segment, wherein the at least one other time segment respectively transmits one other layered video coding segment of the second video frame.

[0081] Optionally, the layered video coding segment includes a base layer and at least one enhancement layer; the sending unit 801 is specifically used to transmit the base layer and at least one enhancement layer of the first layered video coding segment of the second video frame to the display device in the first time segment of the second video transmission period according to the transmission priority of the base layer and at least one enhancement layer of the first layered video coding segment of the second video frame.

[0082] Optionally, the layered video coding segment includes a base layer and at least one enhancement layer; the at least one other time segment includes the second time segment of the second video transmission period; the other layered video coding segments include the second layered video coding segment of the second video frame; the sending unit 801 is specifically used to transmit the discarded part of the first layered video coding segment and the second layered video coding segment to the display device in the second time segment according to the transmission priority of the base layer and at least one enhancement layer of the first layered video coding segment, and the transmission priority of the base layer and at least one enhancement layer of the second layered video coding segment, wherein the transmission priority of the base layer of the first layered video coding segment is higher than the transmission priority of the base layer of the second layered video coding segment, the transmission priority of the base layer of the second layered video coding segment is higher than the transmission priority of the enhancement layer of the first layered video coding segment, and the transmission priority of the enhancement layer of the first layered video coding segment is higher than the transmission priority of the enhancement layer of the second layered video coding segment.

[0083] Optionally, the discarded part of the first layered video coding segment includes the base layer and at least one enhancement layer of the first layered video coding segment; the sending unit 801 is specifically used to transmit the base layer and at least one enhancement layer of the first layered video coding segment to the display device in at least one other time segment after the first time segment.

[0084] Optionally, the discarded portion of the first layered video coding segment includes at least one enhancement layer of the first layered video coding segment; the sending unit 801 is specifically used to transmit at least one enhancement layer of the first layered video coding segment to the display device in at least one other time segment after the first time segment.

[0085] Optionally, the processing unit 802 is further configured to reconstruct a reference frame according to a portion of the first layered video coding segment that is successfully transmitted after the at least one other time segment; and generate the first layered video coding of a third video frame according to the reference frame.

[0086] Among them, all relevant contents of each operation involved in the above method embodiment can be referred to the functional description of the corresponding functional module, and will not be repeated here.

[0087] In this embodiment, the wireless screen projection device 80 is presented in the form of dividing various functional modules in an integrated manner. The "module" here can refer to a specific ASIC, circuit, processor and memory that executes one or more software or firmware programs, integrated logic circuit, and / or other devices that can provide the above functions. In a simple embodiment, those skilled in the art can imagine that the wireless screen projection device 80 can be used Figure 4 The form shown.

[0088] for example, Figure 4 The processor 201 can call the computer execution instructions stored in the memory 203 to enable the wireless screen projection device to execute the wireless screen projection method in the above method embodiment.

[0089] For example, Figure 8 The functions / implementation processes of the sending unit 801, the processing unit 802 and the receiving unit 803 can be realized by Figure 4 The processor 201 in the memory 203 calls the computer execution instruction stored in the memory 203 to implement; or, Figure 8 The function / implementation process of the processing unit 802 can be achieved by Figure 4 The processor 201 in the memory 203 calls the computer execution instruction stored in the memory to implement, Figure 8 The function / implementation process of the sending unit 801 can be achieved by Figure 4 This is achieved by the transmitter in the communication interface 204 in . Figure 8 The function / implementation process of the receiving unit 803 can be achieved by Figure 4 This is achieved by the receiver in the communication interface 204 in .

[0090] Since the wireless screen projection device 80 provided in this embodiment can execute the above-mentioned wireless screen projection method, the technical effects that can be obtained can refer to the above-mentioned method embodiment and will not be repeated here.

[0091] In the embodiment of the present application, the functional modules of the television can be divided according to the above method example. For example, each functional module can be divided according to each function, or two or more functions can be integrated into one processing module. The above integrated modules can be implemented in the form of hardware or in the form of software functional modules. It should be noted that the division of modules in the embodiment of the present application is schematic and is only a logical functional division. In actual implementation, there may be other division methods.

[0092] For example, when the functional modules are divided in an integrated manner, Figure 9The figure shows a schematic diagram of the structure of a wireless screen projection device 90. The wireless screen projection device 90 can be a chip or system-on-chip in the above-mentioned television, or other combined devices, components, etc. that can realize the functions of the above-mentioned television. The wireless screen projection device 90 can be used to perform the functions of the television involved in the above-mentioned embodiments.

[0093] As a possible implementation, Figure 9 The wireless screen projection device 90 shown includes: a receiving unit 901, a processing unit 902, and a sending unit 903. The receiving unit 901 is configured to receive a layered video coded segment of a first video frame transmitted by an electronic device to a display device in a first video transmission cycle; the sending unit 903 is configured to provide the electronic device with feedback of channel state information for transmitting the first video frame, wherein the electronic device selectively initiates a retry mechanism based on the channel state information; the processing unit 902 is configured to decode the layered video coded segment transmitted by the electronic device in the first video transmission cycle; the receiving unit 901 is further configured to receive a first layered video coded segment of a second video frame transmitted by the electronic device in a first time segment in a second video transmission cycle during the retry mechanism; receive a portion of the first layered video coded segment discarded by the electronic device in the first time segment transmitted in at least one other time segment after the first time segment, wherein the at least one other time segment respectively transmits one other layered video coded segment of the second video frame; and the processing unit 902 is further configured to decode the first layered video coded segment of the second video frame transmitted by the electronic device in the first time segment and the portion of the first layered video coded segment discarded in the first time segment transmitted in at least one other time segment after the first time segment.

[0094] Optionally, the layered video coding segment includes a base layer and at least one enhancement layer; the receiving unit 901 is specifically used to receive the base layer and at least one enhancement layer of the first layered video coding segment of the second video frame transmitted by the electronic device in the first time segment in the second video transmission cycle according to the transmission priority of the base layer and at least one enhancement layer of the first layered video coding segment.

[0095] Optionally, the layered video coding segment includes a base layer and at least one enhancement layer; the receiving unit 901 is specifically used to receive the base layer and at least one enhancement layer of the first layered video coding segment transmitted by the electronic device on at least one other time segment after the first time segment.

[0096] Optionally, the layered video coding segment includes a base layer and at least one enhancement layer; the receiving unit 901 is specifically used to receive at least one enhancement layer of the first layered video coding segment transmitted by the electronic device on at least one other time segment after the first time segment.

[0097] Among them, all relevant contents of each operation involved in the above method embodiment can be referred to the functional description of the corresponding functional module, and will not be repeated here.

[0098] In this embodiment, the wireless screen projection device 90 is presented in the form of dividing various functional modules in an integrated manner. The "module" here can refer to a specific ASIC, circuit, processor and memory that executes one or more software or firmware programs, integrated logic circuit, and / or other devices that can provide the above functions. In a simple embodiment, those skilled in the art can imagine that the wireless screen projection device 90 can be used Figure 4 The form shown.

[0099] for example, Figure 4 The processor 201 can call the computer execution instructions stored in the memory 203 to enable the wireless screen projection device 90 to execute the wireless screen projection method in the above method embodiment.

[0100] For example, Figure 9 The functions / implementation processes of the receiving unit 901, the processing unit 902 and the sending unit 903 can be realized by Figure 4 The processor 201 in the memory 203 calls the computer execution instruction stored in the memory 203 to implement; or, Figure 9 The function / implementation process of the processing unit 902 can be achieved by Figure 4 The processor 201 in the memory 203 calls the computer execution instruction stored in the memory to implement, Figure 9 The function / implementation process of the receiving unit 901 can be achieved by Figure 4 This is achieved by the receiver of the communication interface 204 in FIG. Figure 9 The function / implementation process of the sending unit 903 can be achieved by Figure 4 This is achieved by the transmitter of the communication interface 204 in FIG.

[0101] Since the wireless screen projection device 90 provided in this embodiment can execute the above-mentioned wireless screen projection method, the technical effects that can be obtained can refer to the above-mentioned method embodiment and will not be repeated here.

[0102] Optionally, an embodiment of the present application further provides a wireless screen projection device (for example, the wireless screen projection device can be a chip or a chip system), which includes a processor and an interface, and the processor is used to read instructions to execute the method in any of the above method embodiments. In one possible design, the wireless screen projection device also includes a memory. The memory is used to store necessary program instructions and data, and the processor can call the program code stored in the memory to instruct the wireless screen projection device to execute the method in any of the above method embodiments. Of course, the memory may not be in the wireless screen projection device. When the wireless screen projection device is a chip system, it can be composed of chips, or it can include chips and other discrete devices. The embodiment of the present application does not specifically limit this.

[0103] Specifically, when Figure 8 The wireless projection device 80 shown is a mobile phone. Figure 9 When the wireless screen projection device 90 shown is a television, the sending unit 801 and the sending unit 903 can be transmitters when transmitting information, the receiving unit 803 and the receiving unit 901 can be receivers when transmitting information, and the transceiver unit can be a transceiver, and the transceiver, transmitter or receiver can be a radio frequency circuit. Figure 8 The wireless projection device 80 shown, Figure 9 When the wireless projection device 90 shown includes a storage unit, the storage unit is used to store computer instructions. The processor is in communication with the memory, and the processor executes the computer instructions stored in the memory, causing the wireless projection device to perform the method of the method embodiment. The processor can be a general-purpose central processing unit (CPU), a microprocessor, or an application-specific integrated circuit (ASIC).

[0104] when Figure 8 The wireless screen projection device shown, Figure 9 When the wireless screen projection device shown is a chip, the sending unit 801, the sending unit 903, the receiving unit 803 and the receiving unit 901 can be input and / or output interfaces, pins or circuits, etc. The processing unit 802 and the processing unit 902 can execute the computer execution instructions stored in the storage unit to enable the Figure 8 The wireless projection device 80 shown, Figure 9The chip in the wireless projection device 90 shown executes the method involved in the embodiment of the method. Optionally, the storage unit is a storage unit in the chip, such as a register, a cache, etc. The storage unit can also be a storage unit located outside the chip in the terminal device or network device, such as a read-only memory (ROM) or other type of static storage device that can store static information and instructions, a random access memory (RAM), etc.

[0105] In the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware, or any combination thereof. When implemented using a software program, all or part of the embodiments may be implemented in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of the present application are generated. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via a wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) method. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more media integrated therein. The available medium may be a magnetic medium (eg, a floppy disk, a hard disk, a magnetic tape), an optical medium (eg, a DVD), or a semiconductor medium (eg, a solid state disk (SSD)). In the embodiment of the present application, the computer may include the aforementioned device.

[0106] Although the present application is described herein in conjunction with various embodiments, in the process of implementing the claimed application, those skilled in the art can understand and implement other changes to the disclosed embodiments by reviewing the drawings, the disclosure, and the appended claims. In the claims, the word "comprising" does not exclude other components or steps, and "a" or "an" does not exclude multiple situations. A single processor or other unit can implement several functions listed in the claims. Certain measures are recorded in different dependent claims, but this does not mean that these measures cannot be combined to produce good results.

[0107] Although the present application has been described with reference to specific features and embodiments thereof, it is apparent that various modifications and combinations may be made thereto without departing from the spirit and scope of the present application. Accordingly, this specification and the drawings are merely illustrative of the present application as defined by the appended claims and are deemed to cover any and all modifications, variations, combinations or equivalents within the scope of the present application. Obviously, those skilled in the art may make various modifications and variations to the present application without departing from the spirit and scope of the present application. Thus, the present application is intended to include such modifications and variations as fall within the scope of the claims of the present application and their equivalents.

Claims

1. A wireless screen projection method, characterized in that: Applied to Wi-Fi communication systems, including: The electronic device transmits a layered video coding segment of a first video frame to the display device in a first video transmission period; The electronic device obtains the channel state information of the transmission of the first video frame fed back by the display device; The electronic device selectively initiates a retry mechanism based on the channel status information, wherein the retry mechanism is: the electronic device transmits a first layered video coding segment of the second video frame to the display device in a first time segment in a second video transmission cycle, and transmits a portion of the first layered video coding segment discarded in the first time segment to the display device in at least one other time segment after the first time segment, wherein the at least one other time segment respectively transmits an other layered video coding segment of the second video frame.

2. The wireless screen projection method according to claim 1, wherein: The layered video coding segment includes a base layer and at least one enhancement layer; The electronic device transmits the first layered video coding segment of the second video frame in the first time segment of the second video transmission cycle, including: the electronic device transmits the base layer and at least one enhancement layer of the first layered video coding segment of the second video frame to the display device in the first time segment of the second video transmission cycle according to the transmission priority of the base layer and at least one enhancement layer of the first layered video coding segment of the second video frame.

3. The wireless screen projection method according to claim 1, wherein: The layered video coded segment includes a base layer and at least one enhancement layer; the at least one other time segment includes a second time segment of the second video transmission period; the other layered video coded segment includes a second layered video coded segment of the second video frame; The electronic device transmitting, to the display device, in at least one other time segment after the first time segment, the portion of the first layered video coding segment discarded in the first time segment, comprising: The electronic device transmits the discarded part of the first layered video coding segment and the second layered video coding segment to the display device in the second time segment according to the transmission priority of the basic layer and at least one enhancement layer of the first layered video coding segment, and the transmission priority of the basic layer and at least one enhancement layer of the second layered video coding segment, wherein the transmission priority of the basic layer of the first layered video coding segment is higher than the transmission priority of the basic layer of the second layered video coding segment, the transmission priority of the basic layer of the second layered video coding segment is higher than the transmission priority of the enhancement layer of the first layered video coding segment, and the transmission priority of the enhancement layer of the first layered video coding segment is higher than the transmission priority of the enhancement layer of the second layered video coding segment.

4. The wireless screen projection method according to any one of claims 1 to 3, characterized in that: The discarded portion of the first layered video coded segment includes a base layer and at least one enhancement layer of the first layered video coded segment; The electronic device transmitting, to the display device, in at least one other time segment after the first time segment, the portion of the first layered video coding segment discarded in the first time segment, comprising: The electronic device transmits the base layer and at least one enhancement layer of the first layered video coding fragment to the display device in at least one other time segment after the first time segment.

5. The wireless screen projection method according to any one of claims 1 to 3, characterized in that: The discarded portion of the first layered video coded segment includes at least one enhancement layer of the first layered video coded segment; The electronic device transmitting, to the display device, in at least one other time segment after the first time segment, the portion of the first layered video coding segment discarded in the first time segment, comprising: The electronic device transmits at least one enhancement layer of the first layered video coding segment to the display device in at least one other time segment after the first time segment.

6. The wireless screen projection method according to any one of claims 1 to 3, characterized in that: Also includes: After the at least one other time segment, reconstructing a reference frame based on the successfully transmitted portion of the first layered video coding segment; A first layered video code of a third video frame is generated based on the reference frame.

7. A wireless screen projection method, characterized in that: Applied to Wi-Fi communication systems, including: The display device receives a layered video coded segment of a first video frame transmitted by the electronic device to the display device in a first video transmission period; The display device feeds back channel state information of transmitting the first video frame to the electronic device, wherein the electronic device selectively starts a retry mechanism according to the channel state information; The display device decodes the layered video coding segment transmitted by the electronic device in the first video transmission period; The display device receives a first layered video coding segment of a second video frame transmitted by the electronic device at a first time segment in a second video transmission period under the retry mechanism; The display device receives a portion of the first layered video coding segment discarded in the first time segment transmitted by the electronic device in at least one other time segment after the first time segment, wherein the at least one other time segment respectively transmits another layered video coding segment of the second video frame; The display device decodes the first layered video coding segment of the second video frame transmitted by the electronic device on the first time segment and the part of the first layered video coding segment discarded on the first time segment transmitted on at least one other time segment after the first time segment.

8. The wireless screen projection method according to claim 7, wherein: The layered video coding segment includes a base layer and at least one enhancement layer; The display device receives a first layered video coding segment of a second video frame transmitted by the electronic device in a first time segment in a second video transmission period, comprising: The display device receives the base layer and at least one enhancement layer of the first layered video coding fragment of the second video frame transmitted by the electronic device in the first time segment in the second video transmission cycle according to the transmission priority of the base layer and at least one enhancement layer of the first layered video coding fragment.

9. The wireless screen projection method according to claim 7 or 8, characterized in that: The layered video coding segment includes a base layer and at least one enhancement layer; The receiving, by the display device, a portion of the first layered video coding segment discarded in the first time segment transmitted by the electronic device in at least one other time segment after the first time segment, comprising: The display device receives a base layer and at least one enhancement layer of the first layered video coding segment transmitted by the electronic device in at least one other time segment after the first time segment.

10. The wireless screen projection method according to claim 7 or 8, characterized in that: The layered video coding segment includes a base layer and at least one enhancement layer; The receiving, by the display device, a portion of the first layered video coding segment discarded in the first time segment transmitted by the electronic device in at least one other time segment after the first time segment, comprising: The display device receives at least one enhancement layer of the first layered video coding segment transmitted by the electronic device in at least one other time segment after the first time segment.

11. A wireless screen projection device, characterized in that: Applied to Wi-Fi communication systems, including: a transmitter for transmitting a layered video coded segment of a first video frame to a display device during a first video transmission period; a receiver, configured to obtain channel state information of transmitting the first video frame fed back by the display device; A processor is used to selectively start a retry mechanism based on the channel state information obtained by the receiver, and the retry mechanism is: the transmitter is also used to transmit a first layered video coding segment of a second video frame to the display device in a first time segment in a second video transmission cycle, and transmit a part of the first layered video coding segment discarded in the first time segment to the display device in at least one other time segment after the first time segment, wherein the at least one other time segment respectively transmits an other layered video coding segment of the second video frame.

12. A wireless screen projection device, characterized in that: Applied to Wi-Fi communication systems, including: a receiver configured to receive a layered video coded segment of a first video frame transmitted by an electronic device to a display device in a first video transmission period; a transmitter, configured to feed back channel state information for transmitting the first video frame to the electronic device, wherein the electronic device selectively initiates a retry mechanism based on the channel state information; a processor configured to decode the layered video coding segment transmitted by the electronic device in the first video transmission period; The receiver is further configured to receive a first layered video coded segment of a second video frame transmitted by the electronic device in a first time segment in a second video transmission period in the retry mechanism; and receive a portion of the first layered video coded segment discarded by the electronic device in the first time segment and transmitted in at least one other time segment after the first time segment, wherein the at least one other time segment respectively transmits another layered video coded segment of the second video frame; The processor is also used to decode the first layered video coding segment of the second video frame transmitted by the electronic device on the first time segment and the discarded part of the first layered video coding segment on the first time segment transmitted on at least one other time segment after the first time segment.

13. A computer-readable storage medium, characterized in that Used to store a computer program, the computer program including instructions for executing the wireless screen projection method as described in any one of claims 1-6, and / or instructions for the wireless screen projection method as described in any one of claims 7-10.

14. A computer program product, characterized in that The computer program product includes: a computer program code, which, when executed on a computer, enables the computer to execute the wireless screen projection method as described in any one of claims 1 to 6, and / or the wireless screen projection method as described in any one of claims 7 to 10.

Images