A method for streaming media data transmission and related devices

Abstract

An embodiment of the present invention discloses a method and related devices for streaming media data transmission, characterized in that the method includes: determining M key frames in a target group of pictures (GOP); the target GOP includes L frames; M is an integer greater than 0, and L is an integer greater than 2; respectively performing hierarchical encoding on the source data of each of the M key frames to generate M base layer data and N enhancement layer data; the data quality of the enhancement layer data corresponding to each key frame is higher than the data quality of the base layer data; N is an integer greater than or equal to M; performing differential protection on the base layer data and the enhancement layer data of each key frame to obtain first encoded data of the M key frames. Using the embodiment of the present invention can reduce the delay in the process of streaming media data transmission.

Description

Technical Field

[0001] The present invention relates to the field of information processing, and particularly to a method for streaming media data transmission and related devices. Background Art

[0002] With the continuous development of Internet technology, the application of streaming media wireless screen mirroring technology is becoming more and more widespread. Streaming media refers to a media format that is continuously and real-time played on the network using streaming transmission technology, such as audio, video, or multimedia files, etc. Streaming media wireless screen mirroring technology refers to the technology of sending streaming media data from a source end to a destination end for display, such as Miracast, DLNA. Since video encoding, protocol processing, WiFi transmission, video decoding, etc. are involved in streaming media wireless screen mirroring technology, there is a streaming media wireless screen mirroring delay, that is, there is a certain time difference between the time when the source end displays an interface and the time when the same interface information reaches the destination end.

[0003] Currently, during the process of sending the interface information of the source end to the destination end, a large number of packet losses / discards may occur, resulting in problems such as a scrambled screen on the destination end. To avoid the occurrence of the above situation, in the prior art, the source end and the destination end use NACK or ACK methods to send retransmission requests, so as to ensure that the destination end can receive all the streaming media data completely. However, it is found in actual application that, due to using NACK or ACK feedback methods between the source end and the destination end, when packet loss occurs at the destination end, the source end will re-send the interface information to the destination end, which will bring additional delay and increase the streaming media wireless screen mirroring delay.

[0004] Therefore, how to reduce the streaming media data transmission delay to improve the user experience is an urgent problem to be solved. Summary of the Invention

[0005] The technical problem to be solved by the embodiments of the present invention is to provide a method for streaming media data transmission and related devices, which can reduce the streaming media data transmission delay and improve the user experience in streaming media wireless screen mirroring technology.

[0006] In a first aspect, an embodiment of the present invention provides a method for streaming media data transmission, which is characterized in that it is applied to a source end, and the method includes: determining M key frames in a target Group of Pictures (GOP); the target GOP includes L frames; M is an integer greater than 0, and L is an integer greater than 2; respectively performing hierarchical encoding on the source data of each of the M key frames to generate M base layer data and N enhancement layer data; the data quality of the enhancement layer data corresponding to each key frame is higher than the data quality of the base layer data; N is an integer greater than or equal to M; performing differential protection on the base layer data and the enhancement layer data of each key frame to obtain first encoded data of the M key frames; the first encoded data includes M base layer data after the differential protection and N enhancement layer data after the differential protection.

[0007] In an embodiment of the present invention, multiple key frames in a target GOP can be determined first, and then the source data of the multiple key frames can be processed by combining hierarchical encoding technology and differential protection technology to obtain encoded streaming media data, so as to reduce the occurrence of packet loss and retransmission events during the streaming media data transmission process, thereby reducing the streaming media data transmission delay. Specifically, hierarchical encoding is performed on the key frames to process the source data of the key frames into base layer data and enhancement layer data, and then differential protection is performed on the base layer data and the enhancement layer data respectively. Since the data quality of the base layer data is lower than that of the enhancement layer data, and the base layer data can be re-protected while the enhancement layer data can be lightly protected, the base layer data is less likely to be lost and in error compared with the enhancement layer data during the streaming media data transmission process. Even if the enhancement layer data is lost, the destination end can decode and play with the correct base layer data received. In summary, different from the problem in the prior art that directly transmits the information of key frames and requires the source end to retransmit data once packet loss occurs, in the streaming media data transmission of the embodiment of the present invention, by performing quality-domain hierarchical division on the source data of multiple key frames and performing differential protection on the data of different layers, it is ensured that the base layer data can reach the destination end. Even if the phenomenon of enhancement layer data loss occurs, the destination end can decode and play according to the base layer data, which can reduce the occurrence of retransmission events and packet loss phenomenon, thereby reducing the streaming media data transmission delay, and at the same time can also provide the destination end with streaming media data of different qualities for decoding and playing, effectively improving the user experience.

[0008] In a possible implementation manner, the determining M key frames in the target Group of Pictures (GOP) includes: obtaining the target Group of Pictures (GOP) and dividing the target GOP into M picture subgroups in time sequence; M is an integer greater than 0; respectively confirming the first frame in each picture subgroup as the key frame, so as to obtain the M key frames.

[0009] In the prior art, only the first frame (such as an I-frame) in the target picture group is used as a key frame, and then the other frames in the target picture group are sequentially forward-referenced based on this key frame. If this key frame has an error, it will cause errors in the frames of the entire target picture group. In the embodiments of the present invention, the target picture group can be grouped according to time sequence first, and further, the first frame in each group is used as a key frame to obtain multiple key frames. If one of the multiple key frames has an error, it will only affect some frames in the target picture group, reducing the possibility of errors in the streaming media data, thereby reducing the transmission delay of the streaming media data and effectively improving the user experience.

[0010] In a possible implementation manner, the method further includes: dividing each of the M picture subgroups into F frame sequences according to time sequence; F is an integer greater than 1; each of the F frame sequences includes at least two frames; respectively using the frame header of the k-th frame sequence among the F frame sequences as a reference frame with the frame header of one of the k-1 frame sequences arranged before the k-th frame sequence to obtain the first difference frames of the other F-1 frame sequences except the first frame sequence among the F frame sequences; k is an integer greater than 2 and less than or equal to F.

[0011] In the embodiments of the present invention, after the target picture group is divided into multiple picture subgroups according to time sequence, each picture subgroup can be further divided into multiple frame sequences according to time sequence. Then, the frame header of each frame sequence can use the key frame in this picture subgroup as a reference frame, or can use the already obtained first difference frame as a reference frame. Therefore, there is a more flexible reference relationship between different frames in the target picture group, reducing the possibility of errors in the streaming media data, thereby reducing the transmission delay of the streaming media data and effectively improving the user experience.

[0012] In a possible implementation manner, the method further includes: using the (i-1)-th frame as a reference frame for the i-th frame in each of the F frame sequences to obtain the i-th difference frame of each frame sequence; i is an integer greater than 1.

[0013] In the embodiments of the present invention, for the frames other than the first frame in each frame sequence, each can use its previous frame as a reference frame, thereby reducing the data volume of the streaming media data to a certain extent, reducing the transmission delay of the streaming media data, and effectively improving the user experience.

[0014] In a possible implementation, the differential protection of the base layer data and the enhancement layer data of each of the key frames to obtain the first encoded data of the M key frames includes: dynamically adding m redundant data packets to the base layer data corresponding to each of the key frames and dynamically adding n redundant data packets to the corresponding enhancement layer data, to generate the M base layer data and the N enhancement layer data after the differential protection, thereby obtaining the first encoded data; m is greater than n, m is an integer greater than 2, and n is an integer greater than 1.

[0015] In the embodiments of the present invention, the number of redundant data packets added to the base layer data or the enhancement layer data can be dynamically adjusted according to the current network conditions. If the current network conditions are poor, the number of redundant data packets can be appropriately increased. However, for the base layer data and the enhancement layer data of the same key frame, since the importance of the base layer data is higher than that of the enhancement layer data in the embodiments of the present invention, it is required that the number of redundant data packets added to the base layer data is greater than the number of redundant data packets added to the enhancement layer data, so that the base layer data is not easily lost or damaged, thereby ensuring the smoothness of the user's video picture, reducing the streaming media data transmission delay, and effectively improving the user experience.

[0016] In a possible implementation, the method further includes: obtaining target encoded data based on the first encoded data and the i-th differential frame and the corresponding first differential frame of each frame sequence in each picture group; sending the target encoded data to the destination end.

[0017] In the embodiments of the present invention, the target encoded data is obtained based on the key frame-related data after hierarchical encoding and differential protection and the related data of other frames except the key frames in the target picture group, and then the target encoded data is sent to the destination end, which is convenient for the destination end to decode and play, reduces the occurrence of packet loss and retransmission events, thereby reducing the streaming media data transmission delay, and at the same time can also provide streaming media data of different qualities for the destination end to decode and play, effectively improving the user experience.

[0018] In a possible implementation, the sending the target encoded data to the destination end includes: preferentially sending the M base layer data after the differential protection in the first encoded data, and then sending the other data in the target encoded data except the M base layer data after the differential protection.

[0019] In an embodiment of the present invention, different streaming media data are differentially scheduled. In the case of network radio interface congestion or tight system scheduling, it is necessary to ensure that the information with a base layer marker (such as M base layer data after differential protection) is preferentially transmitted, and then other information is sent. This can reduce the access delay of the radio interface, thereby reducing the streaming media data transmission delay and effectively improving the user experience.

[0020] In a possible implementation manner, the preferentially sending the M base layer data after the differential protection in the first encoded data includes: using the M base layer data after the differential protection as a header, and adding the other data in the target encoded data except the M base layer data after the differential protection to the M base layer data after the differential protection to obtain a low-delay transmission queue; and sending the target encoded data to the destination end based on the low-delay transmission queue.

[0021] In an embodiment of the present invention, the information with a base layer marker (such as M base layer data after differential protection) is used as the queue header, and then other information is added to the queue tail to obtain a low-delay transmission queue. Data transmission is performed based on this low-delay transmission queue, which can ensure that the information with a base layer marker is preferentially transmitted, and then other information is sent, reducing the access delay of the radio interface, thereby reducing the streaming media data transmission delay and effectively improving the user experience.

[0022] In a second aspect, an embodiment of the present invention provides a method for streaming media data transmission, which is characterized in that it is applied to a destination end, and the method includes: receiving target encoded data sent by a source end; the target encoded data includes data information of M key frames in a target picture group after hierarchical encoding and differential protection and data information of differential frames in the target picture group; and decoding the target encoded data to obtain the source data of the target picture group.

[0023] In an embodiment of the present invention, the destination end receives the encoded streaming media information sent by the source end, and then decodes and plays the encoded streaming media information according to requirements, reducing the occurrence of packet loss phenomena and retransmission events, thereby reducing the streaming media data transmission delay.

[0024] In a third aspect, an embodiment of the present invention provides a device for streaming data transmission, characterized in that the device includes: a first processing unit for determining M key frames in a target Group of Pictures (GOP); the target GOP includes L frames; M is an integer greater than 0, and L is an integer greater than 2; a first encoding unit for respectively performing hierarchical encoding on the source data of each of the M key frames to generate M base layer data and N enhancement layer data; the data quality of the enhancement layer data corresponding to each key frame is higher than that of the base layer data; N is an integer greater than or equal to M; a second processing unit for performing differential protection on the base layer data and the enhancement layer data of each key frame to obtain first encoded data of the M key frames; the first encoded data includes M base layer data after the differential protection and N enhancement layer data after the differential protection.

[0025] In a possible implementation manner, the first processing unit is specifically configured to: obtain the target Group of Pictures (GOP), and divide the target GOP into M picture groups in time sequence; M is an integer greater than 0; respectively confirm the first frame in each picture group as the key frame, so as to obtain the M key frames.

[0026] In a possible implementation manner, the device further includes: a third processing unit for dividing each of the M picture groups into F frame sequences in time sequence; F is an integer greater than 1; each frame sequence in the F frame sequences includes at least two frames; a fourth processing unit for respectively using the frame header of the kth frame sequence in the F frame sequences as a reference frame with the frame header of one of the k - 1 frame sequences preceding the kth frame sequence to obtain the first difference frames of the other F - 1 frame sequences except the first frame sequence in the F frame sequences; k is an integer greater than 2 and less than or equal to F.

[0027] In a possible implementation manner, the device further includes: a fifth processing unit for using the (i - 1)th frame as a reference frame for the ith frame in each frame sequence in the F frame sequences to obtain the ith difference frame of each frame sequence; i is an integer greater than 1.

[0028] In a possible implementation manner, the second processing unit is specifically configured to: dynamically add m redundant data packets to the base layer data corresponding to each key frame and n redundant data packets to the corresponding enhancement layer data respectively, to generate the M base layer data after the differential protection and the N enhancement layer data after the differential protection, so as to obtain the first encoded data; m is greater than n, m is an integer greater than 2, and n is an integer greater than 1.

[0029] In a possible implementation, the device further includes: a sixth processing unit, configured to obtain target encoded data based on the first encoded data, the i-th differential frame of each frame sequence in each picture group, and the corresponding first differential frame; a first sending unit, configured to send the target encoded data to a destination end.

[0030] In a possible implementation, the first sending unit is specifically configured to: preferentially send the base layer data of the M pieces in the first encoded data after differential protection, and then send other data in the target encoded data except for the base layer data of the M pieces after differential protection.

[0031] In a possible implementation, the first sending unit is specifically configured to: use the base layer data of the M pieces after differential protection as a header, add other data in the target encoded data except for the base layer data of the M pieces after differential protection to the base layer data of the M pieces after differential protection to obtain a low-latency transmission queue; and send the target encoded data to the destination end based on the low-latency transmission queue.

[0032] In a fourth aspect, an embodiment of the present invention provides a device for streaming media data transmission, characterized in that the device includes: a first receiving unit, configured to receive target encoded data sent by a source end; the target encoded data includes data information of source data of M key frames in a target picture group after hierarchical encoding and differential protection and data information of differential frames in the target picture group; a first decoding unit, configured to decode the target encoded data to obtain the source data of the target picture group.

[0033] In a fifth aspect, an embodiment of the present invention provides a chip system, characterized in that the chip system includes at least one processor, a memory, and an interface circuit, the memory, the interface circuit, and the at least one processor are interconnected by lines, and instructions are stored in the at least one memory; when the instructions are executed by the processor, the method according to any one of the first aspects is implemented.

[0034] In a sixth aspect, an embodiment of the present invention provides a chip system, characterized in that the chip system includes at least one processor, a memory, and an interface circuit, the memory, the interface circuit, and the at least one processor are interconnected by lines, and instructions are stored in the at least one memory; when the instructions are executed by the processor, the method according to the second aspect is implemented.

[0035] Seventh aspect, an embodiment of the present invention provides a computer storage medium, characterized in that the computer storage medium stores a computer program, and when the computer program is executed by a processor, the method described in any one of the above first aspects is implemented.

[0036] Eighth aspect, an embodiment of the present invention provides a computer storage medium, characterized in that the computer storage medium stores a computer program, and when the computer program is executed by a processor, the method described in the second aspect above is implemented.

[0037] Ninth aspect, an embodiment of the present invention provides a computer program, characterized in that the computer program includes instructions, and when the computer program is executed by a computer, the computer is caused to execute the method described in any one of the above first aspects.

[0038] Tenth aspect, an embodiment of the present invention provides a computer program, characterized in that the computer program includes instructions, and when the computer program is executed by a computer, the computer is caused to execute the method described in the second aspect above.

[0039] Eleventh aspect, an embodiment of the present invention provides an electronic device, characterized in that it includes a processor, a memory, and a communication interface, wherein the memory is used to store information sending program code, and the processor is used to call the streaming media data transmission device method program code to execute the method described in any one of the above first aspects.

[0040] Twelfth aspect, an embodiment of the present invention provides an electronic device, characterized in that it includes a processor, a memory, and a communication interface, wherein the memory is used to store information sending program code, and the processor is used to call the streaming media data transmission device method program code to execute the method described in any one of the above second aspects. Description of the Drawings

[0041] Figure 1A It is a schematic diagram of a screen mirroring protocol process in the prior art.

[0042] Figure 1B It is a schematic diagram of a transmission method for transmitting streaming media data provided by the prior art.

[0043] Figure 2A It is a schematic diagram of a wireless communication system architecture provided by an embodiment of the present invention.

[0044] Figure 2B It is a schematic block diagram of a video codec system 20 according to an embodiment of the present invention.

[0045] Figure 2C It is a framework diagram of a streaming media data transmission system provided by an embodiment of the present invention.

[0046] Figure 3A It is a schematic flowchart of a media data transmission method in an embodiment of the present application.

[0047] Figure 3B It is a schematic diagram of key frames of a target picture group provided by an embodiment of the present invention.

[0048] Figure 3C It is a schematic flowchart of a process for processing a target picture group provided by an embodiment of the present invention.

[0049] Figure 3D It is a schematic diagram of a frame sequence provided by an embodiment of the present invention.

[0050] Figure 3E It is a schematic flowchart of a key frame source data processing process provided by an embodiment of the present invention.

[0051] Figure 4A It is a schematic flowchart of another streaming media data transmission method provided by an embodiment of the present invention.

[0052] Figure 4B It is a schematic diagram of a low-latency transmission queue provided by an embodiment of the present invention.

[0053] Figure 4C It is a schematic diagram of constructing a low-latency transmission queue provided by an embodiment of the present invention.

[0054] Figure 5A It is a schematic diagram of a device for streaming media data transmission provided by an embodiment of the present invention.

[0055] Figure 5B It is a schematic diagram of another device for streaming media data transmission provided by an embodiment of the present invention. Detailed implementation manners

[0056] The terms used in the following embodiments of the present application are only for the purpose of describing specific embodiments and are not intended to limit the present application. As used in the specification and appended claims of the present application, the singular forms "a", "an", "the", "above-mentioned", "said", and "this" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It should also be understood that the term "and / or" used in the present application refers to and includes any or all possible combinations of one or more of the listed items.

[0057] First, analyze and propose the technical problems specifically to be solved by the present application. In the prior art, for the technology of streaming media data transmission, that is, the technology of sending the interface information at the source end to the destination end for display, the specific process is as follows:

[0058] SeeFigure 1A , Figure 1A is a schematic diagram of a screen mirroring protocol process in the prior art. In the figure, the screen mirroring process is from the Source end (such as a mobile phone) to the Sink end (such as a large screen). The data stream sequentially goes from the application side (Cast+) in the mobile phone to the protocol side (VTP), then to the WiFi chip, then to the WiFi chip on the large screen side, then to the protocol side (VTP), and finally to the application side (Cast+). When packet loss occurs, the sink end (such as the large screen) initiates a NACK request, and the Source end is responsible for retransmitting the lost packets. It should be noted that referring to Figure 1B , Figure 1B is a schematic diagram of a transmission method for transmitting streaming media data provided by the prior art. In the figure, the WiFi chip can provide VO queue, VI queue, BE queue, and BK queue transmission methods, but only the VI queue can be used for transmitting streaming media data in the prior art during the transmission process.

[0059] This solution can send streaming media data information from the source end to the destination end, but it has the following disadvantages:

[0060] Disadvantage 1: Using the NACK feedback method will occupy the air interface resources.

[0061] Disadvantage 2: When packet loss occurs at the destination end and a NACK is generated, causing the source end to retransmit, it will bring additional transmission delay.

[0062] Disadvantage 3: The destination end needs to receive a complete picture group before it can go to the decoder for decoding, which will bring additional delay.

[0063] Disadvantage 4: For streaming media data, only the VI queue can be used in the underlying WiFi transmission, and the VI queue does not perform differential scheduling on media data.

[0064] In summary, for the existing streaming media data transmission method, when the source end device sends streaming media data to the destination end device, it will increase additional delay, resulting in poor user experience. Therefore, the streaming media data transmission method provided in this application is used to solve the above technical problems.

[0065] The embodiments of this application will be described below with reference to the accompanying drawings.

[0066] Based on the above-mentioned technical problems, in order to facilitate the understanding of the embodiments of the present invention, the system architecture on which the embodiments of the present invention are based will be described below. Please refer to Figure 2A , Figure 2AIt is a schematic diagram of a wireless communication system architecture provided by an embodiment of the present invention. The wireless communication system 10 is used to reduce the streaming media data transmission delay in the streaming media wireless screen mirroring technology to improve the user experience. The system architecture may include a source device 101, a destination device 102, and an interference source 103. Among them,

[0067] The source device 101 can be distributed throughout the wireless communication system and can be stationary or mobile. In some embodiments of the present application, the source device 101 can be a mobile device. Common mobile devices include devices such as smartphones and tablets. In future communication systems, the source device 101 can also be the streaming media data transmission device in the present application, or a device configured to include the streaming media data transmission device in the present application. In the embodiment of the present invention, the source device 101 can be understood as a device with streaming media functions, such as a device with video playback functions, etc., and the source device 101 can establish a connection with other devices through the network. For example, when the source device 101 is mobile phone 1 and mobile phone 1 has video playback functions, when mobile phone 1 establishes a connection with other devices through the network, mobile phone 1 can project the screen to this device.

[0068] The destination device 102 can be distributed throughout the wireless communication system and can be stationary or mobile. In some embodiments of the present application, the destination device 102 can be a mobile device. Common mobile devices include some large-screen devices such as smart screens and tablets. In the embodiment of the present invention, the destination device 102 can be understood as a device with streaming media functions, such as a device with video playback functions, etc., and the destination device 102 can establish a connection with the source device 101 through the network. For example, when the source device 101 is mobile phone 1 and the destination device 102 is a large screen, when mobile phone 1 can establish a connection with the large screen through the network, mobile phone 1 can project the video picture played on mobile phone 1 onto the large screen for playback.

[0069] The interference source 103 can be understood as a device that affects the communication between the source device 101 and the destination device 102. For example, in a wireless communication system, the file transfer between mobile phone 2 and mobile phone 3 will interfere with the communication between mobile phone 1 and the large screen, and the communication between mobile phone 4 and the wireless access node will also interfere with the communication between mobile phone 1 and the large screen. It can be understood that the communication between other devices in this wireless communication system will interfere with the communication between mobile phone 1 and the large screen. It should be noted that after the streaming media information sent by the source device is affected by these external interferences, packet loss and other phenomena may occur, thus affecting the communication between the source device and the destination device.

[0070] It can be understood that Figure 2AA wireless communication system architecture among them is only an exemplary implementation manner in the embodiments of the present application. The wireless communication system architecture in the embodiments of the present application includes but is not limited to the above system architectures.

[0071] Based on the above-mentioned technical problems and its system architecture, in order to facilitate the understanding of the embodiments of the present invention, the video coding and decoding system on which the embodiments of the present invention are based will be described first below. Figure 2B It is a schematic block diagram of a video coding and decoding system 20 according to an embodiment of the present invention. As Figure 2B shown, the video coding and decoding system 20 includes a source device 21 and a destination device 22. The source device 21 generates encoded video data. Therefore, the source device 21 can be referred to as a video coding device or a video coding equipment. The destination device 22 can decode the encoded video data generated by the source device 21. Therefore, the destination device 22 can be referred to as a video decoding device or a video decoding equipment. The source device 21 and the destination device 22 can be examples of video coding and decoding devices or video coding and decoding equipments. The source device 21 and the destination device 22 can include a wide range of devices, including desktop computers, mobile computing devices, notebooks (e.g., laptops) computers, tablet computers, set-top boxes, handsets such as smart phones, televisions, cameras, display devices, digital media players, video game consoles, in-vehicle computers, or the like.

[0072] The destination device 22 can receive the encoded video data from the source device 21 via a channel 23. The channel 23 can include one or more media and / or devices capable of moving the encoded video data from the source device 21 to the destination device 22. In one example, the channel 23 can include one or more communication media that enable the source device 21 to directly transmit the encoded video data to the destination device 22 in real time. In this example, the source device 21 can modulate the encoded video data according to a communication standard (e.g., a wireless communication protocol), and can transmit the modulated video data to the destination device 22. The one or more communication media can include wireless and / or wired communication media, such as radio frequency (RF) spectrum or one or more physical transmission lines. The one or more communication media can form a packet-based network (e.g., a local area network, a wide area network, or a portion of a global network (e.g., the Internet). The one or more communication media can include routers, switches, base stations, or other devices that facilitate communication from the source device 21 to the destination device 22.

[0073] The destination device 22 can access the encoded video data via a standard data connection (e.g., an Internet connection). Example types of data connections include wireless channels (e.g., Wi-Fi connections), wired connections (e.g., DSL, cable modems, etc.), or a combination of both suitable for accessing the encoded video data stored on a file server. The transmission of the encoded video data from the file server can be streaming, download transmission, or a combination of both.

[0074] In Figure 2B an example, the source device 21 includes a video source 211, a video encoder 212, and an output interface 213. In some examples, the output interface 213 can include a modulator / demodulator (modem) and / or a transmitter. The video source 211 can include a video capture device (e.g., a video camera), a video archive containing previously captured video data, a video input interface for receiving video data from a video content provider, and / or a computer graphics system for generating video data, or a combination of the above video data sources.

[0075] The video encoder 212 can encode the video data from the video source 211. In some examples, the source device 21 directly transmits the encoded video data to the destination device 22 via the output interface 213. The encoded video data can also be stored on a storage medium or a file server for later access by the destination device 22 for decoding and / or playback.

[0076] In Figure 2B an example, the destination device 22 includes an input interface 221, a video decoder 222, and a display device 223. In some examples, the input interface 221 includes a receiver and / or a modem. The input interface 221 can receive the encoded video data via a channel 23. The display device 223 can be integrated with the destination device 22 or can be external to the destination device 22. Generally, the display device 223 displays the decoded video data. The display device 223 can include various display devices, such as a liquid crystal display (LCD), a plasma display, an organic light-emitting diode (OLED) display, or other types of display devices.

[0077] For example, as Figure 2C shown, Figure 2CA framework diagram of a streaming media data transmission system provided by an embodiment of the present invention. In the figure, MediaCodec, OMX_IF, and OMX_VENC are used for video layer coding. In the embodiment of the present invention, a dual-layer technology in the quality domain and the time domain is adopted, and the layering effect is achieved by setting QP frame by frame, separating the base layer and the extension layer data; Adaptive Forward Error Correction (AFEC) is used for differential protection of the base layer and the extension layer output by layer coding, realizing heavy protection for the base layer and light protection for the extension layer; A new LLT queue is added to the channel, which is used for differential scheduling according to the input parameters of the information source (such as I frame / P frame, base layer / extension layer, etc.), such as adjusting the aggregation degree, ensuring that the priority of the base layer is always higher than that of the extension layer.

[0078] It can be understood that Figure 2B One of the video codec systems in

[0079] The following describes the specific method framework based on the embodiment of the present invention. Refer to Figure 3A , Figure 3A which is a schematic flowchart of a media data transmission method in an embodiment of the present application. The following will be combined with the attached Figure 3A and based on the above Figure 2A and Figure 2B the source end in

[0080] Step S301: The source end determines M key frames in the target Group of Pictures (GOP).

[0081] Specifically, the target GOP includes L frames; M is an integer greater than 0, and L is an integer greater than 2. The source end can be understood as a video coding device or a video coding equipment. For example, the source end can be a smart phone that can generate encoded video data; the target Group of Pictures can be understood as a continuous picture. In the embodiment of the present invention, the first frame in the target Group of Pictures (such as the first key frame in the target Group of Pictures) can be understood as an I frame; an I frame can be understood as a complete picture; M key frames can be understood as important reference frames in the target Group of Pictures. The first key frame among the M key frames can be the I frame in the target Group of Pictures. The S-th key frame among the M key frames can use one of the key frames before the S-th key frame as a reference frame, where S is an integer greater than 2 and less than M. For example, as Figure 3B described,Figure 3B This is a schematic diagram of key frames of a target picture group provided by an embodiment of the present invention. In the figure, in a target picture group of 36 frames, 3 key frames can be included. The first key frame can be an I frame in the target picture group. The second key frame can use the first key frame as a reference frame. The third key frame can use the first key frame as a reference frame, or can use the second key frame as a reference frame.

[0082] In a possible implementation manner, the source end determines M key frames in the target picture group GOP, including: the source end obtains the target picture group GOP and divides the target GOP into M picture subgroups in time sequence; M is an integer greater than 0; the first frame in each of the picture subgroups is respectively confirmed as the key frame, so as to obtain the M key frames. Specifically, the M picture subgroups can be understood as multiple picture subgroups obtained by dividing the target GOP in time sequence, and M is an integer greater than 0, such as Figure 3C shown Figure 3C This is a schematic diagram of a process for processing a target picture group provided by an embodiment of the present invention. In the figure, after the source end obtains an 18-frame target picture group, it can divide these 18 frames into 2 picture subgroups in time sequence. Each picture subgroup can include 9 frames, and the first frame in each picture subgroup is used as a key frame. Therefore, there are a total of 2 key frames. It should be noted that in the embodiment of the present invention, the target picture group can be evenly divided, or the target picture group can be grouped according to actual needs. Therefore, the number of frames in each picture subgroup can be the same or different, which is not limited herein. In the embodiment of the present invention, the target picture group can be first grouped in time sequence, and further the first frame in each group is used as a key frame to obtain multiple key frames. If one of the multiple key frames is incorrect, it will only affect some frames in the target picture group, reducing the possibility of errors in streaming media data, thereby reducing the transmission delay of streaming media data and effectively improving the user experience.

[0083] In a possible implementation manner, the method further includes: the source end divides each of the M picture subgroups into F frame sequences in time sequence; F is an integer greater than 1; each of the F frame sequences includes at least two frames; the frame header of the kth frame sequence in the F frame sequences uses the frame header of one of the k-1 frame sequences before the kth frame sequence as a reference frame, so as to obtain the first difference frames of the other F-1 frame sequences except the first frame sequence in the F frame sequences; k is an integer greater than 2 and less than or equal to F. Specifically, the F frame sequences can be understood as multiple frame sequences obtained by dividing the picture subgroup in time sequence, and F is an integer greater than 1; the frame header of the kth frame sequence can be understood as the first frame in the kth frame sequence among the F frame sequences; the first difference frame can be understood as the first frame obtained based on the reference frame in each sequence. For example, such asFigure 3C As shown in the figure, after dividing the target picture group into 2 picture subgroups, each picture subgroup can be further divided into 3 frame sequences. The first frame in the first frame sequence of the picture subgroup is the key frame; the first frame in the second frame sequence uses the key frame as the reference frame; the first frame in the third frame sequence can use the key frame as the reference frame or the first difference frame of the second frame sequence as the reference frame, so as to obtain the first difference frame of each frame sequence in the picture subgroup. For another example, Figure 3D As shown in Figure 3D This is a schematic diagram of a frame sequence provided by an embodiment of the present invention. In the figure, a 36-frame GOP is divided into 3 picture subgroups in sequence, and then each picture subgroup is divided into 3 frame sequences in sequence. The first frame of the first frame sequence is the key frame. The first frame of the second frame sequence can obtain the first difference frame by referring to the key frame. The first frame of the third frame sequence can obtain the first difference frame of this sequence by referring to the key frame, or can also refer to the first difference frame of the third sequence of the first difference frame of the second frame sequence. In the embodiment of the present invention, after dividing the target picture group into multiple picture subgroups in sequence, each picture subgroup can be further divided into multiple frame sequences in sequence, and then the frame header of each frame sequence can use the key frame in this picture subgroup as the reference frame or the already obtained first difference frame as the reference frame. Therefore, there is a more flexible reference relationship between different frames in the target picture group, reducing the possibility of errors in streaming media data, thereby reducing the transmission delay of streaming media data and effectively improving the user experience.

[0084] In a possible implementation manner, the method further includes: the source end uses the (i - 1)-th frame as the reference frame for the i-th frame in each of the F frame sequences to obtain the i-th difference frame of each frame sequence; i is an integer greater than 1. In the embodiment of the present invention, for other frames in each frame sequence except the first frame, their previous frames can be used as reference frames, which can reduce the data volume of streaming media data to a certain extent, thereby reducing the transmission delay of streaming media data and effectively improving the user experience. For example, as Figure 3C shown in the figure, for each frame sequence, the 3rd frame can use the 2nd frame as the reference frame to obtain the 3rd difference frame; the 2nd frame can use the 1st frame as the reference frame to obtain the 2nd difference frame.

[0085] Step S302: The source end respectively performs hierarchical encoding on the source data of each of the M key frames to generate M base layer data and N enhancement layer data.

[0086] Specifically, the data quality of the extended layer data corresponding to each of the key frames is higher than that of the base layer data; N is an integer greater than or equal to M. The source data can be understood as the original data of the key frames; the base layer data can be understood as the data with lower quality obtained after processing the source data; the extended layer data can be understood as the data with higher quality obtained after processing the source data. For example, if the base layer data is the data capable of restoring a standard-definition video, the extended layer data can restore high-definition video data. For instance, as Figure 3E shown Figure 3E FIG. Figure 3E is a schematic diagram of a key frame source data processing flow provided by an embodiment of the present invention. After processing the source data of the key frames in the figure, base layer data and extended layer data are obtained. Based on the source data of the first key frame, 1 base layer data and 2 extended layer data are obtained, and based on the second key frame, 1 base layer data and 1 extended layer data are obtained. It should be noted that based on the source data of one key frame, one base layer data and one or more extended layer data can be obtained, and the number of extended layer data can vary according to actual situations and is not limited herein.

[0087] Step S303: The source end performs differential protection on the base layer data and the extended layer data of each of the key frames to obtain first encoded data of the M key frames.

[0088] Specifically, the first encoded data includes M base layer data after the differential protection and N extended layer data after the differential protection. Differential protection can be understood as adding redundant data packets to protect the base layer data and the extended layer data respectively on their own data.

[0089] In a possible implementation, the source end performs differential protection on the base layer data and the extension layer data of each of the key frames to obtain first encoded data of the M key frames, including: the source end dynamically adds m redundant data packets to the base layer data corresponding to each of the key frames and dynamically adds n redundant data packets to the corresponding extension layer data, generating the M base layer data after the differential protection and the N extension layer data after the differential protection, so as to obtain the first encoded data; m is greater than n, m is an integer greater than 2, and n is an integer greater than 1. Specifically, the number of redundant data packets added to the base layer data or the extension layer data can be dynamically adjusted according to the current network conditions. If the current network conditions are poor, the number of redundant data packets can be appropriately increased. However, for the base layer data and the extension layer data of the same key frame, since the importance of the base layer data is higher than that of the extension layer data in the embodiments of the present invention, it is required that the number of redundant data packets added to the base layer data is greater than the number of redundant data packets added to the extension layer data, so that the base layer data is not easily lost and damaged, thereby ensuring the smoothness of the user's video picture, reducing the streaming media data transmission delay, and effectively improving the user experience. For example, as Figure 3E shown, using the adaptive forward error correction technology, for the first key frame, 3 FEC packets are added to the base layer data, 2 FEC packets are added to the extension layer data 1, and 1 FEC packet is added to the extension layer data 2; for the second key frame, 3 FEC packets are added to the base layer data, and 1 FEC packet is added to the extension layer data 1. It should be noted that the number of redundant data packets added to the data can change according to the current network conditions. If the current network quality is poor, the number of redundant data packets can be appropriately increased. However, for the base layer data and the extension layer data corresponding to the same key frame, it is required that the number of redundant data packets added to the base layer data is greater than the number of redundant data packets added to the extension layer data.

[0090] See Figure 4A , Figure 4A which is a schematic flowchart of another streaming media data transmission method provided by the embodiments of the present invention. The following will be combined with the attached Figure 4A , and based on the above Figure 2A and Figure 2B the interaction sides of the source end and the destination end in to describe the streaming media data transmission method in the embodiments of the present application.

[0091] Step S401: The source end determines M key frames in the target group of pictures GOP.

[0092] Step S402: The source end respectively performs hierarchical encoding on the source data of each of the M key frames to generate M base layer data and N extension layer data.

[0093] Step S403: The source end performs differential protection on the base layer data and the enhancement layer data of each of the key frames to obtain first encoded data of the M key frames.

[0094] It should be noted that for the detailed descriptions of steps S401 - S403, please refer to the above steps S301 - S303.

[0095] Step S404: The source end obtains target encoded data based on the first encoded data, the i-th differential frame of each frame sequence in each picture group, and the corresponding first differential frame.

[0096] Specifically, the target encoded data is obtained based on the key frame related data that has undergone hierarchical encoding and differential protection and the related data of other frames in the target picture group except for the key frames.

[0097] Step S405: The source end sends the target encoded data to the destination end.

[0098] Specifically, the target encoded data is obtained based on the key frame related data that has undergone hierarchical encoding and differential protection and the related data of other frames in the target picture group except for the key frames, and then the target encoded data is sent to the destination end, which facilitates the destination end to perform decoding and playback, reduces the occurrence of packet loss phenomena and retransmission events, thereby reducing the streaming media data transmission delay, and at the same time can also provide the destination end with streaming media data of different qualities for decoding and playback, effectively improving the user experience.

[0099] In a possible implementation manner, the source end sending the target encoded data to the destination end includes: preferentially sending the base layer data of the M that have undergone the differential protection in the first encoded data, and then sending other data in the target encoded data except for the base layer data of the M that have undergone the differential protection. Specifically, differential scheduling is performed on different streaming media data. In the case of network air interface congestion or tight system scheduling, it is necessary to ensure that the information with the base layer mark (such as the base layer data of the M that have undergone differential protection) is preferentially transmitted, and then other information is sent, which can reduce the access delay of the air interface, thereby reducing the streaming media data transmission delay and effectively improving the user experience.

[0100] In a possible implementation, the step of preferentially sending the base layer data obtained by differentially protecting the M pieces of the first encoded data includes: the source end uses the base layer data obtained by differentially protecting the M pieces as the header, and adds the other data in the target encoded data except the base layer data obtained by differentially protecting the M pieces to the base layer data obtained by differentially protecting the M pieces to obtain a low-latency transmission queue; and sends the target encoded data to the destination end based on the low-latency transmission queue. In the embodiments of the present invention, the information with a base layer tag (such as the base layer data obtained by differentially protecting M pieces) is used as the queue header, and then other information is added to the queue tail to obtain a low-latency transmission queue. Data transmission based on this low-latency transmission queue can ensure that the information with a base layer tag is preferentially transmitted, and then other information is sent, reducing the access latency of the air interface, thereby reducing the streaming media data transmission latency and effectively improving the user experience. For example, as Figure 4B described, Figure 4B FIG. 1 is a schematic diagram of a low-latency transmission queue provided by an embodiment of the present invention. As shown in the low-latency transmission queue 1 in the figure, the data with a base layer tag is arranged at the queue head, and then the data with an extension layer tag can be placed at the queue tail. When transmitting, it can ensure that the data with a base layer tag is preferentially transmitted; as shown in the low-latency transmission queue 2 and the low-latency transmission queue 3, if the queue head is the data with an extension layer tag during the transmission process, but at this time there is a piece of data with a base layer tag that needs to be transmitted, the queue will be emptied and the data with the base layer tag will be added to the queue head, realizing differential scheduling and preferentially transmitting the data with the base layer tag.

[0101] Optionally, as Figure 4C shown, Figure 4C FIG. 2 is a schematic diagram of constructing a low-latency transmission queue provided by an embodiment of the present invention. In the figure, it can be first determined whether the tagged streaming media data is the data with a base layer tag. If so, the low-latency transmission (LLT) queue is emptied, and the data is placed at the head, and then the number of packets of the base layer data is checked to adjust the aggregation degree; if not, the data is placed at the tail of the LLT queue, and then the number of packets of the extension layer data is checked to adjust the aggregation degree.

[0102] Step S406: The destination end decodes the target encoded data to obtain the source data of the target picture group.

[0103] Specifically, the destination end receives the encoded streaming media information sent by the source end, and then decodes and plays the encoded streaming media information as required, reducing the occurrence of packet loss phenomena and retransmission events, thereby reducing the streaming media data transmission latency.

[0104] In an embodiment of the present invention, multiple key frames in a target picture group can be determined first, and then the source data of the multiple key frames can be processed by combining the hierarchical coding technology and the differential protection technology to obtain the encoded streaming media data, so as to reduce the occurrence of packet loss and retransmission events during the transmission of the streaming media data, thereby reducing the transmission delay of the streaming media data. Specifically, the key frames are hierarchically encoded, and the source data of the key frames is processed into base layer data and enhancement layer data, and then differential protection is performed on the base layer data and the enhancement layer data respectively. Since the data quality of the base layer data is lower than that of the enhancement layer data, and the base layer data can be re-protected, and the enhancement layer data can be lightly protected, the base layer data is less likely to be lost and in error than the enhancement layer data during the transmission of the streaming media data. Even in the case of loss of the enhancement layer data, the destination end can decode and play the correct base layer data. By implementing the method of the embodiment of the present invention, in the transmission of streaming media data, the problem that the source end needs to retransmit data once packet loss occurs when directly transmitting the information of the key frames is avoided, the occurrence of packet loss and retransmission events is reduced, thereby reducing the transmission delay of the streaming media data, and at the same time, different quality streaming media data can be provided for the destination end to decode and play, effectively improving the user experience.

[0105] The method of the embodiment of the present invention is described in detail above, and the related devices of the embodiment of the present invention are provided below.

[0106] Please refer to Figure 5A , Figure 5A FIG. is a schematic diagram of a device for transmitting streaming media data provided by an embodiment of the present invention. The streaming media data transmission device 50 may include a first processing unit 501, a first encoding unit 502, a second processing unit 503, a third processing unit 504, a fourth processing unit 505, a fifth processing unit 506, a sixth processing unit 507, and a first sending unit 508. The detailed descriptions of each module are as follows:

[0107] The first processing unit 501 is configured to determine M key frames in a target group of pictures (GOP); the target GOP includes L frames; M is an integer greater than 0, and L is an integer greater than 2;

[0108] The first encoding unit 502 is configured to hierarchically encode the source data of each of the M key frames respectively to generate M pieces of base layer data and N pieces of enhancement layer data; the data quality of the enhancement layer data corresponding to each key frame is higher than that of the base layer data; N is an integer greater than or equal to M;

[0109] A second processing unit 503, configured to perform differential protection on the base layer data and the enhancement layer data of each of the key frames to obtain first encoded data of the M key frames; the first encoded data includes M pieces of the base layer data after the differential protection and N pieces of the enhancement layer data after the differential protection.

[0110] In a possible implementation manner, the first processing unit 501 is specifically configured to: obtain the target group of pictures (GOP), and divide the target GOP into M picture subgroups in time sequence; M is an integer greater than 0; respectively confirm the first frame in each of the picture subgroups as the key frame, so as to obtain the M key frames.

[0111] In a possible implementation manner, the device further includes: a third processing unit 504, configured to divide each of the M picture subgroups into F frame sequences in time sequence; F is an integer greater than 1; each of the F frame sequences includes at least two frames; a fourth processing unit 505, configured to use the frame header of the kth frame sequence in each of the F frame sequences as a reference frame with the frame header of one of the k-1 frame sequences arranged before the kth frame sequence to obtain the first difference frames of the other F-1 frame sequences except the first frame sequence in the F frame sequences; k is an integer greater than 2 and less than or equal to F.

[0112] In a possible implementation manner, the device further includes: a fifth processing unit 506, configured to use the i-th frame in each of the F frame sequences as a reference frame with the (i-1)-th frame to obtain the i-th difference frame of each frame sequence; i is an integer greater than 1.

[0113] In a possible implementation manner, the second processing unit 503 is specifically configured to: dynamically add m redundant data packets to the base layer data corresponding to each of the key frames and dynamically add n redundant data packets to the corresponding enhancement layer data, to generate the M pieces of the base layer data after the differential protection and the N pieces of the enhancement layer data after the differential protection, so as to obtain the first encoded data; m is greater than n, m is an integer greater than 2, and n is an integer greater than 1.

[0114] In a possible implementation manner, the device further includes: a sixth processing unit 507, configured to obtain target encoded data based on the first encoded data, the i-th difference frame of each frame sequence in each of the picture subgroups, and the corresponding first difference frame; a first sending unit 508, configured to send the target encoded data to a destination end.

[0115] In a possible implementation, the first sending unit 508 is specifically configured to: preferentially send the base layer data obtained by performing differential protection on the M pieces of data in the first encoded data, and then send the other data in the target encoded data except for the base layer data obtained by performing differential protection on the M pieces of data.

[0116] In a possible implementation, the first sending unit 508 is specifically configured to: use the base layer data obtained by performing differential protection on the M pieces of data as a header, and add the other data in the target encoded data except for the base layer data obtained by performing differential protection on the M pieces of data to the base layer data obtained by performing differential protection on the M pieces of data to obtain a low-latency transmission queue; and send the target encoded data to the destination end based on the low-latency transmission queue.

[0117] It should be noted that for the functions of the functional units in the streaming media data transmission device 50 described in the embodiments of the present invention, reference can be made to the relevant descriptions of steps S301, S302, and S303 in the method embodiment described above Figure 3A and will not be elaborated here.

[0118] Please refer to Figure 5B , Figure 5B which is a schematic diagram of another streaming media data transmission device provided by the embodiments of the present invention. The streaming media data transmission device 60 may include a first receiving unit 601 and a first decoding unit 602. The detailed descriptions of each module are as follows:

[0119] The first receiving unit 601 is configured to receive the target encoded data sent by the source end; the target encoded data includes the data information obtained by performing hierarchical encoding and differential protection on the source data of M key frames in the target picture group and the data information of the differential frames in the target picture group;

[0120] The first decoding unit 602 is configured to decode the target encoded data to obtain the source data of the target picture group.

[0121] It should be noted that for the functions of the functional units in the streaming media data transmission device 60 described in the embodiments of the present invention, reference can be made to the relevant description of step S406 in the method embodiment described above Figure 4A and will not be elaborated here.

[0122] The embodiments of the present invention provide an electronic device, which is characterized by including a processor, a memory, and a communication interface. Among them, the memory is used to store information sending program code, and the processor is used to call the streaming media data transmission method program code to execute any one of the methods described above with the source end as the execution subject.

[0123] An embodiment of the present invention provides an electronic device, which is characterized by comprising a processor, a memory, and a communication interface. Wherein, the memory is used to store information sending program code, and the processor is used to call the program code of the streaming media data transmission method to execute the method described in any one of the above execution entities as the destination end.

[0124] The present application provides a chip system, which is characterized in that the chip system includes at least one processor, a memory, and an interface circuit. The memory, the interface circuit, and the at least one processor are interconnected by lines, and instructions are stored in the at least one memory. When the instructions are executed by the processor, the method described in any one of the above execution entities as the source end is realized.

[0125] The present application provides a chip system, which is characterized in that the chip system includes at least one processor, a memory, and an interface circuit. The memory, the interface circuit, and the at least one processor are interconnected by lines, and instructions are stored in the at least one memory. When the instructions are executed by the processor, the method described in any one of the above execution entities as the destination end is realized.

[0126] An embodiment of the present invention provides a computer storage medium, which is characterized in that the computer storage medium stores a computer program. When the computer program is executed by a processor, the method described in any one of the above execution entities as the source end is realized.

[0127] An embodiment of the present invention provides a computer storage medium, which is characterized in that the computer storage medium stores a computer program. When the computer program is executed by a processor, the method described in any one of the above execution entities as the destination end is realized.

[0128] The present application provides a computer program, which is characterized in that the computer program includes instructions. When the computer program is executed by a computer, the computer is caused to execute the method described in any one of the above execution entities as the source end.

[0129] The present application provides a computer program, which is characterized in that the computer program includes instructions. When the computer program is executed by a computer, the computer is caused to execute the method described in any one of the above execution entities as the destination end.

[0130] In the above embodiments, the descriptions of the respective embodiments have their own focuses. For parts not detailed in a certain embodiment, reference may be made to the relevant descriptions of other embodiments.

[0131] It should be noted that, for the foregoing method embodiments, for the sake of simple description, they are all expressed as a series of action combinations. However, those skilled in the art should know that this application is not limited by the described action sequence, because according to this application, certain steps may be implemented in other sequences or simultaneously. Secondly, those skilled in the art should also know that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily essential to this application.

[0132] In several embodiments provided by this application, it should be understood that the disclosed device can be implemented in other ways. For example, the device embodiments described above are only illustrative. For example, the above division of units is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed coupling or direct coupling or communication connection to each other can be through some interfaces. The indirect coupling or communication connection of the device or unit can be in an electrical or other form.

[0133] The units described above as separate components may or may not be physically separated. The components displayed as units may or may not be physical units, that is, they can be located in one place, or can be distributed to multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0134] In addition, each functional unit in the embodiments of this application can be integrated in a processing unit, or each unit can exist physically alone, or two or more units can be integrated in one unit. The above integrated unit can be implemented in the form of hardware or in the form of a software functional unit.

[0135] If the above integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of this technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc., specifically, the processor in the computer device) to execute all or part of the steps of the above methods in various embodiments of this application. Among them, the aforementioned storage medium can include: various media that can store program codes such as USB flash drives, mobile hard disks, magnetic disks, optical discs, read-only memory (ROM), or random access memory (RAM).

[0136] As described above, the above embodiments are only used to illustrate the technical solutions of this application and are not intended to limit them; although this application has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions recorded in the foregoing embodiments or perform equivalent replacements for some of the technical features; and these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the various embodiments of this application.

Claims

1. A method for streaming media data transmission, characterized in that, Applied to the source end, the method includes: Determine M key frames in the target Group of Pictures (GOP), including: obtaining the target GOP, dividing the target GOP into M picture subgroups in chronological order, and respectively identifying the first frame in each picture subgroup as the key frame, so as to obtain the M key frames; the target GOP includes L frames; M is an integer greater than 1, and L is an integer greater than 2; Respectively perform hierarchical coding on the source data of each of the M key frames to generate M base layer data and N enhancement layer data; the data quality of the enhancement layer data corresponding to each key frame is higher than the data quality of the base layer data; N is an integer greater than or equal to M; Perform differential protection on the base layer data and the enhancement layer data of each key frame to obtain the first encoded data of the M key frames, where the first encoded data includes M base layer data after the differential protection and N enhancement layer data after the differential protection; The method further includes: Divide each of the M picture subgroups into F frame sequences in chronological order; F is an integer greater than 1; each frame sequence in the F frame sequences includes at least two frames; Respectively use the frame header of the k-th frame sequence in the F frame sequences as a reference frame with the frame header of one of the k - 1 frame sequences before the k-th frame sequence to obtain the first difference frames of the other F - 1 frame sequences except the first frame sequence in the F frame sequences; k is an integer greater than or equal to 2 and less than or equal to F; Based on the first encoded data and the i-th difference frame and the corresponding first difference frame of each frame sequence in each picture subgroup, obtain the target encoded data; the i-th difference frame is the i-th difference frame obtained by using the (i - 1)-th frame as a reference frame for the i-th frame in each frame sequence of the F frame sequences; i is an integer greater than 1; Prioritize sending the M base layer data after the differential protection in the first encoded data, and then send the other data in the target encoded data except the M base layer data after the differential protection.

2. The method according to claim 1, characterized in that Performing differential protection on the base layer data and the enhancement layer data of each key frame to obtain the first encoded data of the M key frames, includes: Respectively and dynamically add m redundant data packets to the base layer data corresponding to each key frame and n redundant data packets to the corresponding enhancement layer data to generate the M base layer data after the differential protection and the N enhancement layer data after the differential protection, so as to obtain the first encoded data; m is greater than n, m is an integer greater than 2, and n is an integer greater than 1.

3. The method according to claim 1 or 2, characterized in that, The prioritizing sending the M base layer data after the differential protection in the first encoded data, includes: Taking the M pieces of base layer data after the differential protection as the header, adding the other data in the target encoded data except the M pieces of base layer data after the differential protection to the M pieces of base layer data after the differential protection, a low-latency transmission queue is obtained; Sending the target encoded data to the destination based on the low-latency transmission queue.

4. A method for streaming media data transmission, characterized in that, Applied to the destination, the method includes: After receiving the base layer data after differential protection in the target encoded data sent by the source, receiving the other data in the target encoded data except the base layer data after the differential protection; the target encoded data includes the data information of the source data of M key frames in the target group of pictures (GOP) after hierarchical encoding and differential protection and the data information of the difference frames in the target GOP; the M key frames are obtained by acquiring the target GOP and dividing the target GOP into M picture groups in sequence, and respectively identifying the first frame in each picture group as the key frame; Decoding the target encoded data to obtain the source data of the target GOP; the target encoded data is generated in the manner of obtaining the target encoded data according to Claim 1.

5. A device for streaming media data transmission, characterized in that, The device includes: A first processing unit, configured to determine M key frames in a target group of pictures (GOP), including: acquiring the target GOP, and dividing the target GOP into M picture groups in sequence, and respectively identifying the first frame in each picture group as the key frame, so as to obtain the M key frames; the target GOP includes L frames; M is an integer greater than 1, and L is an integer greater than 2; A first encoding unit, configured to respectively perform hierarchical encoding on the source data of each of the M key frames to generate M pieces of base layer data and N pieces of enhancement layer data; the data quality of the enhancement layer data corresponding to each key frame is higher than the data quality of the base layer data; N is an integer greater than or equal to M; A second processing unit, configured to perform differential protection on the base layer data and the enhancement layer data of each key frame to obtain the first encoded data of the M key frames, the first encoded data including M pieces of base layer data after the differential protection and N pieces of enhancement layer data after the differential protection; A third processing unit, configured to divide each of the M picture groups into F frame sequences in sequence; F is an integer greater than 1; each of the F frame sequences includes at least two frames; A fourth processing unit, configured to respectively use the frame header of the k-th frame sequence in the F frame sequences as a reference frame with the frame header of one of the k-1 frame sequences before the k-th frame sequence, to obtain the first difference frames of the other F-1 frame sequences except the first frame sequence in the F frame sequences; k is an integer greater than or equal to 2 and less than or equal to F; A sixth processing unit, configured to obtain target encoded data based on the first encoded data, the i-th differential frame of each frame sequence in each picture group, and the corresponding first differential frame; the i-th differential frame is obtained by using the (i-1)-th frame as a reference frame for the i-th frame in each of the F frame sequences, and the i-th differential frame of each frame sequence is obtained; i is an integer greater than 1; A first sending unit, configured to preferentially send the base layer data after differential protection of the M data in the first encoded data, and then send other data in the target encoded data except the base layer data after differential protection of the M data.

6. The device according to claim 5, characterized in that, The second processing unit is specifically configured to: Dynamically add m redundant data packets to the base layer data corresponding to each key frame and dynamically add n redundant data packets to the corresponding enhancement layer data, respectively, to generate the M base layer data after differential protection and the N enhancement layer data after differential protection, so as to obtain the first encoded data; m is greater than n, m is an integer greater than 2, and n is an integer greater than 1.

7. The device according to claim 5 or 6, characterized in that, The first sending unit is specifically configured to: Use the M base layer data after differential protection as a header, and add other data in the target encoded data except the M base layer data after differential protection to the M base layer data after differential protection to obtain a low-latency transmission queue; Send the target encoded data to the destination end based on the low-latency transmission queue.

8. A device for streaming data transmission, characterized in that, The device includes: A first receiving unit, configured to receive other data in the target encoded data except the base layer data after differential protection after receiving the base layer data after differential protection in the target encoded data sent by the source end; the target encoded data includes data information of the source data of M key frames in the target picture group after hierarchical encoding and differential protection and data information of the differential frames in the target picture group; the M key frames are obtained by acquiring the target picture group, dividing the target picture group into M picture groups in time sequence, and respectively identifying the first frame in each picture group as the key frame; A first decoding unit, configured to decode the target encoded data to obtain the source data of the target picture group; the target encoded data is generated in the manner of obtaining the target encoded data according to claim 1.

9. A chip system, characterized in that, The chip system includes at least one processor, a memory, and an interface circuit. The memory, the interface circuit, and the at least one processor are interconnected by lines, and instructions are stored in the memory; when the instructions are executed by the processor, the method according to any one of claims 1-3 is implemented.

10. A chip system, characterized in that, The chip system includes at least one processor, a memory, and an interface circuit. The memory, the interface circuit, and the at least one processor are interconnected by lines, and instructions are stored in the memory; when the instructions are executed by the processor, the method according to claim 4 is implemented.

11. A computer storage medium, characterized in that, The computer storage medium stores a computer program, and when the computer program is executed by a processor, it implements the method described in any one of the above-mentioned claims 1-3.

12. A computer storage medium, characterized in that, The computer storage medium stores a computer program, and when the computer program is executed by a processor, it implements the method described in claim 4 above.

13. A computer program product, characterized in that, The computer program product includes a computer program or instruction, and when the computer program or instruction is executed by a computer, it causes the computer to execute the method described in any one of claims 1-3.

14. A computer program product, characterized in that, The computer program product includes a computer program or instruction, and when the computer program or instruction is executed by a computer, it causes the computer to execute the method described in claim 4.

Images