Data processing method and device, equipment, chip and storage medium

Abstract

The invention discloses a data processing method and device, equipment, a chip and a storage medium, and belongs to the technical field of audio and video. The method comprises the following steps: acquiring a video data stream from an application layer; carrying out subpackaging operation on the video data stream by taking a coding frame in the video data stream as subpackaging granularity to obtain a plurality of video data packets; sending the plurality of video data packets to a receiving end; wherein each video data packet in the plurality of video data packets comprises at least one coding frame. For a large-scale video data stream, each coding frame is usually greater than the MTU, and each RTP packet obtained when the RTP protocol of the related technology is subpackaged is not greater than the MTU, so that compared with the RTP protocol of the related technology, the method and the device provided by the invention finally divide fewer video data packets, and the transmission efficiency is improved.

Description

Technical Field

[0001] The present application relates to the field of audio and video technology, and in particular to a data processing method, device, equipment, chip and storage medium. Background Art

[0002] With the development of technology, it is a common scenario for a transmitting device to send real-time data to a receiving device. The related technology provides the RTP (Real-time Transport Protocol) protocol, which is used to provide end-to-end real-time transmission services for multimedia data that needs to be transmitted in real time, such as voice, images, fax, etc.

[0003] In the related art, the RTP protocol is a transmission protocol for the network card. The MTU (Maximum Transmission Unit) of the network card transmission is usually 1500 bytes. The RTP protocol located in the transport layer of the sending device will perform packet operations on the received real-time data based on the size of the MTU.

[0004] In the related art, the RTP packets allocated by the RTP protocol cannot be larger than the MTU. For a large video data stream, the RTP protocol will be allocated with a large number of RTP packets, and the transmission rate will be slow. Summary of the invention

[0005] The present application provides a data processing method, device, equipment, chip and storage medium to improve transmission efficiency. The technical solution includes the following contents.

[0006] According to one aspect of the present application, a data processing method is provided, the method being executed by a sending end, the method comprising:

[0007] Get the video data stream from the application layer;

[0008] Taking the coded frames in the video data stream as the packet granularity, performing a packetization operation on the video data stream to obtain a plurality of video data packets;

[0009] Sending multiple video data packets to a receiving end;

[0010] Each of the multiple video data packets includes at least one coded frame.

[0011] According to one aspect of the present application, a data processing method is provided, the method being executed by a receiving end, the method comprising:

[0012] receiving a plurality of video data packets corresponding to a video data stream, wherein the plurality of video data packets are obtained by performing a packetization operation on the video data stream by a transmitter using a coding frame as a packetization granularity, and each of the plurality of video data packets contains at least one coding frame;

[0013] Extract payload data from multiple video packets.

[0014] According to one aspect of the present application, a data processing device is provided, the device running on a sending end, the device comprising:

[0015] An acquisition module, used for acquiring a video data stream from an application layer;

[0016] A packetization module is used to perform packetization operations on the video data stream using the coded frames in the video data stream as the packetization granularity to obtain multiple video data packets;

[0017] A sending module, used for sending multiple video data packets to a receiving end;

[0018] Each of the multiple video data packets includes at least one coded frame.

[0019] According to another aspect of the present application, a data processing device is provided, the device running on a receiving end, the device comprising:

[0020] A receiving module, used for receiving a plurality of video data packets corresponding to a video data stream, wherein the plurality of video data packets are obtained by performing a packetization operation on the video data stream by a transmitting end with a coded frame as a packetization granularity, and each of the plurality of video data packets contains at least one coded frame;

[0021] The extraction module is used to extract the payload data from the multiple video data packets.

[0022] According to another aspect of the present application, a communication device is provided, the communication device comprising a memory and a processor, the memory is used to store a computer program, and the processor is used to load and execute the computer program to implement the above data processing method.

[0023] According to another aspect of the present application, a computer-readable storage medium is provided, wherein the storage medium stores a computer program, and the computer program is loaded and executed by a processor to implement the above data processing method.

[0024] According to another aspect of the present application, a computer program product or a computer program is provided, the computer program product or the computer program comprising computer instructions, the computer instructions being stored in a computer-readable storage medium. A processor of a communication device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the communication device performs the above-mentioned data processing method.

[0025] The beneficial effects brought by the technical solution provided by the embodiment of the present application include at least:

[0026] In the packetization method of the RTP protocol used in the related art, each RTP packet does not exceed the MTU size, and thus, for a large video data stream, a large number of RTP packets will be segmented. However, the present application will perform the packetization operation based on the coded frame as the granularity to obtain multiple video data packets. For a large video data stream, each coded frame will usually be larger than the MTU, which will eventually result in the segmentation of fewer video data packets, and fewer video data packets need to be checked less frequently, which is beneficial to improving transmission efficiency. That is, the present application provides a packetization method suitable for large video data streams;

[0027] In addition, the present application uses coding frames as the packet granularity. When the receiving end cannot restore the video image of a certain coding frame, the receiving end can feedback to the sending end, and the sending end retransmits the data packet containing the coding frame. At this time, packet segmentation based on coding frames is beneficial to retransmitting based on coding frames as the granularity when an error occurs. For coding frames I frames, P frames, P frames, P frames..., the decoding of multiple P frames will depend on the same I frame. In the present application, if one of the P frames is wrong, only the P frame needs to be retransmitted, and the decoding of other P frames can proceed normally. In the related art, packet segmentation based on coding frames is not performed at the granularity. If a data packet is wrong, the content of the I frame may be retransmitted additionally (the actual error is the P frame data), and the decoding of other P frames cannot proceed normally. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0029] Figure 1 It is a flowchart of a data processing method at a sending end provided by an embodiment of the present application.

[0030] Figure 2 It is a schematic diagram of a packetization method of a video data stream provided by an embodiment of the present application.

[0031] Figure 3 It is a flowchart of a data processing method at a sending end provided by an embodiment of the present application.

[0032] Figure 4 It is a schematic diagram of a packetization method of an audio data stream provided in an embodiment of the present application.

[0033] Figure 5It is a schematic diagram of a first streaming media data packetization method provided by an embodiment of the present application.

[0034] Figure 6 It is a schematic diagram of the interaction between a sending end and a receiving end provided by an embodiment of the present application.

[0035] Figure 7 It is a schematic diagram of the interaction between a sending end and a receiving end provided by an embodiment of the present application.

[0036] Figure 8 It is a flowchart of a data processing method at a receiving end provided by an embodiment of the present application.

[0037] Fig. 9 It is a schematic diagram of the interaction between a sending end and a receiving end provided by an embodiment of the present application.

[0038] Fig.10 It is a schematic diagram of distributed transmission provided by an embodiment of the present application.

[0039] Fig.11 It is a schematic diagram of the data scatter phenomenon provided by an embodiment of the present application.

[0040] Fig.12 It is a structural block diagram of a data processing device provided by an embodiment of the present application.

[0041] Fig.13 It is a structural block diagram of a data processing device provided by an embodiment of the present application.

[0042] Fig.14 It is a structural block diagram of a data processing device provided by an embodiment of the present application.

[0043] Fig.15 It is a structural block diagram of a communication device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0044] In order to make the objectives, technical solutions and advantages of the present application clearer, the implementation methods of the present application will be further described in detail below with reference to the accompanying drawings.

[0045] First, the nouns involved in the embodiments of the present application are briefly introduced.

[0046] RTP (Real-time Transport Protocol) protocol: used to provide end-to-end real-time transmission services for various multimedia data that need to be transmitted in real time, such as voice, images, fax, etc.

[0047] In a typical application, the RTP protocol and the UDP (User Datagram Protocol) protocol jointly complete the protocol transmission of the transport layer. The RTP protocol is built on the UDP protocol. The RTP protocol is used to provide time stamps and other technologies to ensure the time accuracy of streaming media during real-time transmission, while the UDP protocol is responsible for transmitting data packets.

[0048] The RTP protocol's packetization methods include the following three modes: single packet mode, combined packet mode, and fragmented packet mode.

[0049] Single packet mode: For the NALU (Network Abstract Layer Unit) unit received from the upper layer, if the length of the NALU unit is less than the MTU, the NALU is encapsulated into a data packet separately.

[0050] Combined packet mode: If the NALU unit received from the upper layer is particularly small, multiple NALU units are encapsulated into one data packet, and the encapsulated data packet is no larger than the MTU.

[0051] Fragmented packet mode: If the NALU unit received from the upper layer exceeds the MTU, the NALU unit is split and encapsulated into multiple data packets.

[0052] It can be seen that the RTP protocol is packetized based on MTU, and the size of the RTP packet will not exceed MTU. For large real-time data, if the RTP protocol is used for packetization, more data packets will be divided, which makes the data packets at the network layer abnormally large, and also causes the sender and receiver to process the data packetization and consolidation more frequently, and the transmission efficiency of large real-time data is low. When the CPU (Central Processing Unit) frequency is not high, freezes, packet loss, screen distortion and other phenomena are often encountered.

[0053] Application layer: The application layer is the highest layer in the network structure. It directly faces users and applications and provides various network services and application functions. The application layer will format and parse the received streaming media data, including data encoding, decoding, parsing and presentation.

[0054] Transport layer: It is the layer below the application layer in the network structure and is used to provide logical communication between processes. After receiving the real-time data sent by the application layer, the transport layer will perform packet operations on the real-time data.

[0055] Figure 1A flowchart of a data processing method provided by an exemplary embodiment of the present application is shown, and the method is illustrated by taking the method executed by a transmitting end device as an example. The method includes:

[0056] Step 120, obtaining a video data stream from the application layer;

[0057] Video data stream refers to streaming media data whose multimedia type is video. In this application, the application layer of the sending end will encode the original video data and perform a series of operations to obtain the video data stream, and send the video data stream to the transport layer.

[0058] Optionally, the application layer uses the H264 protocol or the H265 protocol for encoding to obtain a video data stream. There are multiple coded frames in the encoded video data stream, and the multiple coded frames include I frames and P frames (there are no B frames in the streaming media data transmitted in real time). If the H264 protocol is used for encoding, each I frame contains SPS (Sequence Parameter Set) and PPS (Picture Parameter Set). If the H265 protocol is used for encoding, each I frame contains SPS, PPS and VPS (Video Parameter Set).

[0059] Step 140, using the coded frames in the video data stream as the packetization granularity, performing a packetization operation on the video data stream to obtain a plurality of video data packets;

[0060] Each of the multiple video data packets includes at least one coded frame.

[0061] In one embodiment, the transport layer of the transmitting end divides the video stream data transmitted by the application layer into coded frames to obtain I frames, P frames, P frames, P frames, P frames, etc. Optionally, each coded frame is divided into a data packet, that is, each data packet after division contains only one coded frame. Schematically, an I frame is divided into a first data packet, and a P frame is divided into a second data packet. For large video stream data, the sizes of I frames and P frames are relatively large, both larger than MTU (commonly 1500 bytes). Schematically, in combination with reference Figure 2 , Figure 2 It is shown that a video data stream 201 includes key frames and multiple non-key frames. For a non-key frame (P frame) 202, the non-key frame 202 is divided into a payload field in a data packet 203. A data packet 203 includes an interval field, a header field and a payload field.

[0062] In the packetization method using the RTP protocol in the related art, each RTP packet obtained does not exceed the MTU size, and thus, for a large video data stream, a large number of RTP packets will be segmented. However, this embodiment will separate the I frame into a data packet, and the size of the I frame is often much larger than the MTU, for example, the I frame is 1 megabyte. For a large video data stream, the P frame is also large and will also be larger than the MTU. Therefore, this application separates each I frame and each P frame into a data packet. For a large video data stream, fewer data packets will eventually be segmented, and fewer data packets need to be checked less times, which is conducive to improving transmission efficiency. That is, this application provides a packetization method suitable for large video data streams.

[0063] Optionally, for a more extreme case of the encoder in the application layer, the P frame of a large video data stream may also be smaller. Optionally, multiple P frames whose overall size does not exceed the second threshold are divided into one data packet, and P frames whose size is greater than the second threshold are divided into one data packet separately. Optionally, the size of the second threshold is 10M, and the second threshold is much larger than the MTU (the common MTU is 1500 bytes).

[0064] Optionally, the multiple P frames whose overall size does not exceed the second threshold refer to multiple P frames whose overall size is in a critical state, that is, if one more P frame is added, the overall size will exceed the second threshold, and the second threshold can be 10M, 1M, etc.

[0065] Optionally, for video stream data encoded using the H264 protocol, the SPS, PPS and I frame are divided into one data packet. Optionally, for video stream data encoded using the H265 protocol, the SPS, PPS, VPS and I frame are divided into one data packet;

[0066] Step 160: Send multiple video data packets to a receiving end.

[0067] The sending end performs a packetization operation on the video data stream to obtain multiple video data packets, and sends the multiple video data packets to the receiving end. Optionally, the lower network layer of the transport layer also performs certain processing on the multiple video data packets, and then the sending end sends the processed data packets to the receiving end. Optionally, the transport layer at the receiving end will perform unpacking and merging operations on the multiple video data packets.

[0068] Figure 3 A flowchart of a data processing method provided by an exemplary embodiment of the present application is shown, and the method is illustrated by taking the method executed by a transmitting end device as an example. The method includes:

[0069] Step 320, obtaining first streaming media data from the application layer;

[0070] Streaming media refers to a media format that is played on the Internet in a streaming manner. The first streaming media data can be a video data stream or an audio data stream. In this application, the application layer of the sending end will perform a series of operations such as encoding to obtain the first streaming media data and send the first streaming media data to the transport layer.

[0071] Optionally, the application layer uses H264 protocol or H265 protocol for encoding to obtain first streaming media data, and the first streaming media data is a video data stream.

[0072] Optionally, the application layer uses AAC (Advanced Audio Coding) for encoding to obtain first streaming media data, where the first streaming media data is an audio data stream. Each frame in the encoded audio data stream is an AAC structure, and the encoded audio data stream has multiple AAC structures, each AAC structure including an ADTS header field and an audio basic code stream.

[0073] Step 340, determining a first multimedia type corresponding to the first streaming media data;

[0074] The transport layer of the sending end device determines the multimedia type of the received first streaming media data and obtains a first multimedia type. The first multimedia type may be video or audio.

[0075] Step 360: using a first packetization method corresponding to the first multimedia type, perform a packetization operation on the first streaming media data to obtain a plurality of data packets.

[0076] For different types of first streaming media data, the transport layer will use different packetization methods for packetization.

[0077] In the case where the first multimedia type is video and the first streaming media data is a video data stream, a packetization operation is performed on the video data stream to obtain a plurality of video data packets, which has been introduced above and will not be described again here.

[0078] When the first multimedia type is audio and the first streaming media data is an audio data stream, the transport layer uses the AAC structure in the audio data stream as the packetization granularity to perform a packetization operation on the audio data stream to obtain multiple audio data packets; wherein each of the multiple audio data packets contains at least one AAC structure.

[0079] The audio data stream transmitted by the application layer has multiple AAC structures, each of which includes an ADTS header field and an audio elementary code stream. Generally, the AAC structure is small. In order to meet the real-time requirements of audio (users are more sensitive to the real-time nature of audio), multiple AAC structures whose overall size does not exceed the first threshold are divided into a third data packet, and for a single AAC structure whose size is greater than the first threshold, it will be divided into a fourth data packet separately. Optionally, the first threshold is MTU, which is 1500 bytes.

[0080] Optionally, the multiple AAC structures whose overall size does not exceed the first threshold refer to multiple AAC structures whose overall size is in a critical state, that is, if one more AAC structure is added, the overall size will exceed the first threshold, and the first threshold can be MTU, which can be 1500 bytes.

[0081] In an embodiment of the present application, an AAC structure larger than the MTU is allowed to be divided into a fourth data packet separately, while in the RTP protocol of the related technology, the RTP packet after packetization cannot be larger than the MTU. Therefore, for the audio data stream, the packetization method provided by the present application makes the capacity of each data packet larger, and ultimately fewer data packets will be obtained, and fewer data packets need to be checked fewer times, which is beneficial to improving transmission efficiency.

[0082] Indicative, with reference Figure 4 , Figure 4 An audio data stream 401 is shown, which includes a plurality of AAC structures 402 ( Figure 4 Only one AAC structure is shown in Figure 4 It shows that the AAC structure 402 is divided into the payload field of a data packet 403. An AAC structure includes an ADTS header structure and an audio elementary stream. A data packet 403 includes an interval field, a header field and a payload field.

[0083] In one embodiment, a plurality of poor quality AAC structures are screened out from the audio data stream through a neural network model, and the plurality of poor quality AAC structures are grouped into the same audio data packet. The poor quality AAC structure refers to an AAC structure that is easily lost during transmission.

[0084] In one training round of the neural network model, multiple sample AAC structures from the same audio are obtained, the multiple sample AAC structures are input into the neural network model, and the neural network model predicts a sample AAC structure with poor quality; and, a transmission test is performed on the multiple sample AAC structures, and the sample AAC structure with the most loss times in the transmission test is determined as the real sample AAC structure with poor quality;

[0085] According to the difference between the actual sample AAC structure with poor quality and the predicted sample AAC structure with poor quality, the model parameters of the neural network model are adjusted. For example, the actual sample AAC structure with poor quality includes sample AAC structures 1, 2, and 5, and the predicted sample AAC structure with poor quality includes sample AAC structures 1, 2, and 3. The model parameters of the neural network model are optimized by the gradient descent method.

[0086] Multiple training rounds are performed until the model parameters of the neural network model converge.

[0087] The above has introduced the method of dividing the first streaming media data. The first streaming media data can be a video data stream or an audio data stream.

[0088] Next, we will introduce how to encapsulate data packets.

[0089] Figure 5 The schematic diagram shows the transport layer of the sending end device encapsulating the first streaming media data after segmentation. After the transport layer of the sending end device obtains the original data (first streaming media data) 501, it cuts the original data 501 into multiple small blocks of data 502, which are the data obtained after the first streaming media data is segmented. The transport layer encapsulates each small block of data 502 to obtain multiple data packets 503.

[0090] Any one of the plurality of data packets 503 includes an interval field, a header field, and a payload field.

[0091] The start code is used to separate the current data packet from multiple data packets. The start code can be configured as any data string. For example, the start code is 0x01 0x02 0x03 0x04, a total of 4 bytes and 32 bits, indicating the start position of a data packet.

[0092] The header field (Header) is used to describe the multimedia type corresponding to the payload data of the current data packet, and to describe the length of the payload field. The payload data is the data obtained by segmenting the first streaming media data (i.e., small blocks of data). The header field includes a type field and a length field. The type field includes 1 byte, which is used to identify whether the current data packet is an audio data packet, a video data packet, or other data packet. Optionally, the character type of the type field is Char (character type). The length field includes 4 bytes, which are used to identify the length of the payload field. Optionally, the character type of the length field is Unsigned Int (unsigned integer).

[0093] The payload field (Payload) is used to carry payload data and to describe the offset position of the current data packet in multiple data packets. Schematically, the first eight bits of the payload field are the offset field, which is used to describe the position of the current data packet in multiple data packets. The payload field is used to carry the payload data after the shell is added. The shell is to prevent the payload data from having the same data string as the interval field. Therefore, special characters are inserted in the middle of the data string that is the same as the interval field contained in the payload data. For example, the original payload data contains "0x01 0x02 0x03 0x04", which is the same as the data string of the interval field. After the shell is added, it will be changed to "0x01 0x02 0x00 0x03 0x04". The length field in the header field is used to describe the actual length of the payload data after the shell is added. Optionally, in order to prevent data tampering, the entire payload field is also encrypted, or the sha1sum hash value is added to the entire payload field after negotiation with the receiving end.

[0094] In the encapsulation method provided in the present application, the data packet will carry description information (interval field, type, length and offset position). When a transmission error occurs, it can be repaired according to the description information to improve the transmission reliability and reduce the use of ACK to ensure transmission reliability, thereby increasing the transmission rate and reducing the transmission delay.

[0095] About the sending process:

[0096] Figure 6 A schematic diagram of a sending process of multiple data packets provided by an exemplary embodiment of the present application is shown. The sending end 601 initially sends a status request 603 (Request Status) to the receiving end 602, i.e., an inquiry message, which is used to inquire about the status of the receiving end 602. The receiving end 602 returns a status response 604 to the sending end 601.

[0097] When the inquiry message carries the SSRC (Synchronization Source) parameter of the first streaming media data, the receiving end 602 determines the most recent offset position from the local offset record, and returns the status response 604 to the sending end 601 carrying the most recent offset position, which is the offset position corresponding to the data packet of the first streaming media data most recently received by the receiving end 602.

[0098] In the case that the inquiry message does not carry the SSRC parameter, the receiving end 602 generates the SSRC parameter corresponding to the first streaming media data, and returns the status response 604 to the sending end 601 carrying the SSRC parameter.

[0099] After receiving the most recent offset position, the sending end 601 sends a data packet 606 corresponding to the next offset position of the most recent offset position to the receiving end 602 if there is a data packet corresponding to the most recent offset position in the local cache 605. After receiving the data packet 606, the receiving end 602 records the offset position of the data packet in the local offset record.

[0100] After receiving the most recent offset position, if there is no data packet corresponding to the most recent offset position in the local cache 605, the sending end 601 prompts the business layer to supplement the data packets between the most recent offset position and the current offset position of the local cache 605, and sends a data packet 606 corresponding to the next offset position of the most recent offset position to the receiving end 602. After receiving the data packet 606, the receiving end 602 records the offset position of the data packet 606 in the local offset record.

[0101] The local cache 605 may delete the data packet corresponding to the most recent offset position due to timeout or data packet being too large. If the data packet corresponding to the most recent offset position is deleted, the application layer of the transmitting end 601 will call the callback function to prompt the business layer to supplement the data packet between the most recent offset position and the current offset position of the local cache 605.

[0102] The sending end 601 continuously obtains the next data packet 607 and data packet 608 from the local cache 605 and sends them to the receiving end 602. After receiving the data packets 607 and 608, the receiving end 602 records the offset positions of the data packets 607 and 608 in the local offset record.

[0103] To summarize, in the sending process provided by the present application, by aligning the offset positions of the data packets at both ends of the sender and receiver, the sender can ensure that the data packets sent subsequently are correct, the sender can continuously send the next data packet, and the receiver can continuously receive data packets. The receiver does not need to feedback an ACK to the sender for each data packet received. The present application can reduce the use of ACK, thereby improving the transmission rate and reducing the transmission delay.

[0104] Regarding transmission errors:

[0105] After packetization by the method of the present application, multiple data packets are obtained. The multiple data packets are also transmitted through the TCP network layer. Errors in the TCP network layer include blocking. Therefore, in the present application, the receiving end also opens up a cache space for caching backup data to facilitate continued transmission.

[0106] For the first streaming media data of the video type, the receiving end caches a first number of GOPs (Group of Pictures), the first number is a preset value, that is, the size of the cache space of the receiving end is the first number of GOPs. For the first streaming media data of the audio type, the receiving end caches audio data corresponding to the first number of GOPs, that is, the size of the cache space of the receiving end is the size of the audio data corresponding to the first number of GOPs. Optionally, the first number is three, and two GOPs are separated by an I frame.

[0107] Figure 7 The figure shows a transmission continuation process when a transmission error occurs provided by an exemplary embodiment of the present application. The sending end 701 finds the next data packet 704 to be sent from the local cache 703 , and sends the data packet 704 to the receiving end 702 .

[0108] The receiving end 702 receives and parses the data packet 704. If the receiving end 702 cannot parse the correct offset position from the data packet 704 (i.e., the offset position does not match), the receiving end 702 will send a Rebase message (a rebase message carrying the correct offset position) 705 to the sending end 701. After receiving the correct offset position, the sending end 701 addresses and resends in the local cache 703, and sends a data packet 706 corresponding to the correct offset position to the receiving end 702. The receiving end 702 receives and parses the data packet 706, confirms that the offset position of the data packet 706 is correct (i.e., the offset position of the data packet matches), and the receiving end 702 records the offset position of the data packet 706 in the local offset record.

[0109] In one embodiment, the receiving end 702 will not store erroneous data packets (ie, data packets with mismatched offset positions) in the buffer space, so the receiving end 702 can tolerate a certain number of erroneous data packets.

[0110] In summary, in the case of transmission errors, the receiving end will send the offset position of the correct data packet to the sending end, and then the sending end will send the correct data packet. That is, in this application, the sending and receiving ends can correct the transmission error through the offset position of the data packet, and the receiving end does not need to feed back an ACK message to the sending end every time a data packet is received, thereby reducing the use of ACK, thereby increasing the transmission rate and reducing the transmission delay.

[0111] Next, we will introduce the content on the receiving end.

[0112] Figure 8 A flowchart of a data processing method provided by an exemplary embodiment of the present application is shown, and the method is illustrated by taking the method being executed by a receiving end device as an example. The method includes:

[0113] Step 810, continuously reading multiple data packets;

[0114] The multiple data packets are obtained by the sending end device performing a packetization operation according to the first multimedia type corresponding to the first streaming media data. Optionally, the first streaming media data is a video data stream, in which case the multiple data packets are multiple video data packets; optionally, the first streaming media data is an audio data stream, in which case the multiple data packets are multiple audio data packets. Any data packet in the multiple data packets includes an interval field, a header field, and a payload field. For a detailed introduction to the multiple data packets, please refer to the content on the sending end side above.

[0115] The transport layer of the receiving device continuously reads data from the first socket interface. After reading the data, the receiving device merges the data into the local cache space. In one embodiment, for the first streaming media data of the video type, the receiving end caches a first number of GOPs (Group of Pictures), and the first number is a preset value, that is, the size of the cache space is the first number of GOPs. For the first streaming media data of the audio type, the receiving end caches the audio data corresponding to the first number of GOPs. That is, the size of the cache space is the size of the audio data corresponding to the first number of GOPs. Optionally, the first number is three, and two GOPs are separated by an I frame.

[0116] The receiver searches for the interval field in the local buffer space. The interval field is the start mark of a data packet. For example, if the interval field is configured as 0x01 0x02 0x03 0x04, the receiver will continue to search for the data string of 0x01 0x02 0x03 0x04.

[0117] Step 820, when the interval field is read, continue to read the header field;

[0118] When the receiving end reads the interval field, it will continue to read the header field. The length of the header field is fixed and optional, which is 5 bytes. Optionally, the header field includes a type field (1 byte) and a length field (4 bytes). If the header field cannot be read completely, the receiving end will continue to read the first Socket interface until the complete header field is read.

[0119] Step 830, continue reading the payload field according to the length of the payload field recorded in the header field;

[0120] The receiving end continues to read the payload field according to the length of the payload field recorded in the header field.

[0121] Step 840, extracting the payload data carried in the payload field when the offset position recorded in the payload field matches;

[0122] The payload field includes an offset field and payload data. The length of the offset field is fixed and optional, which is 8 bits. The receiving end reads the offset position recorded in the offset field, and extracts the payload data carried in the payload field when the offset position matches. The payload data is part of the first streaming media data, and the offset position is used to describe the position of the current data packet in multiple data packets.

[0123] In one embodiment, when the offset position matches, the receiving end needs to perform an unpacking operation on the payload data after the unpacking operation. In contrast to the unpacking operation, the unpacking operation is to find the target data string and remove the pre-inserted special characters from the target data string. The target data string is a data string with special characters inserted in the data string of the interval field. For example, the target data string is 0x01 0x02 0x00 0x03 0x04, and the special character inserted by the sending end is 0x00. Therefore, after the unpacking operation, 0x01 0x02 0x03 0x04 is obtained.

[0124] Step 850: When the offset position recorded in the payload field does not match, a rebase message is sent to the sending end, and the rebase message is used to instruct the sending end to continue transmitting the data packet.

[0125] The receiving end reads the offset position recorded in the offset field, and sends a rebase message (Rebase message) to the sending end if the offset position does not match. The rebase message is used to instruct the sending end to continue transmitting the data packet. Optionally, the rebase message carries the correct offset position, and the sending end continues to transmit the data packet corresponding to the correct offset position according to the indicated correct offset position.

[0126] In summary, the receiving device will continue to read the header field according to the received interval field; and continue to read the payload field according to the length of the payload field indicated in the header field, that is, the present application provides a method for reading a data packet. The packet structure of the data packet is the packet structure used by the peer device (sending device) when encapsulating the packet.

[0127] Moreover, in the present application, the receiving device sends a rebase message to the sending device only when the offset position does not match, instead of feeding back an ACK message for each data packet received in the related art. The method of repairing errors (offset position mismatch) through messages provided by the present application can effectively reduce the use of ACKs, thereby reducing transmission delays. The reliability of transmission is ensured by the method of reading data packets provided by the present application, instead of feeding back an ACK message for each data packet received to ensure reliability.

[0128] In one embodiment, if the current data packet obtained by the receiving end from the two interval fields contains useless data and / or empty packet data, the receiving end will use the PING-PONG mechanism to send a PING message to the sending end. When the PONG message returned by the sending end is received, the receiving end confirms that the current data packet is legal and continues to read data. When the PONG message sent by the sending end is not received, the receiving end disconnects from the sending end to protect its own system.

[0129] Indicative, Fig. 9 A schematic diagram showing the PING-PONG mechanism.

[0130] The sending end 901 reads the next data packet 904 from the local cache 903 and sends the data packet 904 to the receiving end 902. The receiving end 902 reads the data packet 904 and parses it to find that there is abnormal data (useless data and / or empty packet data) in the data packet 904. The receiving end 902 sends a PING message 905 to the sending end 901, and the sending end 901 returns a PONG message 906 to the receiving end within a specified time. The receiving end 902 confirms that the data packet 904 is legal based on the received PONG message 906 and continues to read data.

[0131] In the present application, there is a PING-PONG sniffing mechanism between the sending device and the receiving device. The reliability of transmission is guaranteed by the PING-PONG sniffing mechanism instead of using ACK messages to ensure the reliability of transmission. This can effectively reduce the use of ACKs and thus reduce transmission delays.

[0132] In one embodiment, multiple data packets are sent in a distributed manner, that is, multiple sending ends send multiple data packets of the first streaming media data to the receiving end in a distributed manner. Fig.10 As shown, the multiple transmitting ends include a master transmitting end 1001 and at least one slave transmitting end 1002 .

[0133] In the process of distributed transmission, initially, the main transmitting end 1001 sends a status request 1004, i.e., an inquiry message, to the receiving end 1003 to inquire about the status of the receiving end 1003, and the receiving end 1003 returns a status response 1005 to the main transmitting end 1001. Optionally, the status request 1004 includes distributed prompt information to prompt the receiving end 1003 that this transmission is a distributed transmission task, and the receiving end 1003 will open a cache space to temporarily store the data packets sent by each transmitting end in the cache space. Optionally, the size of the cache space is (first number*GOP*number of transmitting ends), and optionally, the first number is three, schematically, Fig.10In the example, there are only two senders, and the size of the buffer space is (3*GOP*2).

[0134] After receiving the status response 1005, the master sender 1001 will coordinate with the slave sender 1002 to perform parallel data transmission (distributed data transmission). The master sender 1001 sends a master synchronization task 1009 to the slave sender 1002, instructing the slave sender 1002 to send a data packet 1010 from multiple data packets, and the master sender 1001 itself also finds the data packet 1007 to be sent to the receiving end 1003 from the local cache 1006 and sends it. After receiving the master synchronization task 1009, the slave sender 1002 sends the data packet 1010 to the receiving end 1003 according to the instruction of the master synchronization task 1009, and returns a slave synchronization task 1011 to the master sender 1001, informing the master sender 1001 that the corresponding data packet has been sent and waiting for the next master synchronization task 1012.

[0135] Due to asynchronous transmission, the offset positions of data packets sent by different senders are different, and data scattering may occur. Data scattering refers to the phenomenon that the received data packets are discontinuous.

[0136] In the process of receiving multiple data packets, if the received data packets continue to have data scattering phenomenon, the receiving end 1003 will send a filling data message to the main sending end 1001. The filling data message is used by the main sending end 1001 to send or direct at least one data packet missing from the sending end 1002 due to the data scattering phenomenon.

[0137] Indicative, Fig.11 The figure shows the change process of the buffer space at the receiving end. Fig.11 Part (A) shows that, in the initial state, a data packet of "offset position 0, length 1024" is stored in the buffer space of the receiving end, and the receiving end receives a new data packet of "offset position 2048, length 1024".

[0138] exist Fig.11 In part (B), the receiving end stores the new data packet in the buffer space and removes the data packet at "offset position 0, length 1024". At this time, the receiving end finds that it has received the data packet at "offset position 2048, length 1024", but has not received the data packet at "offset position 1024, length 1024". After a period of time, if the receiving end still does not receive the data packet at "offset position 1024, length 1024", the receiving end will send a padding data message to the main sending end to fill the data packet at "offset position 1024, length 1024". The main sending end sends or directs the slave sending end to send the data packet at "offset position 1024, length 1024".

[0139] In one case, due to asynchronous processing, a missing data packet that was originally to be sent may be received while waiting for the missing data packet corresponding to the fill data message. The receiving device will also perform a de-redundancy operation after receiving two missing data packets and remove one of the data packets.

[0140] In summary, the present application provides a distributed sending method, and the distributed sending (asynchronous sending) can improve the transmission efficiency of multiple data packets corresponding to the first streaming media data.

[0141] In one embodiment, in the case of a distributed task, the receiving device will continuously read data packets sent by multiple sending ends from the first socket interface. After reading the data, the receiving device merges the data into the local cache space, and the receiving device reads the data packets from the local cache space.

[0142] In one embodiment, in the case of a non-distributed task, the receiving end will continuously read multiple data packets sent by the first sending end from the first socket Socket interface, and bind the first Socket interface and the first sending end, so that the first Socket interface will be blocked when other sending ends try to continue to transmit multiple data packets (corresponding to the first streaming media data). When the first sending end completes the transmission of multiple data packets (corresponding to the first streaming media data), the binding relationship between the first sending end and the first Socket interface will be untied.

[0143] Fig.12 A data processing device 1200 provided by an exemplary embodiment of the present application is shown. The data processing device 1200 runs on a sending end. The data processing device 1200 includes:

[0144] An acquisition module 1201 is used to acquire a video data stream from an application layer;

[0145] A packetization module 1202 is used to perform a packetization operation on the video data stream using the coded frames in the video data stream as the packetization granularity to obtain multiple video data packets;

[0146] The sending module 1203 is used to send multiple video data packets to the receiving end;

[0147] Each of the multiple data packets contains at least one coded frame.

[0148] In an optional embodiment, the encoded frame includes an I frame or a P frame; the packetization module 1202 is further used to divide each I frame in the video data stream into a first data packet separately; and each P frame in the video data stream into a second data packet separately.

[0149] In an optional embodiment, the acquisition module 1201 is further used to acquire an audio data stream from the application layer;

[0150] The packetization module 1202 is further used to perform packetization operations on the audio data stream using the Advanced Audio Coding AAC structure in the audio data stream as the packetization granularity to obtain multiple audio data packets; wherein each of the multiple audio data packets contains at least one AAC structure.

[0151] In an optional embodiment, the packetization module 1202 is further used to divide the AAC structures in the audio data stream into a third data packet, wherein the AAC structures whose overall size does not exceed the first threshold are divided into a third data packet, and the AAC structures whose size is greater than the first threshold are divided into a fourth data packet separately.

[0152] In an optional embodiment, any one of the plurality of video data packets includes an interval field, a header field, and a payload field;

[0153] An interval field, used to separate a current video data packet from multiple video data packets;

[0154] The header field is used to describe the multimedia type corresponding to the payload data of the current video data packet, and to describe the length of the payload field, where the payload data is the data obtained by performing a packetization operation on the video data stream;

[0155] The payload field is used to carry payload data and to describe the offset position of the current video data packet in multiple video data packets.

[0156] In an optional embodiment, the sending module 1203 is further configured to send an inquiry message to the receiving end;

[0157] The acquisition module 1201 is further configured to acquire, when the query message carries a synchronization source SSRC parameter corresponding to the video data stream, a most recent offset position determined by the receiving end from the offset record, where the most recent offset position is an offset position corresponding to a video data packet most recently received by the receiving end;

[0158] The sending module 1203 is further configured to send a video data packet corresponding to a next offset position to the receiving end based on the most recent offset position.

[0159] In an optional embodiment, the sending module 1203 is further configured to send a video data packet corresponding to a next offset position of the most recent offset position to the receiving end when a video data packet at the most recent offset position exists in the local cache of the sending end.

[0160] Combined with reference Fig.13In an optional embodiment, the device also includes a prompt module 1204, which is used to prompt the business layer to supplement the video data packet between the most recent offset position and the current offset position of the local cache when there is no video data packet at the most recent offset position in the local cache of the sending end; the sending module 1203 is also used to send the video data packet corresponding to the next offset position of the most recent offset position to the receiving end.

[0161] In an optional embodiment, the sending module 1203 is further used to send an inquiry message to the receiving end. The device also includes a receiving module 1205, which is further used to obtain the SSRC parameter corresponding to the first streaming media data generated by the receiving end when the inquiry message does not carry the SSRC parameter.

[0162] In an optional embodiment, the receiving module 1205 is further configured to receive a message carrying a correct offset position, where the message carrying the correct offset position is sent by the receiving end when the receiving end is unable to parse the correct offset position from an erroneous video data packet sent by the sending end;

[0163] The sending module 1203 is further configured to send the video data packet corresponding to the correct offset position to the receiving end.

[0164] In an optional embodiment, the receiving end buffers a first number of groups of pictures GOP, where the first number is a preset value.

[0165] In summary, in the packetization method of the RTP protocol used in the related art, each RTP packet does not exceed the MTU size, and thus, for a large video data stream, a large number of RTP packets will be segmented. However, this application will perform the packetization operation based on the coded frame as the granularity to obtain multiple video data packets. For a large video data stream, each coded frame will usually be larger than the MTU, which will eventually result in fewer video data packets being segmented. Fewer video data packets require fewer checks, which is beneficial to improving transmission efficiency. That is, this application provides a packetization method suitable for large video data streams;

[0166] In addition, the present application uses coding frames as the packetization granularity. When the receiving end cannot restore the video image of a certain coding frame, the receiving end can feedback to the sending end, and the sending end retransmits the data packet containing the coding frame. At this time, packetization based on coding frames is beneficial to retransmission based on coding frames as the granularity when an error occurs.

[0167] Fig.14 1 is a structural block diagram of a data processing device 1400 provided by an exemplary embodiment of the present application. The data processing device 1400 runs on a receiving end, and the device includes:

[0168] The receiving module 1401 is used to receive multiple video data packets corresponding to the video data stream. The multiple video data packets are obtained by the transmitter performing a packetization operation on the video data stream with the coding frame as the packetization granularity. Each of the multiple video data packets contains at least one coding frame.

[0169] The extraction module 1402 is used to extract payload data from multiple video data packets.

[0170] In an optional embodiment, any one of the plurality of video data packets includes an interval field, a header field, and a payload field. The apparatus further includes a reading module 1403, the reading module 1403 being configured to continuously read the plurality of video data packets;

[0171] When the interval field is read, continue reading the header field;

[0172] Continue reading the payload field according to the length of the payload field recorded in the header field;

[0173] The extraction module 1402 is further used to extract the payload data carried in the payload field when the offset position recorded in the payload field matches, where the payload data is part of the data in the video data stream, and the offset position is used to describe the position of the current video data packet in the multiple video data packets;

[0174] The device also includes a sending module 1404, which is used to send a rebase message to the sending end when the offset position recorded in the payload field does not match, and the rebase message is used to instruct the sending end to continue transmitting the video data packet.

[0175] In an optional embodiment, the sending module 1404 is further configured to send a PING message to the sending end when useless data and / or empty packet data exist in the read current video data packet;

[0176] The data processing device 1400 further includes a confirmation module 1405. The confirmation module 1405 is configured to confirm that the current video data packet is legal when receiving a PONG message sent by the sending end;

[0177] The processing module 1406 is used to disconnect the connection if the PONG message sent by the sending end is not received.

[0178] In an optional embodiment, the receiving end supports multiple sending ends to jointly send the first streaming media data in a distributed manner, and the multiple sending ends include a master sending end and at least one slave sending end.

[0179] The sending module 1404 is also used to send a filling data message to the main sending end if the received video data packets continue to have data scattering phenomenon during the process of receiving multiple video data packets. The filling data message is used by the main sending end to send or instruct at least one slave sending end to send the missing video data packets due to the data scattering phenomenon.

[0180] In an optional embodiment, the data processing device 1400 further includes a binding module 1407. The reading module 1403 is further used to continuously read multiple video data packets sent by the first sending end from the first socket interface in the case of a non-distributed task, and the binding module 1407 is further used to bind the first socket interface and the first sending end.

[0181] In summary, the receiving device will continue to read the header field according to the received interval field; and continue to read the payload field according to the length of the payload field indicated in the header field, that is, the present application provides a method for reading a video data packet. The packet structure of the video data packet is the packet structure used by the peer device (sending device) when encapsulating the packet.

[0182] Moreover, in the present application, the receiving device sends a rebase message to the sending end only when the offset position does not match, instead of feeding back an ACK message for each video data packet received in the related art. The method of reading video data packets provided by the receiving device of the present application can effectively reduce the use of ACKs, thereby reducing transmission delays. The reliability of transmission is ensured by the method of reading video data packets provided by the present application, instead of using ACK messages to ensure reliability.

[0183] Fig.15 A schematic diagram of the structure of a communication device provided by an exemplary embodiment of the present application is shown. The communication device can be a transmitting device or a receiving device. The communication device includes: a processor 1501, a receiver 1502, a transmitter 1503, a memory 1504 and a bus 1505.

[0184] The processor 1501 includes one or more processing cores. The processor 1501 executes various functional applications and information processing by running software programs and modules.

[0185] The receiver 1502 and the transmitter 1503 may be implemented as a communication component, which may be a communication chip.

[0186] The memory 1504 is connected to the processor 1501 via a bus 1505 .

[0187] The memory 1504 may be used to store at least one instruction, and the processor 1501 may be used to execute the at least one instruction to implement each step in the above method embodiment.

[0188] In addition, the memory 1504 can be implemented by any type of volatile or non-volatile storage device or a combination thereof. The volatile or non-volatile storage device includes but is not limited to: a magnetic disk or an optical disk, an electrically erasable programmable read-only memory (EEPROM), an erasable programmable read-only memory (EPROM), a static random-access memory (SRAM), a read-only memory (ROM), a magnetic memory, a flash memory, and a programmable read-only memory (PROM).

[0189] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory including instructions, and the instructions can be executed by a processor of a communication device to complete the above-mentioned coverage enhancement level determination method. For example, the non-transitory computer-readable storage medium can be a ROM, a random access memory (RAM), a compact disc read-only memory (CD-ROM), a magnetic tape, a floppy disk, an optical data storage device, etc.

[0190] A non-transitory computer-readable storage medium, when instructions in the non-transitory computer storage medium are executed by a processor of a communication device, enables the communication device to execute the above-mentioned data processing method.

[0191] The present application also provides a communication device, which includes a processor and a memory, wherein a computer program is stored in the memory, and the computer program is loaded and executed by the processor to implement the data processing method provided in the above method embodiment.

[0192] The present application also provides a computer-readable storage medium, in which at least one instruction, at least one program, a code set or an instruction set is stored. The at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by a processor to implement the data processing method provided by the above method embodiment.

[0193] The present application provides a computer program product or a computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a communication device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the communication device executes the data processing method provided by the above method embodiment.

[0194] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0195] Those skilled in the art can understand that all or part of the steps of implementing the above embodiments can be completed by hardware, or by a program to instruct the relevant hardware to complete, and the program can be stored in a computer-readable storage medium, and the above-mentioned storage medium can be a read-only memory, a disk or an optical disk, etc. The above description is only an optional embodiment of the present application and is not intended to limit the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A data processing method, characterized in that: The method is performed by a sending end, and the method includes: Get the video data stream from the application layer; Taking the coded frames in the video data stream as the packetization granularity, performing a packetization operation on the video data stream to obtain a plurality of video data packets; Sending the multiple video data packets to a receiving end; Each of the plurality of video data packets contains at least one encoded frame.

2. The method according to claim 1, characterized in that The coded frame includes an I frame or a P frame; the packetization operation is performed on the video data stream using the coded frame in the video data stream as the packetization granularity to obtain multiple data packets, including: For each I frame in the video data stream, divide it into a first data packet separately; Each P frame in the video data stream is individually divided into a second data packet.

3. The method according to claim 1 or 2, characterized in that: The method further comprises: Acquire an audio data stream from the application layer; Taking the Advanced Audio Coding (AAC) structure in the audio data stream as the packetization granularity, performing a packetization operation on the audio data stream to obtain a plurality of audio data packets; Each of the multiple audio data packets contains at least one of the AAC structures.

4. The method according to claim 3, characterized in that The step of performing a packetization operation on the audio data stream using the AAC structure in the audio data stream as the packetization granularity to obtain a plurality of audio data packets includes: For the AAC structure in the audio data stream, multiple AAC structures whose overall size does not exceed a first threshold are divided into a third data packet, and an AAC structure whose single size is greater than the first threshold is individually divided into a fourth data packet.

5. The method according to claim 1 or 2, characterized in that: Any one of the plurality of video data packets comprises an interval field, a header field and a payload field; The interval field is used to separate the current video data packet from the multiple video data packets; The header field is used to describe the multimedia type corresponding to the payload data of the current video data packet, and to describe the length of the payload field, wherein the payload data is data obtained by performing the packetization operation on the video data stream; The payload field is used to carry the payload data and to describe the offset position of the current video data packet in the multiple video data packets.

6. The method according to claim 1 or 2, characterized in that: The method further comprises: Sending a query message to the receiving end; In a case where the query message carries a synchronization source SSRC parameter corresponding to the video data stream, obtaining a most recent offset position determined by the receiving end from the offset record, where the most recent offset position is an offset position corresponding to a video data packet most recently received by the receiving end; Based on the most recent offset position, a video data packet corresponding to a next offset position is sent to the receiving end.

7. The method according to claim 6, characterized in that The sending, based on the most recent offset position, a video data packet corresponding to a next offset position to the receiving end includes: In the case where the video data packet at the most recent offset position exists in the local cache of the sending end, sending the video data packet corresponding to the offset position next to the most recent offset position to the receiving end; When the video data packet at the most recent offset position does not exist in the local cache of the sending end, the service layer is prompted to supplement the video data packets between the most recent offset position and the current offset position of the local cache, and the video data packet corresponding to the next offset position of the most recent offset position is sent to the receiving end.

8. The method according to claim 1 or 2, characterized in that: The method further comprises: Sending a query message to the receiving end; In a case where the query message does not carry an SSRC parameter, an SSRC parameter corresponding to the video data stream generated by the receiving end is obtained.

9. The method according to claim 1 or 2, characterized in that: The method further comprises: receiving a message carrying a correct offset position, wherein the message of the correct offset position is sent by the receiving end when the receiving end cannot parse the correct offset position from an erroneous video data packet sent by the sending end; The video data packet corresponding to the correct offset position is sent to the receiving end.

10. A data processing method, characterized in that: The method is performed by a receiving end, and the method includes: Receive multiple video data packets corresponding to a video data stream, wherein the multiple video data packets are obtained by a transmitter performing a packetization operation on the video data stream with a coding frame as a packetization granularity, and each of the multiple video data packets contains at least one coding frame; The payload data in the plurality of video data packets is extracted.

11. The method according to claim 10, characterized in that Any one of the plurality of video data packets comprises an interval field, a header field and a payload field; the method further comprises: Continuously reading the multiple video data packets corresponding to the video data stream; When the interval field is read, continue reading the header field; Continue reading the payload field according to the length of the payload field recorded in the header field; The extracting the payload data from the plurality of video data packets comprises: When the offset position recorded in the payload field matches, the payload data carried in the payload field is extracted, the payload data is part of the data in the video data stream, and the offset position is used to describe the position of the current video data packet in the multiple video data packets.

12. The method according to claim 11, characterized in that The method further comprises: In the case where the offset position recorded in the payload field does not match, a rebasing message is sent to the sending end, wherein the rebasing message is used to instruct the sending end to continue transmitting the video data packet.

13. The method according to claim 11, characterized in that The method further comprises: When useless data and / or empty packet data exist in the read current video data packet, sending a PING message to the sending end; Upon receiving the PONG message sent by the transmitting end, confirming that the current video data packet is legal; If the PONG message sent by the sending end is not received, the connection is disconnected.

14. The method according to claim 11, characterized in that The receiving end supports multiple sending ends to jointly send the video data stream in a distributed manner, the multiple sending ends include a master sending end and at least one slave sending end, and the method further includes: In the process of receiving the multiple video data packets, if the received video data packets continue to have data scattering phenomenon, a filling data message is sent to the main sending end, and the filling data message is used by the main sending end to send or command the at least one slave sending end to send the video data packets missing in the data scattering phenomenon.

15. The method according to claim 11, characterized in that The continuously reading the plurality of video data packets comprises: In the case of a non-distributed task, the plurality of video data packets sent by the first sending end are continuously read from the first socket interface, and the first socket interface and the first sending end are bound.

16. A data processing device, characterized in that: The device is run at a transmitting end, and the device includes: An acquisition module, used for acquiring a video data stream from an application layer; A packetization module, configured to perform a packetization operation on the video data stream using the coded frames in the video data stream as the packetization granularity, to obtain a plurality of video data packets; A sending module, used for sending the plurality of video data packets to a receiving end; Each of the plurality of video data packets contains at least one encoded frame.

17. A data processing device, characterized in that: The device is run at a receiving end, and the device includes: A receiving module, configured to receive a plurality of video data packets corresponding to a video data stream, wherein the plurality of video data packets are obtained by a transmitting end performing a packetization operation on the video data stream with a coding frame as a packetization granularity, and each of the plurality of video data packets contains at least one coding frame; The extraction module is used to extract the payload data in the plurality of video data packets.

18. A communication device, characterized in that: The communication device comprises a processor and a memory, wherein the memory is used to store a computer program, and the processor is used to execute the computer program to implement the data processing method according to any one of claims 1 to 15.

19. A chip, characterized in that: The chip includes a processor for reading and executing a computer program from a memory to implement the data processing method as claimed in any one of claims 1 to 15.

20. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and the computer program is loaded and executed by a processor to implement the data processing method according to any one of claims 1 to 15.

Images