Video retransmission method and device based on active detection of sending end, equipment and medium

Abstract

The invention discloses a video retransmission method and device based on active detection of a sending end, equipment and a medium, and relates to the technical field of streaming media transmission, and the method comprises the steps: building a video transmission session between the sending end and a receiving end, receiving original video frame data at the sending end, carrying out the frame splitting and packaging processing, obtaining video data, and sending the video data to the receiving end, and receiving feedback data transmitted by the receiving end, determining a packet loss detection result based on the feedback data and a sending record of the sending end, and sending the packet loss detection result to the receiving end, the feedback data being a video packet sequence number received according to a period and a minimum video packet sequence number in an analog playing cache after the receiving end performs analog framing playing processing on the video data. According to the method and the device, the data of the original video frame is retransmitted on the basis of the playing time offset and the packet loss detection result, the retransmission is concentrated to the video transmitting end, the convenience of video retransmission protocol development, test and deployment is effectively improved, and the pertinence and the real-time performance of retransmission data are ensured.

Description

Technical Field

[0001] This application relates to the technical field of streaming media transmission, and particularly to a video retransmission method, apparatus, device and medium based on active detection at the sending end. Background Art

[0002] With the development of wireless networks, video transmission, as the main service carried by wireless networks, has become increasingly widespread. Wireless IP networks generally only guarantee best-effort delivery. Since wireless links are vulnerable to interference and unstable, packet loss is very common in this transmission scenario. After video data experiences packet loss, it can cause video stuttering, latency, and even black screens in severe cases. Therefore, how to ensure the complete end-to-end delivery of video streams over an unstable transmission link is an important issue in video transmission. End-to-end retransmission methods are common ways to address link packet loss problems.

[0003] In related technologies, most video retransmission protocols encapsulate and transmit data based on existing transmission protocols such as UDP, TCP, and RTP. The packet loss detection logic is often deployed at the video receiving end and achieved by sending retransmission requests. Therefore, how to ensure the pertinence and timeliness of retransmitted data for the characteristics of video service data needs to be solved. Summary of the Invention

[0004] In view of the above problems, this application provides a video retransmission method, apparatus, device and medium based on active detection at the sending end to at least solve the problems existing in related technologies.

[0005] In a first aspect, an embodiment of this application provides a video retransmission method based on active detection at the sending end. The video retransmission method based on active detection at the sending end includes:

[0006] Establish a video transmission session between the sending end and the receiving end;

[0007] After receiving the original video frame data at the sending end, perform frame splitting and encapsulation processing to obtain video data, and send it to the receiving end;

[0008] Receive the feedback data transmitted by the receiving end, where the feedback data is the video packet sequence number received periodically and the smallest video packet sequence number in the simulated playback cache after the receiving end performs simulated frame assembly and playback processing on the video data;

[0009] Determine the packet loss detection result based on the feedback data and the sending record of the sending end, and calculate the link RTT to obtain the playback time offset;

[0010] Perform retransmission of the original video frame data based on the playback time offset and the packet loss detection result.

[0011] In some embodiments, performing retransmission of the original video frame data based on the playback time offset and the packet loss detection result includes:

[0012] Query the sequence numbers of the packets sent and received in the packet loss detection result, filter out the sequence numbers that are sent but not returned, and mark them as suspected packet losses;

[0013] Record the data packets corresponding to the marked suspected packet losses into the retransmission queue, and periodically detect the retransmission queue;

[0014] When the time stamp of the video data packet sent in the playback time offset exceeds the delay of one RTT from the current time stamp and it is a valid playable packet, retransmit the data packet corresponding to this sequence number.

[0015] In some embodiments, during the process of establishing a video transmission session between the sender and the receiver, it includes:

[0016] The sender simultaneously sends down the transmission parameters and configurations to the receiver to configure the feedback time period and the video frame buffer of the receiver;

[0017] The sender initializes the video including the starting sequence number and the starting sequence number of the video frame.

[0018] In some embodiments, the feedback data includes: a delay time stamp, where the delay time stamp is the time difference between the time when the data packet of the sender arrives at the receiver and the periodically feedback ACK in the feedback data of the receiver.

[0019] In some embodiments, after receiving the original video frame data, the sender performs frame splitting processing, including:

[0020] Perform segmentation processing on the original video frame data to obtain data packets;

[0021] Define the frame sequence number, packet sequence number, sampling time, and sending time in the packet header of the data packet.

[0022] In some embodiments, the video retransmission method based on active detection by the sender further includes:

[0023] Receive the target frame sequence number and target data packet sequence number of the current video playback recorded by the receiver in the feedback data;

[0024] Retransmit the data after the target frame sequence number and target data packet sequence number.

[0025] In some embodiments, the video caching function time of the receiver is based on the sampling time in the data packet of the sender, and plays the video frame data cached by the receiver at the interval time of the sender's video frames.

[0026] In a second aspect, an embodiment of the present application provides a video retransmission device based on active detection at the sending end, including:

[0027] A session establishment module, configured to establish a video transmission session between the sending end and the receiving end;

[0028] A processing module, configured to perform frame splitting and encapsulation processing on the original video frame data after the sending end receives it to obtain video data, and send the video data to the receiving end;

[0029] A receiving module, configured to receive feedback data transmitted by the receiving end, where the feedback data is the video packet sequence number received periodically and the smallest video packet sequence number in the simulated playback cache after the receiving end performs simulated frame assembly and playback processing on the video data;

[0030] A calculation module, configured to determine a packet loss detection result based on the feedback data and the sending record of the sending end, and perform link RTT calculation to obtain a playback time offset;

[0031] A retransmission module, configured to retransmit the original video frame data based on the playback time offset and the packet loss detection result.

[0032] In a third aspect, an embodiment of the present application provides an electronic device, including a memory and a processor. A program code that can run on the processor is stored on the memory. When the program code is executed by the processor, the video retransmission method based on active detection at the sending end introduced in any implementation manner of the first aspect is implemented.

[0033] In a fourth aspect, an embodiment of the present application provides a computer storage medium. One or more programs are stored in the computer storage medium. The one or more programs can be executed by the electronic device introduced in the third aspect to implement the video retransmission method based on active detection at the sending end introduced in any implementation manner of the first aspect.

[0034] A video retransmission method, apparatus, device, and medium based on active detection at the sender provided by an embodiment of the present application establish a video transmission session between the sender and the receiver. After the sender receives the original video frame data, it performs frame splitting and encapsulation processing to obtain video data and sends it to the receiver. The sender receives the feedback data transmitted by the receiver, where the feedback data is the video packet sequence number received periodically and the smallest video packet sequence number in the simulated playback cache after the receiver performs simulated frame grouping and playback processing on the video data. Based on the feedback data and the sender's sending record, the packet loss detection result is determined, and the link RTT is calculated to obtain the playback time offset. Based on the playback time offset and the packet loss detection result, the original video frame data is retransmitted. By concentrating the retransmission at the video sender, the algorithm cohesion at the sender side is improved, the convenience of video retransmission protocol development, testing, and deployment is effectively enhanced, and the pertinence and real-time nature of the retransmitted data are ensured.

[0035] It should be understood that the content described in this part is not intended to identify the key or important features of the embodiments of the present application, nor is it used to limit the scope of the present application. Other features of the present application will become easily understandable through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Hereinafter, the present application will be described in more detail based on embodiments with reference to the drawings.

[0037] Figure 1 FIG. shows a schematic flowchart of a video retransmission method based on active detection at the sender proposed in an embodiment of the present application;

[0038] Figure 2 FIG. shows a schematic flowchart of an exemplary video retransmission process completed by interaction between the sender and the receiver proposed in an embodiment of the present application;

[0039] Figure 3 FIG. shows a schematic flowchart of algorithm interaction in an exemplary video retransmission method proposed in an embodiment of the present application;

[0040] Figure 4 FIG. shows a schematic block diagram of an exemplary video retransmission system based on active detection at the sender proposed in an embodiment of the present application;

[0041] Figure 5 FIG. shows a schematic diagram of data encapsulation of the sender's data during an exemplary connection establishment process;

[0042] Figure 6 FIG. shows a schematic block diagram of an exemplary video retransmission apparatus based on active detection at the sender proposed in an embodiment of the present application;

[0043] Figure 7The block diagram of an electronic device for executing a video retransmission method based on active detection at the sending end according to an embodiment of the present application is shown;

[0044] Figure 8 A computer-readable storage medium for storing or carrying an implementation of a video retransmission method based on active detection at the sending end according to an embodiment of the present application is shown. Detailed implementation manners

[0045] In order to make the objectives, technical solutions and advantages of the present invention more clear and understandable, the present invention will be further described in detail below with reference to the embodiments and the accompanying drawings. The illustrative embodiments of the present invention and the descriptions thereof are only used to explain the present invention and shall not be construed as limiting the present invention.

[0046] In the related art, the existing solutions completely rely on the best-effort manner of the underlying link, which cannot guarantee the integrity of data transmission. When the link is unstable, data loss is likely to occur, resulting in the receiving party not receiving complete data packets. Although retransmission based on the link layer can ensure data integrity to a certain extent, the link lacks an understanding of the video data content layer, and the retransmission design often lacks targeted consideration of data service characteristics, resulting in low retransmission efficiency, and the real-time delivery may also be affected. Further retransmission designs often use the method of active request at the receiving end. During project implementation, protocol development, testing, and deployment need to be operated on different video sending and receiving devices simultaneously, which increases the development, testing, and deployment costs.

[0047] Analyzing the existing problems, the applicant has found that most of the existing video retransmission protocols currently encapsulate and transmit data based on existing transport protocols such as UDP, TCP, and RTP. The packet loss detection logic is often deployed at the video receiving end and implemented by sending a retransmission request.

[0048] Therefore, the inventors have proposed a video retransmission method, apparatus, device, and medium based on active detection at the sending end. By establishing a video transmission session between the sending end and the receiving end, after receiving the original video frame data at the sending end, the video data is obtained through frame splitting and encapsulation processing and sent to the receiving end. The feedback data transmitted by the receiving end is received. The feedback data is the video packet sequence number received periodically and the smallest video packet sequence number in the simulated playback cache after the receiving end performs simulated frame grouping and playback processing on the video data. Based on the feedback data and the sending record of the sending end, the packet loss detection result is determined, and the link RTT is calculated to obtain the playback time offset. Based on the playback time offset and the packet loss detection result, the original video frame data is retransmitted, comprehensively considering the characteristics of video service data. Finally, the pertinence and real-time nature of the retransmitted data are ensured, and the development, testing, and deployment efficiency are taken into account. Among them, the video retransmission method based on active detection at the sending end will be described in detail in the subsequent embodiments.

[0049] The application scenarios of the video retransmission method based on active detection at the sending end provided in the embodiments of the present application are introduced as follows:

[0050] Please refer to Figure 1 , Figure 1 which is a schematic flowchart of a video retransmission method based on active detection at the sending end provided in the embodiments of the present application. In this embodiment, the video retransmission method based on active detection at the sending end can be applied to a video retransmission device 300 based on active detection at the sending end as shown in Figure 6 , an exemplary structural block diagram of a video retransmission system based on active detection at the sending end as shown in Figure 4 and an electronic device 200 as shown in Figure 7 . Among them, the electronic device may include one or more. Information can be transmitted between multiple electronic devices in a wireless and / or wired manner, and multiple electronic devices can cooperate to complete the video retransmission method based on active detection at the sending end. Exemplarily, the electronic device may include a computer, a mobile terminal, a communication device, etc., and the present application does not limit it. The following elaborates in detail on the process shown in Figure 1 . The video retransmission method based on active detection at the sending end may include S110 to S150.

[0051] S110: Establish a video transmission session between the sending end and the receiving end.

[0052] In the embodiments of the present application, in the video transmission session connection establishment step, the video sending end listens for video transmission instructions at the service layer. After receiving the instructions, it sends a video session establishment request signaling to the video receiving end. After the video receiving end listens for the port and receives the session establishment request, it initializes the video receiving module. After the initialization is completed, it sends a request confirmation signaling back to the video sending end. After the video sending end listens for the port and receives the request confirmation signaling, it initializes the video sending end module. Through the connection signaling interaction, both the sending and receiving ends complete the video transmission preparation work.

[0053] S120: After receiving the original video frame data at the sending end, perform frame splitting and encapsulation processing to obtain video data, and send it to the receiving end.

[0054] In the embodiments of the present application, in the frame splitting and encapsulation step of the video sending end, the video sending end obtains the complete video frame data collected and encoded in real time, splits the video frame data, and encapsulates it. The video frame information and the continuous data packet number information are encapsulated into the data packet header and placed in the queue to be sent for sending.

[0055] S130: Receive the feedback data transmitted by the receiving end, where the feedback data is the video packet sequence number and the smallest video packet sequence number in the analog play cache received at regular intervals after the receiving end performs analog frame assembly and playback processing on the video data.

[0056] In an embodiment of the present application, packets are received and framed at a video receiving end, wherein after receiving a video data packet, the video receiving end inserts the data into a corresponding position in a video simulation playback buffer area at the receiving end according to the video packet sequence and video frame sequence number information encapsulated in a packet header protocol, and then performs simulation playback based on the time offset information of the video frame. The video receiving end provides periodic feedback, and the video receiving end receives video data packets through a link, and records the number of each video data packet. When a periodic timer is triggered, all video data packet sequence numbers recorded in the period and the minimum video packet sequence number in the video simulation playback buffer area at the receiving end are sent to the video sending end through a feedback protocol.

[0057] S140: Determine the packet loss detection result based on the feedback data and the sending record of the sending end, and calculate the link RTT to obtain the playback time offset.

[0058] In an embodiment of the present application, a video transmitter performs retransmission detection, wherein the video transmitter receives information periodically fed back by the video receiver, and first calculates the end-to-end RTT of the link. The transmitter calculates the end-to-end RTT of the link through the sending and receiving timestamps of data packets with the same sequence number, and further notifies the RTT to calibrate the minimum video packet sequence number, effectively retransmits the delay offset, and obtains a retransmittable data packet that can be effectively played and meets the real-time requirements.

[0059] S150: Retransmitting the original video frame data based on the playback time offset and the packet loss detection result.

[0060] In an embodiment of the present application, by querying the sequence numbers of the sent packets and the received packets, the sequence numbers that were sent but not returned are screened out, marked as suspected packet loss, and recorded in the queue to be retransmitted. The queue to be retransmitted is checked periodically. If the packet sending timestamp of the video data packet is delayed by more than one RTT from the current timestamp and is a valid packet that can be played, a retransmission is performed on the data packet corresponding to the sequence number.

[0061] In this embodiment, the algorithm's sending end cohesion is improved through the design of retransmission check at the sending end and simple passive feedback at the receiving end, and the development and deployment efficiency of the method are improved. By framing the video packets at the receiving end and simulating the playback function, the integrity statistics and real-time guarantee of the video frame boundary are obtained, and the data packets that have lost their retransmission value are screened out, which improves the efficiency and pertinence of retransmission, solves the problem of data packet loss during video transmission, ensures end-to-end real-time and smooth video playback, and improves the development, operation and deployment efficiency of the protocol at the engineering level.

[0062] In some embodiments, S150 includes S151 to S153.

[0063] S151: Query the sequence numbers of the sent and received packets in the packet loss detection result, filter out the sequence numbers that are sent but not returned, and mark them as suspected packet loss.

[0064] Among them, the sequence number sent and returned at the same time indicates that the data transmission is correctly completed and is not marked as a suspected packet loss.

[0065] S152: Record the data packets corresponding to the marked suspected packet losses into the retransmission queue, and periodically detect the retransmission queue to be processed.

[0066] S153: When it is obtained that the packet sending timestamp in the video packet to the current timestamp exceeds the delay of one RTT in the playback time offset and it is a valid playable packet, retransmit the data packet corresponding to the sequence number.

[0067] In this embodiment, through the video packet sequence numbering and the receiver periodic confirmation mechanism, and through the mechanism based on RTT check, the lost data packets in the end-to-end link transmission process are obtained, and retransmission is performed to improve the data integrity of the end-to-end transmission.

[0068] In some embodiments, during the process of S110 establishing a video transmission session between the sender and the receiver, it includes S111 to S112.

[0069] S111: The sender simultaneously sends the transmission parameters and configurations to the receiver to configure the feedback time period and the video frame buffer of the receiver;

[0070] S112: The sender initializes the video including the starting sequence number and the starting sequence number of the video frame.

[0071] In some embodiments, the feedback data includes: a delay timestamp, where the delay timestamp is the time difference between the time when the data packet of the sender arrives at the receiver and the periodically feedback ACK in the feedback data of the receiver.

[0072] In this embodiment, the end-to-end link RTT value obtained based on the delay timestamp is more accurate.

[0073] In some embodiments, after the sender receives the original video frame data in S120, frame splitting processing is performed, including S121 to S123.

[0074] S121: Perform segmentation processing on the original video frame data to obtain data packets.

[0075] S122: Define the frame sequence number, packet sequence number, sampling time, and sending time in the packet header of the data packet.

[0076] In some embodiments, the video retransmission method based on the sender's active detection further includes:

[0077] Receiving the target frame sequence number and the target data packet sequence number of the current video playback recorded by the receiver in the feedback data;

[0078] Retransmission is performed on the data after the target frame sequence number and the target data packet sequence number.

[0079] In this embodiment, the data before this frame sequence number and the data packet sequence number do not need to be retransmitted. Only the data after this frame sequence number and the data sequence number are considered for retransmission, which improves the video retransmission efficiency of the system and the link load.

[0080] In some embodiments, the video caching function time at the receiving end is based on the sampling time in the data packet of the sending end, and the video frame data cached at the receiving end is played at the interval time of the video frames of the sending end.

[0081] In this embodiment, the above technical features are adopted to achieve the effects of smooth video and reduced packet loss rate.

[0082] In some embodiments, the video retransmission method based on active detection at the sending end can be applied to Figure 4 the exemplary structural block diagram of the video retransmission system with active detection at the sending end as shown. Among them, the video retransmission system with active detection at the sending end includes: a sending end and a receiving end. Among them, the sending end includes: a video frame encapsulation module and a retransmission detection module. The retransmission detection module includes an RTT sampling sub-module, a hole screening sub-module, a playback offset calibration sub-module, and a retransmission polling sub-module.

[0083] The receiving end includes an analog playback module and a feedback module.

[0084] The video frame encapsulation module sends the acquired video frame data to the analog playback module and then transmits it to the feedback module for processing. The feedback module transmits it to the RTT sampling sub-module, the hole screening sub-module, and the playback offset calibration sub-module.

[0085] Among them, the video retransmission system with active detection at the sending end also includes other modules, which are not limited in this application.

[0086] In the video retransmission system with active detection at the sending end, the following are completed:

[0087] Perform the establishment and initialization of the session connection between the video transceiver ends, and align the transmission parameter configuration including the initial video packet sequence number, the initial video frame sequence number, the feedback period of the video receiving end, the pre-caching time of the video frames at the receiving end, and preliminarily complete the preparation work at the connection level before video transmission.

[0088] Receive complete video frame data, split the video frames into data packets suitable for transmission over the link, and number the split video data packets in an orderly manner. At the same time, for the video packet data split from the same video frame, perform in-frame numbering, and record the video frame type, the total number of packets after video frame splitting, and the time offset relative to the initial frame. These information are encapsulated in the header of each video packet, and through these information, the video frame number acquisition status can be reconstructed orderly at the video receiving end.

[0089] Execute the sequence numbers of the received video packets within the statistical period, calculate the delay processed by the receiving end for each video packet, and obtain the status information of the video frame buffer at the receiving end. Then, package and encapsulate this information in a feedback protocol packet and send it back to the video sender.

[0090] Execute the reception of video data packets, complete the sorting of video packets and the framing of video packet groups according to the packet information and frame information encapsulated in the video data packet header, and the sorting of the frames after framing. Through the time offset timestamp of the video frame acquisition time, complete the simulation playback function, use the frame interval as the timing period, and submit the combined video frames to the real video frame player to complete decoding and playback.

[0091] The RTT sampling sub-module, the hole check sub-module, the playback offset correction sub-module, and the retransmission polling sub-module are driven by the information of the timer and the received-end video feedback module.

[0092] Among them, after obtaining the feedback information, the RTT sampling sub-module records the reception timestamp of the feedback packet, and calculates the actual end-to-end time consumption of each feedback data packet based on the video data packet processing delay and the sending timestamp in the feedback information, and performs RTT sampling evaluation based on this calculated data.

[0093] After obtaining the feedback information, the hole check sub-module obtains the sequence numbers of the video data packets that have been sent but not successfully received based on the record of the sent video data packet sequence numbers and the feedback video data packet sequence numbers, and marks them as the sequence numbers to be retransmitted.

[0094] After obtaining the feedback information, the playback offset correction sub-module obtains the time offset value of the frame where the video packet is located relative to the initial frame after obtaining the smallest video packet sequence number in the receiving-end video frame buffer feedback from the receiving end. This time offset value continues to accumulate the reference RTT value calculated in the above RTT sampling sub-module to obtain the effective playback offset value. Then, continue to check the sequence numbers of the video packets to be retransmitted marked by the hole check sub-module. If the offset value of the frame where the packet is located is less than the effective playback offset value, these video packets will be excluded from the record table of the sequence numbers of the video packets to be retransmitted.

[0095] The retransmission polling sub-module periodically checks the record table of the retransmission video packet sequence numbers, checks the sending timestamp of these video packet sequence numbers and the time offset value at the current moment. If the offset value is greater than the reference RTT obtained in the above steps, then add this video packet to the queue to be sent and wait for retransmission, and execute the actual retransmission of the video data packet.

[0096] The specific work implementation process is as follows:

[0097] Refer to Figure 2 , Figure 2 , which is a schematic diagram of an exemplary video retransmission process for the interaction between the sender and the receiver of this application, including the following steps:

[0098] S101: Establish a video transmission session: The video sender sends a session establishment request to the video receiver. After the video receiver successfully obtains the session establishment request from the sender, it initializes the video receiving module. After success, it sends a request confirmation instruction to the corresponding sender. After receiving the request confirmation instruction, the video sender completes the initialization of the sender module and completes the video transmission session.

[0099] During the connection establishment process in this step, the sender will simultaneously send transmission parameters and configurations to the receiver, configure the feedback time period, and configure the video frame buffer area of the receiver. The sender will initialize the starting sequence number of the video packet and the starting sequence number of the video frame.

[0100] In some embodiments, since there are no special requirements for the wireless channel transmission protocol and the spatial environment between the sender and the receiver by parsing the request signaling and the parsed reply request confirmation instruction, therefore, a channel protocol with the ability to transmit both on the ground and in the air, such as shortwave and ultra-shortwave frequencies, can be selected to establish a connection.

[0101] S102: Sender: Split frames, encapsulate, and send. The video sender receives the original video frame data, unpacks the video frames, sorts the data packets using consecutive sequence numbers, and finally combines the video frame information and encapsulates it into the data packet header; finally, it is sent to the network link.

[0102] In this step, the sender splits the video frame data according to the link MTU, and numbers the split video packets sequentially according to the starting frame required after the initialization in step S101 and the starting packet sequence number. Each packet is numbered sequentially within the current frame, and the total number of packets after the video frame is split is recorded, and the time offset value of the current frame relative to the initial frame is recorded. When the current frame is the first frame received by the sender, the offset value is recorded as 0.

[0103] The above packet sequence and frame information are encapsulated into the video packet header. The encapsulated video packets are placed in the queue to be sent and sent later. At the moment of sending the video packet, the sender records the timestamp of the current sending moment of the packet and the video packet sequence number for the subsequent end-to-end RTT calculation and retransmission detection in the subsequent steps.

[0104] S103: Receiver: Receive packets and assemble frames. The receiver continuously receives video data packets, completes the sorting of video packets, frame assembly, and video frame sorting according to the description in the packet header. And according to the frame offset timestamp in the packet header, the assembled frames are sent to the video player in an orderly manner.

[0105] In this step, each time a video data packet is received, the receiver records the current video packet sequence number, and at the same time records the current time as the packet reception timestamp. According to the video frame coordinate information recorded in the packet header, the video packets are inserted into the video frame buffer area of the receiver in an orderly manner to complete the assembly of the video packets.

[0106] At the receiving end, the video frames are played back analogously in the video frame buffer according to the video frame offset time recorded in the packet header. The video frame time offset can be understood as the acquisition time offset. The video frames use the acquisition time offset as the fixed time interval, and the collected video frame data is delivered to the player for playback.

[0107] Perform periodic data feedback for received packets. Record the current feedback time as the feedback timestamp, record the packet numbers within the period, obtain the receive timestamp corresponding to each numbered packet, calculate the difference from the feedback timestamp to obtain the receiving-end processing delay for each packet within this period. At the same time, obtain the smallest video packet sequence number in the smallest video frame among the current buffered data in the receiving-end video frame buffer. Finally, encapsulate the video packet sequence numbers received within the period, the receiving-end processing delay corresponding to each sequence number, and the smallest video packet sequence number obtained from the video frame into a periodic feedback packet and send it to the sending end to complete the periodic receive status feedback.

[0108] S104: Receiving end: Periodic feedback. The receiving end records the video packet sequence numbers received within the period, calculates the receiving-end processing delay for each packet, and at the same time obtains the lowest-order video packet sequence number from the analog playback module as feedback data to be sent to the video sending end.

[0109] In this step, after the sending end receives the periodic packet reception feedback from the receiving end, it records the receive timestamp, and successively obtains the video packet sequence number and the receiving-end processing delay of the current feedback. Combining with the send timestamp recorded at the video packet sending time, calculate the total time it takes for each data packet to be successfully confirmed from being sent, and then subtract the receiving-end processing delay for each packet to obtain the per-packet RTT sample value. And filter the RTT sample value through the EWMA algorithm to obtain the reference RTT.

[0110] Step 105: Sending end: Packet loss detection and retransmission execution. The video sending end, according to the received periodic feedback data and combined with the sending record, performs real-time calculation of the link RTT, calculation of the playback offset, and packet loss check. Screen out the data packets with retransmission value and execute packet retransmission.

[0111] In this step, after the sending end receives the video packet sequence numbers for which the periodic receiving-end feedback confirmation is obtained, it checks the retransmission queue and removes the successfully received video packets from the retransmission queue. At the same time, according to the sending packet sequence number record table at the sending end, continue to check the video data packet sequence numbers that have been sent but not confirmed, and record them as the retransmission video packet sequence number table.

[0112] After obtaining the smallest video packet sequence number in the receiver video frame buffer area fed back by the periodic receiver, obtain the time offset value of the frame where the video packet is located relative to the initial frame. This time offset value continues to accumulate the reference RTT value calculated in the above steps to obtain the effective playback offset value. Continue to check the above-mentioned list of video packet sequence numbers to be retransmitted. If the offset value of the frame where the packet is located is less than the effective playback offset value, these video packets are removed from the record table of video packet sequence numbers to be retransmitted.

[0113] In this step, after the above retransmission check, the video packets corresponding to the finally confirmed video packet sequence numbers to be retransmitted are added back to the transmission queue to be sent and retransmitted at the sender.

[0114] Triggered by the periodic retransmission timer, periodically check the retransmission queue, check the transmission timestamps of these video packet sequence numbers, and the time offset value from the current moment. If the offset value is greater than the reference RTT obtained in the above steps, then add the video packet to the transmission queue to wait for retransmission, and perform the actual retransmission of the video data packet.

[0115] The video retransmission method based on active detection at the sender provided in this embodiment can shield the differences in the link layer and physical layer, ensure the integrity of the data transmitted end-to-end, and improve the integrity statistics and real-time guarantee at the video frame level through the receiver video packet framing and simulated playback functions.

[0116] Refer to Figure 3 In the schematic diagram of the algorithm interaction process of an exemplary video retransmission method shown, it includes the steps:

[0117] S201: The sender collects video frame information from the video data, sends a connection establishment request, and carries configuration data and transmission data.

[0118] S202: After receiving the connection establishment request, the receiver initializes the receiver and replies to confirm the connection establishment request.

[0119] S203: After receiving the confirmation request, the sender initializes the sender and starts transmitting video data packets.

[0120] S204: After receiving the video data packets, the receiver periodically feeds back video packet data, including: delay timestamp, received packet frame sequence number, received data packet sequence number, etc.

[0121] In some embodiments, the specific periodic time for the receiver to periodically feed back video packet data is determined according to the video sampling time, frame rate, etc. In this application, the periodic feedback time is 20 milliseconds.

[0122] Refer to Figure 4 In the block diagram of an exemplary video retransmission system with active detection at the sender shown, in the figure, the execution steps of S301 to S304 in each module include:

[0123] S301. A connection is established between the sender and the receiver. The sender obtains a video transmission instruction and includes transmission parameters, etc. After receiving the transmission instruction, the receiver completes initialization and replies with a transmission request confirmation instruction.

[0124] S302. The video frame encapsulation module receives video frame data, splits the video frame data into data packets, and writes information such as frame sequence number, packet sequence number, and sampling time in the encapsulated frame header, and then sends it to the wireless channel.

[0125] In some embodiments, the splitting of video frame data is used to cut video frames with a large amount of data, which is more conducive to channel transmission and preventing channel congestion. For example, in the UDP transmission protocol, there is a limit on the size of data packets. Different MTU values are allocated according to the channel environment. The sequence numbers and index numbers for packet splitting are to ensure the integrity of UDP transmission.

[0126] S303. The analog playback module at the receiver receives the encapsulated data from the sender via the wireless channel and makes periodic feedback on the data from the sender, and writes the feedback data into the feedback module.

[0127] S304. The feedback module receives the packet information (arrival time, delay timestamp, etc.) fed back by the analog playback module, processes the data, and then feeds it back to the retransmission detection module.

[0128] S305. The retransmission detection module calculates the RTT of the end-to-end link based on the data from the feedback module, and queries the sequence numbers of the sent packets and the received packets. The sequence numbers of the sent packets that have not been returned are marked as suspected lost packets. The sequence numbers of the sent packets that have been returned at the same time are considered to have completed data transmission correctly and are not marked as suspected lost packets.

[0129] If the time difference between the video packet sending timestamp and the current timestamp exceeds one RTT, then the packet is retransmitted once; otherwise, the data of the packet is not retransmitted.

[0130] In some embodiments, for the packets pointed to by the frame sequence number and packet sequence number in the data of the feedback module, they are first marked as suspected lost packets. If the time difference between the video packet sending timestamp and the current timestamp is less than one RTT, then this data packet is not retransmitted. Only when the current timestamp is greater than one RTT, this data packet is retransmitted.

[0131] Refer to Figure 5 The schematic diagram of the encapsulation of the sender's data in an exemplary connection establishment process shown in

[0132] S401: First, perform video frame processing on the data video.

[0133] S402: Segment each video frame into data packets arranged in sequence.

[0134] S403: Before sending the data of each data packet, perform data encapsulation in the order of the frame header and the frame body. The encapsulation of the data packet is divided into the frame header and the frame body. The frame header stores parameter information such as the frame sequence number, packet sequence number, adoption time, sending time, data length, etc., and the frame body stores the video frame image pixel information.

[0135] In some embodiments, before the data packet encapsulation process, the following processing is also required:

[0136] Sort the video data in the order of the sampling time of the frames, denoted as the frame sequence number. In each frame of data, the image is segmented into different data packets, and the data packet sequence numbers are marked in the order of segmentation.

[0137] Please refer to Figure 6 , Figure 6 FIG.

[0138] FIG. 300 is a structural block diagram of a video retransmission device based on active detection at the sending end provided by the present application. The video retransmission device 300 based on active detection at the sending end includes: a session establishment module 310, a processing module 320, a receiving module 330, a calculation module 340, and a retransmission module 350, where:

[0139] The session establishment module 310 is used to establish a video transmission session between the sending end and the receiving end.

[0140] The processing module 320 is used to perform frame splitting and encapsulation processing on the original video frame data after receiving it at the sending end to obtain video data, and send it to the receiving end.

[0141] The receiving module 330 is used to receive the feedback data transmitted by the receiving end. The feedback data is the video packet sequence number and the smallest video packet sequence number in the analog play cache received at regular intervals after the receiving end performs analog frame assembly and playback processing on the video data.

[0142] The calculation module 340 is used to determine the packet loss detection result based on the feedback data and the sending record of the sending end, and perform link RTT calculation to obtain the playback time offset.

[0143] The device embodiment in the present application may further include other modules, which specifically correspond to the content in the above method part.

[0144] It should be noted that the device embodiment in the present application corresponds to the foregoing method embodiment. The specific principle in the device embodiment can be referred to the content in the foregoing method embodiment, and will not be elaborated here.

[0145] In several embodiments provided in this embodiment, the coupling between modules can be electrical, mechanical, or other forms of coupling.

[0146] In addition, in each embodiment of the present invention, each functional module can be integrated in a processing module, or each module can exist physically alone, or two or more modules can be integrated in one module. The above integrated modules can be implemented in the form of hardware or in the form of software functional modules.

[0147] Please refer to Figure 7 , Figure 7 FIG. 200 is a structural block diagram of an electronic device 200 that can execute the above-mentioned video retransmission method based on active detection at the sending end provided by an embodiment of the present application. The electronic device 200 can be a communication device, a mobile phone, a computer, or a portable computer, etc.

[0148] The electronic device 200 further includes a processor 202 and a memory 204. Among them, the memory 204 stores a program that can execute the content in the foregoing embodiments, and the processor 202 can execute the program stored in the memory 204.

[0149] Please refer to Figure 8 , Figure 8 FIG. 400 shows a structural block diagram of a computer-readable storage medium provided by an embodiment of the present application. The computer-readable storage medium 400 stores program code 410, and the program code 410 can be called by a processor to execute the method described in the above method embodiments.

[0150] The embodiment of the present application further provides a computer program product or a computer program. The computer program product or the computer program includes computer instructions, and the computer instructions are stored in a computer-readable storage medium. The processor of the computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the video retransmission method based on active detection at the sending end described in the above various optional implementation manners.

[0151] The above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions described in the foregoing embodiments, or perform equivalent replacements for some of the technical features; and these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A video retransmission method based on active detection at the sender, characterized in that The method includes: Establishing a video transmission session between a sender and a receiver; After receiving the original video frame data at the sender, performing frame splitting and encapsulation processing to obtain video data, and sending it to the receiver; Receiving the feedback data transmitted by the receiver, where the feedback data is the video packet sequence number received periodically and the smallest video packet sequence number in the simulated playback cache after the receiver performs simulated frame grouping and playback processing on the video data; Determining the packet loss detection result based on the feedback data and the sender's sending record, and calculating the link RTT to obtain the playback time offset; Performing retransmission of the original video frame data based on the playback time offset and the packet loss detection result.

2. The video retransmission method based on active detection at the sending end according to claim 1, wherein The performing retransmission of the original video frame data based on the playback time offset and the packet loss detection result includes: Querying the sequence numbers of the sent packets and the received packets in the packet loss detection result, screening out the sequence numbers that are sent but not returned, and marking them as suspected packet losses; Recording the data packets corresponding to the marked suspected packet losses into the retransmission queue, and periodically detecting the retransmission queue; When the delay from the packet sending timestamp of the video data packet in the playback time offset to the current timestamp exceeds one RTT and it is a valid playable packet, retransmitting the data packet corresponding to this sequence number.

3. The video retransmission method based on active detection at the sending end according to claim 1, characterized in that During the process of establishing the video transmission session between the sender and the receiver, it includes: The sender simultaneously sends down transmission parameters and configurations to the receiver to configure the feedback time period and the video frame buffer of the receiver; The sender initializes the video including the starting sequence number and the starting sequence number of the video frame.

4. The video retransmission method based on active detection at the sending end according to claim 1, wherein The feedback data includes: a delay timestamp, where the delay timestamp is the time difference between the time when the sender's data packet arrives at the receiver and the periodic feedback ACK in the receiver's feedback data.

5. The video retransmission method based on active detection at the sending end according to claim 1, wherein After the sender receives the original video frame data, the frame splitting process includes: Performing segmentation processing on the original video frame data to obtain data packets; Defining the frame sequence number, packet sequence number, sampling time, and sending time in the packet header of the data packet.

6. The video retransmission method based on active detection at the sending end according to claim 1, wherein The method further includes: Receiving the target frame sequence number and target data packet sequence number of the current video playback recorded by the receiver in the feedback data; Performing retransmission on the data after the target frame sequence number and target data packet sequence number.

7. The video retransmission method based on active detection at the sending end according to claim 6, characterized in that, The video caching function time of the receiver is based on the sampling time in the sender's data packet, and plays the video frame data cached by the receiver at the interval time of the sender's video frames.

8. A video retransmission device based on active detection at the sending end, characterized in that The device includes: A session establishment module for establishing a video transmission session between a sender and a receiver; A processing module for performing frame splitting and encapsulation processing on the original video frame data after receiving it at the sender to obtain video data, and sending it to the receiver; A receiving module for receiving the feedback data transmitted by the receiver, where the feedback data is the video packet sequence number received periodically and the smallest video packet sequence number in the simulated playback cache after the receiver performs simulated frame grouping and playback processing on the video data; A calculation module, configured to determine a packet loss detection result based on the feedback data and the transmission record of the sending end, and perform link RTT calculation to obtain a playback time offset; A retransmission module, configured to retransmit the original video frame data based on the playback time offset and the packet loss detection result.

9. An electronic device, characterized in that, The electronic device includes a memory and a processor. A program code that can run on the processor is stored on the memory. When the program code is executed by the processor, the video retransmission method based on active detection at the sending end according to any one of claims 1-7 is implemented.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a program code, and the program code can be called and executed by one or more processors to perform the video retransmission method based on active detection at the sending end according to any one of claims 1-7.

Images