Data processing method and device

Abstract

The embodiment of the invention discloses a data processing method and device. The method comprises the following steps: the server forwards a data packet sent by a sending end to a plurality of receiving ends; the method further comprises the step that the server receives first feedback information sent by the multiple receiving ends respectively, wherein the first feedback information comprises the packet numbers of the data packets received by the receiving ends and the time when the receiving ends receive the first feedback information of the corresponding packet numbers. In addition, the method further comprises the steps of responding to the fact that the number of the first feedback information is equal to the number of the multiple receiving ends, determining second feedback information based on the multiple pieces of first feedback information sent by all the receiving ends, and sending the second feedback information to the sending end, the second feedback information comprises the packet numbers of the data packets received by all the receiving ends and the time when the server receives the last first feedback information corresponding to the packet number. Through the mode, the transmitting end can know the conditions of receiving and releasing the data packets by the server and receiving the data packets by the plurality of receiving ends, namely, the full-link state from the transmitting end to the receiving ends can be sensed.

Description

Technical Field

[0001] One or more embodiments of the present specification relate to the field of data processing technology, and in particular, to a data processing method and device. Background Art

[0002] WebRTC (Web Real-Time Communication) is a multi-point communication technology. Generally, there are more than two clients participating in the communication. The client as the sender sends the audio and video data to the server in the form of RTP data packets, and the server forwards the RTP data packets to the client as the receiver. If the receiver receives the data packet, it returns the RTCP packet to the server.

[0003] In some related technologies, the network status can be evaluated by understanding the data transmission and reception status in the communication link, so that effective data transmission can be performed according to the network status to avoid network congestion. Summary of the invention

[0004] One or more embodiments of this specification describe a data processing method and device.

[0005] In the first aspect of the present specification, a data processing method is provided, which is executed on a server. The method includes forwarding a data packet sent by a sending end to multiple receiving ends. The method also includes receiving first feedback information sent by each of the multiple receiving ends, the first feedback information including the packet number of the data packet received by the receiving end, and the time when the receiving end receives the first feedback information corresponding to the packet number. In addition, the method also includes, in response to the number of the first feedback information being equal to the number of the multiple receiving ends, determining second feedback information based on the multiple first feedback information sent by all the receiving ends to send to the sending end, the second feedback information including the packet number of the data packet received by all the receiving ends, and the time when the server receives the last first feedback information corresponding to the packet number.

[0006] In a second aspect of the present specification, a data processing method is provided, which is executed at a transmitting end. The method includes sending a data packet to a plurality of receiving ends via a server. In addition, the method also includes receiving second feedback information sent by the server, the second feedback information is determined based on first feedback information sent by each of the plurality of receiving ends, the first feedback information includes a packet number of a data packet received by the receiving end, and a time when the receiving end receives the first feedback information corresponding to the packet number, and the second feedback information includes a packet number of a data packet received by all receiving ends, and a time when the server receives the last first feedback information corresponding to the packet number.

[0007] In a third aspect of the present specification, a data processing device is provided, which is arranged in a server. The device includes a forwarding module, which is configured to forward a data packet sent by a sending end to multiple receiving ends. The device also includes a receiving module, which is configured to receive first feedback information sent by each of the multiple receiving ends, and the first feedback information includes the packet number of the data packet received by the receiving end, and the time when the receiving end receives the first feedback information of the corresponding packet number. In addition, the device also includes a determination module, which is configured to determine second feedback information to be sent to the sending end based on multiple first feedback information sent by all receiving ends in response to the number of first feedback information being equal to the number of multiple receiving ends, and the second feedback information includes the packet number of the data packet received by all receiving ends, and the time when the server receives the last first feedback information of the corresponding packet number.

[0008] In a fourth aspect of the present specification, a data processing device is provided, which is arranged at a transmitting end. The device includes a transmitting module, which is configured to send a data packet to a plurality of receiving ends via a server. In addition, the device also includes a receiving module, which is configured to receive second feedback information sent by the server, the second feedback information is determined based on first feedback information sent by each of the plurality of receiving ends, the first feedback information includes a packet number of a data packet received by the receiving end, and a time when the receiving end receives the first feedback information corresponding to the packet number, and the second feedback information includes a packet number of a data packet received by all receiving ends, and a time when the server receives the last first feedback information corresponding to the packet number.

[0009] In a fifth aspect of the present specification, an electronic device is provided, comprising a processor and a memory. The processor is connected to the memory. The memory is used to store executable program code. The processor runs a program corresponding to the executable program code by reading the executable program code stored in the memory, so as to execute the method provided according to the first aspect or the second aspect.

[0010] In a sixth aspect of the present specification, a computer-readable storage medium is provided, on which a computer program is stored. The computer program is executed by a processor to implement the method provided according to the first aspect or the second aspect.

[0011] In a seventh aspect of the present specification, a computer program product is provided, comprising a computer program, where the computer program is executed by a processor to implement the method provided according to the first aspect or the second aspect.

[0012] It should be understood that the contents described in the Summary of the Invention are not intended to limit the key or important features of the embodiments of this specification, nor are they intended to limit the scope of this specification. Other features of this specification will become easy to understand through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] In order to more clearly illustrate the technical solutions in the embodiments of this specification, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this specification. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0014] Figure 1 A schematic diagram showing an example environment in which some embodiments of the present specification may be implemented;

[0015] Figure 2 A flowchart showing a data processing method according to some embodiments of the present specification;

[0016] Figure 3 A flowchart showing a data processing method according to other embodiments of the present specification;

[0017] Figure 4 A flowchart of a second feedback information determination process according to some embodiments of this specification is shown;

[0018] Figure 5 A schematic diagram showing a multi-party interaction process according to some embodiments of the present specification;

[0019] Figure 6 A flowchart of a packet loss determination process according to some embodiments of the present specification is shown;

[0020] Figure 7 A flowchart of a coding strategy determination process according to some embodiments of the present specification is shown;

[0021] Figure 8 A block diagram showing a data processing device according to some embodiments of the present specification;

[0022] Fig. 9 A block diagram showing a data processing device according to other embodiments of the present specification;

[0023] Fig.10 A block diagram of an electronic device in which various embodiments of the present specification can be implemented is shown. DETAILED DESCRIPTION

[0024] The embodiments of the present specification will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present specification are shown in the accompanying drawings, it should be understood that the present specification can be implemented in various forms and should not be construed as being limited to the embodiments described herein. Instead, these embodiments are provided for a more thorough and complete understanding of the present specification. It should be understood that the drawings and embodiments of the present specification are only for exemplary purposes and are not intended to limit the scope of protection of the present specification. The technical solutions in the embodiments of the present specification will be clearly and completely described below in conjunction with the drawings in the embodiments of the present specification.

[0025] The terms "first", "second", "third", etc. in this specification and claims and the above drawings are used to distinguish different objects, rather than to describe a specific order. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units that are not listed, or may optionally include other steps or units inherent to these processes, methods, products or devices.

[0026] As mentioned above, WebRTC technology is generally used in multi-point communication scenarios, such as multi-party conferences or multi-user real-time interaction. Generally, a segment of audio and video data can be split into multiple data packets. Each client first sends the data packet to the server, and then the server forwards the data packet to other clients. If a client receives a data packet, it will feedback information to the server, and the server will immediately forward the feedback information to the client that sent the data. In this process, the communication link is divided into two links, including the link segment between the sender and the server, and the link segment between the server and the receiver. The server can sense whether the receiver has received the data packet sent by the server, and the sender can sense the link status of the link segment between the sender and the server, but cannot sense the full link status from the sender to the receiver. In this way, the sender cannot globally evaluate the network status and can only regulate data transmission according to the status of the link segment between the sender and the server. In this way, the link segment between the server and the receiver may have network congestion problems.

[0027] Although, in some related technologies, in order to adapt to the changes in network bandwidth from the server to each receiving end, the sending end adopts encoding strategies such as simulcast or SVC for encoding, and these two strategies usually implement rate control at a coarse granularity and often require encoding of multiple layers, such as 500K, 1M, and 1.5M. These three code streams cannot be adapted to each receiving end, and the disadvantages are more prominent when the number of receiving ends is small.

[0028] To this end, an embodiment of the present specification proposes a data processing method, where the server temporarily stores the feedback information received from the receiving end (i.e., the first feedback information referred to in this article), and when the number of received feedback information is consistent with the number of receiving ends, all feedback information related to the same data packet is aggregated to form new feedback information (i.e., the second feedback information referred to in this article) and returns it to the sending end.

[0029] In this way, the sender can know which data packet is received by all receiving ends based on the second feedback information, and can also know how many packets are lost and which data packet is lost according to the number of packets sent and the number of second feedback information received. In this way, the sender can perceive the state of the entire link from the sender to the receiving end. In addition, the server must wait until the first feedback information of all receiving ends is collected before it can aggregate the second feedback information of the data packet. In this way, the perceived delay, packet loss rate and bandwidth changes are actually the lower limit values ​​of the network status in all aggregated receiving ends. When network evaluation is performed based on delay, packet loss rate and bandwidth changes, and data transmission is regulated based on network evaluation, the state of the global link can be considered, network congestion can be effectively avoided, and the user experience of each receiving end is improved.

[0030] Figure 1 1 is a schematic diagram of an example environment 100 in which some embodiments of the present specification may be implemented. Figure 1 As shown, the environment 100 includes a sending end 102, a server 104, and multiple receiving ends 106-1, 106-2, ..., 106-N. The sending end 102 is communicatively connected to the server 104, and the server 104 is communicatively connected to multiple receiving ends 106-1, 106-2, ..., 106-N. The sending end 102 and multiple receiving ends 106-1, 106-2, ..., 106-N can be provided in a mobile terminal, a tablet computer, a desktop computer, a laptop computer, or other device with a communication function. According to the data sending and receiving functions, the client is defined as a sending end and a receiving end. In an actual application scenario, when multiple clients are communicating and interacting, the same client can serve as both a sending end and a receiving end.

[0031] like Figure 1As shown, the transmitting end 102 splits a frame 108 into multiple data packets 108-1, 108-2, ..., 108-M, and sends the multiple data packets 108-1, 108-2, ..., 108-M to the server 104, and the server 104 sends multiple data packets 110-1, 110-2, ..., 110-M (i.e., the data packets 108-1, 108-2, ..., 108-M sent by the transmitting end) to multiple receiving ends 106-1, 106-2, ..., 106-N. Take a data packet 110-1 as an example, the data packet 110-1 is forwarded by the server to all receiving ends 106-1, 106-2, ..., 106-N. If each receiving end receives the data packet 110-1, a return data packet is generated, which contains the first feedback information. The first feedback information includes the packet number of the data packet received by the receiving end and the receiving time of the receiving end. As shown in the figure, if all receiving ends receive data packet 110-1, multiple receiving ends 106-1, 106-2, ..., 106-N each generate return data packets 112-11, 112-21, ..., 112-NM. If each receiving end can receive all data packets, M data packets can generally be received. However, when the network status is unstable, not all receiving ends can receive M data packets, and packet loss may occur. Even if a receiving end can receive M data packets, it does not rule out that some of the data packets are recovered after retransmission or FEC. Among them, M and N are natural numbers, which are the number of data packets and the number of receiving ends respectively.

[0032] like Figure 1As shown, the receiving terminals 106-1, 106-2, ..., 106-N send the formed return data packets 112-11 to 110-1M, 112-21 to 112-2M, ..., 112-N1 to 112-NM to the server 104. The server 104 receives different return data packets at different times and temporarily stores them in the server 104 after receiving them, and stores them as return data packets 114-11 to 114-NM. The server 104 makes a judgment. Once the number of return data packets receiving the same data packet is equal to the number of multiple receiving terminals, the multiple first feedback information is aggregated to form a second feedback information, and the second feedback information is included in the return data packets 116-1 to 116-m. The second feedback information includes the data packet number and the time when the server receives the return data packet of the corresponding packet number sent by the last receiving terminal. Among them, m is a natural number, m is the number of return data packets containing the second feedback information, and when M data packets are received by all receiving terminals, then m=M, otherwise, m is less than M. When the server 104 forms a return data packet, it sends it to the sending end 102. For example, when the server 104 receives a return data packet of a data packet received by all receiving ends, the server 104 forms a return data packet 116-1 containing the second feedback information and sends it to the sending end 102. Afterwards, the sending end 102 receives multiple return data packets 118-1 to 118-m containing the second feedback (i.e., return data packets 116-1 to 116-m sent by the server). The sending end 102 splices according to the packet numbers and finally splices them into a frame of picture 120. When the network status is not good, the number of return data packets received by the sending end 102 is less than the number of data packets sent, then packet loss 122 occurs and the picture 120 cannot be spliced. In this way, the sending end 102 can know whether all receiving ends can receive a frame of picture sent by it.

[0033] In some implementations, the transmitter 102 sends a plurality of data packets related to a video frame sequence to each receiving end 106-1, 106-2, ..., 106-N via the server 104. After each receiving end 106-1, 106-2, ..., 106-N receives all the data packets, it sends a return data packet containing the first feedback information to the server 104, and the server 104 generates a return data packet containing the second feedback information and sends it to the transmitter 102. The transmitter first splices the return data packets containing the second feedback information into one frame, and then splices all the frames together according to the time sequence, so that the transmitter can know that all receiving ends can receive the video sent by the transmitter.

[0034] In this way, the server temporarily stores the first feedback information of the data packet. When the number of first feedback information of the same data packet is equal to the number of multiple receiving ends, the second feedback information is determined based on the multiple first feedback information. In this way, the sending end can determine which data packet is received by all receiving ends based on the second feedback information sent by the server, and for the second feedback information that is not received, the packet loss situation can be learned after comparing the number of the sent data packet. In other words, the sending end can perceive the global link status between the sending end and the receiving end. Afterwards, the sending end can effectively send data based on the currently perceived global link status to avoid network congestion.

[0035] In addition, for some scenarios, multiple receiving ends are divided into important receiving ends and general receiving ends. Important receiving ends need to ensure that they can always receive data, and general receiving ends can allow delays or a small probability of packet loss when receiving data. Based on this, the server can also pre-configure strategies and can choose to aggregate the first feedback information of some receiving ends, such as aggregating the first feedback information sent by important receiving ends. In some examples, a live symposium, the people in the symposium are aggregated as important receiving ends, and the people watching the live broadcast are not aggregated as general receiving ends. This specification is also applicable to this scenario. When aggregating the second feedback information based on multiple first feedback information, the judgment condition can also be that the number of returned data packets of the same data packet is equal to the number of multiple important receiving ends (that is, the number of multiple first feedback information received from important receiving ends referred to in the following text is equal to the number of important receiving ends).

[0036] It should be understood that the purpose of describing the architecture and functions in the example environment 100 is only exemplary, and does not imply any limitation on the scope of the present description. The embodiments of the present description may also be applied to other environments with different structures and / or functions.

[0037] Figure 2 2 shows a flow chart of a data processing method 200 according to some embodiments of the present specification. The method 200 may be, for example, Figure 1 The server 104 in the environment 100 shown is executed. Figure 2 As shown, in block 202, method 200 may forward a data packet sent by a transmitting end to multiple receiving ends. The server does not encode the forwarded data, but forwards the received data packet. In some embodiments, when the transmitting end sends multiple RTP data packets, the server forwards each RTP data packet to all receiving ends.

[0038] In box 204, method 200 can receive first feedback information sent by multiple receiving ends respectively, and the first feedback information includes the packet number of the data packet received by the receiving end, and the time when the receiving end receives the first feedback information of the corresponding packet number. In some embodiments, the feedback information can be congestion control transmission information (Transport-wide Congestion Control, TCC) feedback information, which is an extended data of the RTP packet and uniquely identifies an RTP packet on a certain link. When the receiving end receives the data packet, the first feedback information including the packet number of the data packet received by the receiving end and the time when the data packet was received by the receiving end can be formed, and the first feedback information is sent to the server along with the RTCP packet. The server temporarily stores the multiple first feedback information received. When the receiving end does not receive a data packet, the RTCP packet cannot be generated, and the server will not receive the first feedback information.

[0039] In box 206, method 200 can determine second feedback information to be sent to the sending end based on multiple first feedback information sent by all receiving ends in response to the number of first feedback information being equal to the number of multiple receiving ends, and the second feedback information includes the packet number of the data packet received by all receiving ends, and the time when the server receives the last first feedback information of the corresponding packet number. In some implementations, the server will receive first feedback information sent from different receiving ends at different times. The same receiving end may send multiple first feedback information related to different data packets. When the server recognizes that the number of first feedback information related to a certain data packet received is equal to the number of multiple receiving ends, a second feedback information is formed based on the multiple first feedback information related to the data packet. At this time, the second feedback information is sent to the sending end, and in the case where the number of first feedback information related to a certain data packet received is less than the number of multiple receiving ends, the first feedback information is still temporarily stored in the server, and after the response condition is met, the second feedback information is formed and sent to the sending end.

[0040] like Figure 2 As shown, in some embodiments, based on multiple first feedback information, the time when the server receives the last first feedback information of the corresponding packet number is determined. The second feedback information is determined based on the packet number of the data packet related to the first feedback information and the time when the server receives the last first feedback information of the corresponding packet number. In this way, the second feedback information and the first feedback information both include the packet number of the data packet received by the receiving end, and the second feedback information also includes the local time of the server when the last receiving end sends the first feedback information to the server. That is, the latest receiving time among the multiple first feedback information.

[0041] In this way, the server temporarily stores the received first feedback information, and determines whether the data packet is received by all receiving ends based on the response conditions on the server side, that is, the server side first evaluates the link status of the link segment between the server and the receiving end. Afterwards, the server will form a second feedback information to the sending end based on the evaluation result of all receiving ends receiving the data packet. The sending end can determine which data packets are received by all receiving ends based on the second feedback information. In this way, the sending end can perceive the full link network status from the sending end to the receiving end.

[0042] Figure 3 FIG. 3 is a flow chart showing a data processing method 300 according to some other embodiments of the present specification. Figure 3 As shown, the method 300 may be performed by, for example Figure 1 The sending end 102 in the environment 100 shown in FIG. Figure 3 As shown, in block 302, method 300 may send data packets to multiple receiving ends via a server. In some embodiments, a sending end sends a video to multiple receiving ends. The sending end splits the video into frames according to the video frame sequence of the video, and each frame is split into multiple data packets. Each minute of video may be split into thousands of data packets. The sending end sends multiple data packets to multiple receiving ends via a server.

[0043] In box 304, method 300 may receive second feedback information sent by the server, the second feedback information is determined based on first feedback information sent by multiple receiving ends respectively, the first feedback information includes the packet number of the data packet received by the receiving end, and the time when the receiving end receives the first feedback information of the corresponding packet number, and the second feedback information includes the packet number of the data packet received by all receiving ends, and the time when the server receives the last first feedback information of the corresponding packet number. In one example, the sending end sends M data packets to multiple receiving ends, and when M second feedback information is received, all receiving ends receive M data packets. In another example, when M-2 second feedback information is received, 2 data packets are not received by all receiving ends. By comparing the data packet number in the received second feedback information and the packet number of the M data packets sent by the sending end, it can be known which two data packets are lost. In addition, it is also possible to evaluate whether a data packet has been recovered due to retransmission or FEC during segmented transmission based on the data packet reception time in the second feedback information.

[0044] Figure 4A flowchart of a second feedback information determination process 400 according to some embodiments of the present specification is shown. In box 402, the process 400 may receive first feedback information. In box 404, the process 400 may determine whether the number of first feedback information of a certain data packet received is equal to the number N of receiving terminals. If they are equal, the second feedback information 406 is determined based on the first feedback information of the data packet. In an example, data packet 1 is sent to the server, and the server forwards it to receiving terminals 1, 2, and 3. Receiving terminals 1, 2, and 3 form first feedback information 1, first feedback information 2, and second feedback information 3, which are (data packet 1, receiving time 1:30), (data packet 1, receiving time 1:23), and (data packet 1, receiving time 1:27), respectively. The server receives three first feedback information, all of which are data packet 1, and the received local times are 1:45, 1:30, and 1:40, respectively. The number of first feedback information is equal to the number of receiving terminals. Then, based on the three first feedback information, the server determines that the second feedback information is (data packet 1, receiving time 1:45). The receiving end 1 is the last one to send the first feedback information 1 to the server, and the local time when the server receives the first feedback information 1 is 1:45.

[0045] like Figure 4 As shown, when block 404 determines that the number is not equal, the process returns to block 402 and waits for receiving other first feedback information. If other first feedback information is received within a certain period of time, the process of subsequent blocks 404 and 406 is performed. If other first feedback information is not received after a certain period of time, a receiving end fails to receive a data packet.

[0046] Figure 5A schematic diagram 500 of a multi-party interaction process 500 according to some embodiments of the present specification is shown. The process 500 interacts between a sending end 502, a server 504, and two receiving ends 506-1 and 506-2. The figure shows two receiving ends in an example form and explains in the specification. In practice, it is not limited to two receiving ends according to the interaction scenario, and there may be more receiving ends. The sending end 502 splits a frame of the picture into multiple data packets, and in process 510, the sending module 508 of the sending end 502 sends the data packet to the server 504. The server 504 forwards the received data packet through the forwarding module 518 of the server 504 in processes 512-1 and 512-2, and forwards it to the receiving ends 506-1 and 506-2 respectively. After the receiving end 506-1 receives the data packet, the processing module 514-1 of the receiving end 506-1 generates a return data packet, which contains the first feedback information. Similarly, after receiving the data packet, the processing module 514-2 of the receiving end 506-1 also generates a return data packet, which contains the first feedback information. In processes 516-1 and 516-2, the processing module 514-1 of the receiving end 506-1 and the processing module 514-2 of the receiving end 506-2 send the return data packet to the server 504 respectively. The determination module 520 of the server 504 determines that the two received first feedback information are equal to the number of the receiving end, and then determines the second feedback information based on the two first feedback information. Afterwards, in process 522, the server 504 sends the return data packet containing the second feedback information to the sending end 502.

[0047] like Figure 5 As shown, the method of the embodiment of this specification can also determine the network delay based on the second feedback information. In some implementations, the network delay is determined based on the time when the server receives the last corresponding packet number of the first feedback information, the time required for the server to send the second feedback information to the sender, and the time when the sender sends the data packet. Figure 5 For example, the time it takes for the sender to send a data packet is T send The local time when the receiving end 506-1 receives the data packet to form the first feedback information is T r0 , when the receiving end 506-1 is the last to send the first feedback information to the server 504, the time when the server 504 receives the first feedback information sent by the receiving end 506-1 is T r1 The server 504 generates a second feedback message based on the two first feedback messages and sends it to the sender 502. The sender receives the second feedback message at a local time of T r2 . T r2 =T is equal to the sum of the time it takes for the server to receive the last corresponding packet number of the first feedback information in the second feedback information and the time it takes for the server to send the second feedback information to the sender. In this way, the sender can calculate the network delay = Tr2 -T send .

[0048] In some implementations, the sender can determine changes in network bandwidth based on network delay. When the network delay increases, it means that the network is congested, which means that the sender sends too much data and needs to reduce the amount of data sent, so the sender reduces the encoding bit rate. When the network delay does not increase, it may decrease, or fluctuate within a range, which means that the data sent by the sender does not reach the network bandwidth. At this time, you can try to increase the encoding bit rate. In this way, the sender can determine a bit stream based on the perception of the current network state. Compared with the existing mulcast or SVC encoding strategies, the bit rate control granularity is finer and can adapt to changes in the network bandwidth from the server to each receiving end. When the network state changes, a bit stream can be re-determined, which can still adapt to changes in the network bandwidth from the server to each receiving end.

[0049] Figure 6 FIG. 6 is a flow chart of a packet loss determination process 600 according to some embodiments of the present specification. Figure 6 As shown, the process 600 can determine the packet loss rate based on the second feedback information. The transmitting end sends multiple data packets related to the video to multiple receiving ends via the server. The packet loss rate can be determined based on the result of transmitting a frame of the picture, or based on the result of transmitting a video frame sequence, or based on the result of transmitting for a period of time. In some embodiments, the number of packet losses is determined based on multiple pieces of the second feedback information and all the data packets sent. The packet loss rate is determined based on the number of packet losses and the number of all the data packets sent. Figure 6 As shown, in box 602, the sending end can obtain the second feedback information 602, and box 604 determines whether the packet numbers of all the data packets in the obtained second feedback information are consistent with all the packet numbers of the sent data packets. If they are consistent, there is no packet loss 608. If they are inconsistent, box 610 can determine the packet number of the data packet not in the second feedback information as the packet number of the lost packet, and can also determine the number of lost packets.

[0050] like Figure 6As shown, box 606 also compares the time in the second feedback information with the standard time accuracy. Generally, the standard time accuracy of normal transmission is 1ms. If there is a decimal point in the time of the second feedback information, that is, it is judged as no, then it is considered that the packet has been lost during the transmission process, and it may be obtained by the receiving end requesting the server to retransmit, or it may be obtained through the FEC method. Box 612 determines the data packet judged as no as a recovery packet. Box 606 makes an accuracy judgment on each second feedback information one by one. If the accuracy judgment is consistent, it returns to box 602, obtains the next second feedback information, and performs the judgment of box 606. Afterwards, box 614 adds the number of lost packets to the number of recovered packets to obtain the total number of lost packets. Box 616 calculates the percentage of the total number of lost packets to the total number of sent data packets to determine the packet loss rate.

[0051] Figure 7 FIG. 7 is a flowchart of a coding strategy determination process 700 according to some embodiments of the present specification. In some implementations, the transmitting end may also determine the network status based on network delay, packet loss rate, and bandwidth change. Furthermore, based on the network status, the coding strategy of the video frame sequence corresponding to the plurality of data packets is determined. Figure 7 As described, box 702 determines whether the packet loss rate in the network state is greater than 0. If it is equal to 0, there is no packet loss, and the first encoding strategy of box 704 is executed. If it is greater than 0, there is packet loss, and the hybrid encoding strategy of box 706 is executed, that is, the first encoding strategy and the second encoding strategy are used in combination. Optionally, according to box 710, the first encoding strategy difference encoding of the next frame based on the previous frame as the reference frame of some frames in the video frame sequence, and according to box 712, the remaining frames in the video frame sequence are encoded based on the second encoding strategy, and the reference frame is selected from the frames that can be received by all receiving ends for encoding. After the sending end receives the second feedback information of the same number as the sent data packet, it can be known that the network state is good and there is no packet loss, and the previous frame is selected as the reference frame to encode the next frame according to box 708.

[0052] In some embodiments, a reference frame sequence is formed based on the frames corresponding to the data packets received by all receiving ends. Based on the coding efficiency and packet loss rate, some frames in the video frame sequence are encoded based on any frame in the reference frame sequence as a reference frame, and some frames in the video frame sequence are encoded with the previous frame as a reference frame. In some examples, a reference frame sequence consisting of multiple frames without packet loss is obtained. These frames are owned by each receiving end, and the transmitting end encodes them with these reference frames. After receiving them, each receiving end can decode them normally. However, since these reference frames and encoded frames may not be directly adjacent, the coding efficiency will be reduced. Therefore, we will mix the reference frames and the physically directly adjacent previous encoded frames according to the network packet loss status to achieve a balance between coding efficiency and anti-packet loss. For example, the odd-numbered frames adopt the second strategy encoding, that is, select a frame from the reference frame sequence as a reference frame for differential encoding, and the even-numbered frames adopt the first encoding strategy, that is, the previous frame in the adjacent frames is used as a reference frame to encode the next frame. Assume that frames 6 and 7 are lost, so the receiving end cannot decode frame 8, because frame 8 is encoded based on the first encoding strategy, and frame 9 is encoded based on the second encoding strategy, which can be decoded by the receiving end. If each frame is to be received and decoded, frame 8 can be encoded based on the second encoding strategy according to the packet loss situation. If frame 7 is not lost, considering the encoding efficiency, the first encoding strategy is preferentially used to encode frame 8.

[0053] In this way, the sender perceives the network status based on the second feedback information received, and needs to change the reference frame encoding for frames that are not received normally, so that the code stream encoded by the sender can be received by all receiving ends and can be correctly decoded.

[0054] The above is a description of a specific embodiment of the present specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0055] Figure 8 8 shows a block diagram of a data processing device 800 according to some embodiments of the present specification. Fig. 9 As shown, the device 800 can be arranged, for example, at Figure 1The server 104 in the environment 100. The device 800 includes a forwarding module 802, which is configured to forward a data packet sent by a sending end to multiple receiving ends. The device 800 also includes a receiving module 804, which is configured to receive first feedback information sent by each of the multiple receiving ends, the first feedback information includes the packet number of the data packet received by the receiving end, and the time when the receiving end receives the first feedback information of the corresponding packet number. In addition, the device 800 also includes a determination module 806, which is configured to determine second feedback information to be sent to the sending end based on multiple first feedback information sent by all receiving ends in response to the number of the first feedback information being equal to the number of the multiple receiving ends, the second feedback information includes the packet number of the data packet received by all the receiving ends, and the time when the server receives the last first feedback information of the corresponding packet number.

[0056] Fig. 9 FIG. 9 is a block diagram of a data processing device 900 according to some other embodiments of the present specification. The device 900 may be arranged, for example, at Figure 1 The sending end 102 in the environment 100. The device includes a sending module 902, which is configured to send a data packet to multiple receiving ends via a server. In addition, the device also includes a receiving module 904, which is configured to receive second feedback information sent by the server, the second feedback information is determined based on the first feedback information sent by each of the multiple receiving ends, the first feedback information includes the packet number of the data packet received by the receiving end, and the time when the receiving end receives the first feedback information corresponding to the packet number, and the second feedback information includes the packet number of the data packet received by all the receiving ends, and the time when the server receives the last first feedback information corresponding to the packet number.

[0057] In some embodiments, the apparatus 900 further includes a network evaluation module configured to determine network delay, packet loss rate, and bandwidth changes based on the second feedback information. The network evaluation module is further configured to determine a network state based on the network delay, packet loss rate, and bandwidth changes, and determine an encoding strategy for a video frame sequence corresponding to a plurality of data packets based on the network state.

[0058] Fig.10 1 shows a block diagram of an electronic device 1000 that can implement multiple embodiments of the present specification. Fig.10 As shown, the device 1000 includes a processor 1001, which can perform various appropriate actions and processes according to computer program instructions stored in a read-only memory (ROM) 1002 and loaded into a random access memory (RAM) 1003. In the RAM 1003, various programs and data required for the operation of the device 1000 can also be stored. The processor 1001, the ROM 1002, and the RAM 1003 are connected to each other via a bus 1004. An input / output (I / O) interface 1005 is also connected to the bus 1004.

[0059] The various processes and processing described above, such as methods 200, 300, may be performed by processor 1001. For example, in some embodiments, methods 200, 300 may be implemented as a software program, which is tangibly contained in a machine-readable medium. In some embodiments, part or all of the software program may be loaded and / or installed on device 1000 via ROM 1102. When the software program is loaded into RAM 1003 and executed by processor 1001, one or more actions of methods 200, 300 described above may be performed.

[0060] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chip systems (SOCs), load programmable logic devices (CPLDs), and the like.

[0061] The program code for implementing the method of this specification can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, causes the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code can be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0062] This specification can be a method, device, system and / or program product. The program product can include a machine-readable storage medium, which is loaded with machine-readable program instructions for executing various aspects of this specification. The machine-readable program instructions described here can be downloaded from the machine-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device through a network, such as the Internet, a local area network, a wide area network and / or a wireless network. The network can include copper transmission cables, optical fiber transmissions, wireless transmissions, routers, firewalls, switches, gateway computers and / or edge servers. The network adapter card or network interface in each computing / processing device receives the machine-readable program instructions from the network, and forwards the machine-readable program instructions for storage in the machine-readable storage medium in each computing / processing device.

[0063] The machine program instructions for performing the operation of this specification can be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-related instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, etc., and conventional procedural programming languages ​​such as "C" language or similar programming languages. The machine-readable program instructions can be executed completely on the user's computer, partially on the user's computer, as an independent software package, partially on the user's computer and partially on the remote computer, or completely on the remote computer or server. In the case of a remote computer, the remote computer can be connected to the user's computer through any type of network including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (for example, using an Internet service provider to connect through the Internet). In some embodiments, by using the state information of the machine-readable program instructions to personalize an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA) or a programmable logic array (PLA), the electronic circuit can execute the machine-readable program instructions, thereby realizing various aspects of this specification.

[0064] In the context of this specification, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In addition, although each operation is depicted in a specific order, this should be understood as requiring such operations to be performed in the specific order shown or in a sequential order, or requiring that all illustrated operations should be performed to obtain the desired result. In certain environments, multitasking and parallel processing may be advantageous. Likewise, although several specific implementation details are included in the above discussion, these should not be construed as limiting the scope of this specification. Certain features described in the context of separate embodiments may also be implemented in a single implementation in combination. Conversely, various features described in the context of a single implementation may also be implemented in multiple implementations individually or in any suitable sub-combination.

[0065] Although the subject matter has been described in language specific to structural features and / or methodological logical actions, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. On the contrary, the specific features and actions described above are merely example forms of implementing the claims.

Claims

1. A data processing method, executed on a server, comprising: Forwarding data packets sent by the sender to multiple receivers; receiving first feedback information sent by each of the multiple receiving ends, wherein the first feedback information includes a packet number of the data packet received by the receiving end and a time when the receiving end receives the first feedback information corresponding to the packet number; as well as In response to the number of the first feedback information being equal to the number of the multiple receiving ends, second feedback information is determined based on the multiple first feedback information sent by all receiving ends to be sent to the sending end, the second feedback information including the packet number of the data packet received by all receiving ends, and the time when the server received the last first feedback information corresponding to the packet number.

2. The method according to claim 1, wherein in response to the number of the first feedback information being equal to the number of the plurality of receiving ends, determining the second feedback information based on the plurality of first feedback information sent by all receiving ends comprises: In response to the number of the first feedback information being equal to the number of the plurality of receiving ends, confirming the data packet associated with the first feedback information as the data packet received by all receiving ends; Based on the plurality of first feedback information, determining the time at which the server receives the last first feedback information corresponding to the packet number; as well as The second feedback information is determined based on the packet number of the data packet related to the first feedback information and the time when the server receives the last first feedback information corresponding to the packet number.

3. A data processing method, executed at the sending end, comprising: Send data packets to multiple receiving ends via the server; as well as receiving second feedback information sent by the server, where the second feedback information is determined based on the first feedback information sent by each of the multiple receiving ends, the first feedback information including a packet number of the data packet received by the receiving end, and a time when the receiving end receives the first feedback information corresponding to the packet number, and the second feedback information including the packet number of the data packet received by all receiving ends, and a time when the server receives the last first feedback information corresponding to the packet number.

4. The method according to claim 3, further comprising: Based on the second feedback information, changes in network delay, packet loss rate, and bandwidth are determined.

5. The method according to claim 4, wherein determining the network delay based on the second feedback information comprises: The network delay is determined based on the time when the server receives the last first feedback information corresponding to the packet number in the second feedback information, the time required for the server to send the second feedback information to the sending end, and the time when the sending end sends the data packet.

6. The method according to claim 4, wherein the sending end sends a plurality of the data packets related to the video, the sending end receives a plurality of the second feedback information sent by the server, and determining the packet loss rate based on the second feedback information comprises: Determining the number of packet losses based on the plurality of second feedback information and all sent data packets; as well as The packet loss rate is determined based on the number of packet losses and the number of all sent data packets.

7. The method according to claim 6, wherein determining the number of packet losses based on the plurality of second feedback information and all sent data packets comprises: Determine the packet number of the data packet that has not been received based on the packet numbers of the data packets in the plurality of second feedback information and the packet numbers of all sent data packets; In response to the time when the server in the second feedback information receives the last first feedback information corresponding to the packet number being not equal to the standard time precision, determining that the data packet corresponding to the packet number is a recovery packet; The number of packet losses is determined based on the data packets and the recovery packets that have not been received.

8. The method according to claim 4 or 5 or 6 or 7, further comprising: Determining a network status based on the network delay, the packet loss rate, and the bandwidth change; Based on the network status, an encoding strategy for the video frame sequences corresponding to the multiple data packets is determined.

9. The method according to claim 8, wherein based on the network status, determining the encoding strategy comprises: In response to the packet loss rate in the network state being greater than 0, determining a packet number of a lost packet; Based on the frames corresponding to the data packets received by all the receiving ends, a reference frame sequence is formed; Based on the coding efficiency and the packet loss rate, some frames in the video frame sequence are encoded based on any frame in the reference frame sequence as a reference frame, and some frames in the video frame sequence are encoded based on a previous frame as a reference frame.

10. The method according to claim 9, wherein based on the network status, determining the encoding strategy further comprises: In response to the packet loss rate in the network state being equal to 0, a previous frame among adjacent frames in a video frame sequence is determined as a reference frame required for encoding a subsequent frame.

11. A data processing device, located on a server, comprising: A forwarding module, configured to forward data packets sent by a sending end to multiple receiving ends; a receiving module, configured to receive first feedback information sent by each of the multiple receiving ends, wherein the first feedback information includes a packet number of the data packet received by the receiving end, and a time when the receiving end receives the first feedback information corresponding to the packet number; as well as A determination module is configured to determine, in response to the number of the first feedback information being equal to the number of the multiple receiving ends, second feedback information based on multiple first feedback information sent by all receiving ends to be sent to the sending end, wherein the second feedback information includes the packet number of the data packet received by all receiving ends, and the time when the server receives the last first feedback information corresponding to the packet number.

12. A data processing device, provided at the sending end, comprising: A sending module, configured to send data packets to multiple receiving ends via the server; as well as A receiving module is configured to receive second feedback information sent by the server, the second feedback information is determined based on the first feedback information sent by each of the multiple receiving ends, the first feedback information includes the packet number of the data packet received by the receiving end, and the time when the receiving end receives the first feedback information corresponding to the packet number, and the second feedback information includes the packet number of the data packet received by all receiving ends, and the time when the server receives the last first feedback information corresponding to the packet number.

13. The apparatus according to claim 12, further comprising: The network evaluation module is configured to determine network delay, packet loss rate and bandwidth changes based on the second feedback information.

14. The apparatus according to claim 13, wherein the network assessment module is further configured to: Determining a network status based on the network delay, the packet loss rate, and the bandwidth change; Based on the network status, an encoding strategy for the video frame sequences corresponding to the multiple data packets is determined.

15. Electronic equipment, including processor and memory; The processor is connected to the memory; The memory is used to store executable program code; The processor runs a program corresponding to the executable program code by reading the executable program code stored in the memory, so as to execute the method according to any one of claims 1 to 10.

16. A computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the method according to any one of claims 1 to 10.

Images