Data processing method and apparatus, and device, program and readable medium

Abstract

The present disclosure relates to the field of data processing. Provided are a data processing method and apparatus, and a device, a program and a readable medium. The method is applied to a first terminal device, and comprises: acquiring a plurality of first data packets; performing sniffing processing on the plurality of first data packets, so as to determine a target adversarial mode, wherein the target adversarial mode comprises a first adversarial mode or a second adversarial mode, the first adversarial mode is used for generating error correction data for random packet loss, and the second adversarial mode is used for generating error correction data for burst packet loss; according to the target adversarial mode, performing an exclusive-OR operation on the plurality of first data packets, so as to generate a plurality of pieces of error correction data; and combining the plurality of pieces of error correction data and the plurality of first data packets into a second data packet, and sending the second data packet to a server.

Description

A data processing method, device, equipment, program and readable medium

[0001] The present disclosure claims priority to the Chinese patent application No. 2024106350430, filed on May 21, 2024, entitled "A data processing method, system, device, equipment, program and readable medium", the entire content of which is incorporated herein by reference. TECHNICAL FIELD

[0002] Embodiments of the present disclosure relate to the field of data processing, in particular to a data processing method, device, equipment, program and readable medium. BACKGROUND

[0003] At present, there are many products based on Android on the market, and audio-video products are one of them. We often see various forms of multimedia advertisements in elevators, department stores and buses, and large-screen displays of audio-video programs in many occasions (such as government posters, public welfare propaganda, department stores, etc.). These multimedia data need to be monitored and played, that is, monitored and broadcast.

[0004] However, in a weak network environment (such as an underground garage, a remote area, an elevator, etc.), due to weak signal and slow network speed, it is extremely difficult to monitor and broadcast multimedia data streams. The commonly used solution is to set up an edge gateway to increase a step of caching, so as to communicate between the gateway and the monitoring and broadcasting service. However, this method greatly reduces the real-time performance and practicability of multimedia data monitoring and broadcasting. Therefore, how to effectively monitor and broadcast multimedia data in a weak network environment has become a problem to be solved in the field. SUMMARY

[0005] The first aspect of the embodiments of the present disclosure provides a data processing method applied to a first terminal device, and the method comprises:

[0006] obtaining a plurality of first data packets;

[0007] performing sniffing processing on the plurality of first data packets to determine a target countermeasure mode, the target countermeasure mode comprising a first countermeasure mode or a second countermeasure mode, the first countermeasure mode being used to generate error correction data for random packet loss, and the second countermeasure mode being used to generate error correction data for burst packet loss;

[0008] performing exclusive OR operation on the plurality of first data packets according to the target countermeasure mode to generate a plurality of error correction data;

[0009] combining the plurality of error correction data and the plurality of first data packets into a second data packet and sending the second data packet to a server.

[0010] In an optional implementation, the sniffing processing of the plurality of first data packets and the determination of the target countermeasure mode comprise:

[0011] In the initial frame sequence, the first data packets are sequentially transmitted to the server according to the first countermeasure mode, and feedback messages returned by the server are attempted to be received.

[0012] In a case where the target frame does not receive the feedback message returned by the server, the target countermeasure mode is determined as the first countermeasure mode, the target frame being any frame in the initial frame sequence.

[0013] In an optional implementation, the transmitting of the first data packets to the server according to the first countermeasure mode and the attempting to receive the feedback message returned by the server comprise:

[0014] Based on the packet loss information of the target frame, the plurality of first data packets are subjected to an exclusive OR operation according to the first countermeasure mode, to generate a second data packet of the target frame.

[0015] The second data packet of the target frame is transmitted to the server, and a first feedback message returned by the server is attempted to be received, the first feedback message being a feedback message for the target frame.

[0016] In an optional implementation, in a case where the target frame is a first frame of the initial frame sequence, the packet loss information of the target frame is initial packet loss information.

[0017] In a case where the target frame is a non-first frame of the initial frame sequence, the packet loss information of the target frame is packet loss information determined based on the first feedback message.

[0018] In an optional implementation, in a case where the target frame receives the feedback message returned by the server, the method further comprises:

[0019] The redundancy of the first data packet is expanded, and data transmission to the server is performed according to the first countermeasure mode in a next frame of the target frame.

[0020] In a case where a second feedback message returned by the server is received, the step of expanding the redundancy of the first data packet and performing data transmission to the server according to the first countermeasure mode in the next frame of the target frame is continuously performed until the redundancy of the first data packet is greater than or equal to a first threshold value, the target countermeasure mode is determined as a second countermeasure mode, and the second feedback message is a feedback message for the next frame of the target frame.

[0021] In an optional implementation, the method further comprises:

[0022] In response to the feedback message for the target frame, determining packet loss information of the target frame;

[0023] Augmenting the redundancy of the first data packet, and performing XOR operation on the augmented first data packet based on the packet loss information of the target frame according to the first countermeasure mode, to generate a second data packet of a next frame of the target frame;

[0024] Sending the second data packet of the next frame of the target frame to the server, and attempting to receive the second feedback message.

[0025] In an optional implementation, in the case that the target frame receives the feedback message returned by the server, the method further comprises:

[0026] In response to the feedback message for the target frame, obtaining packet loss distribution characteristics of a first sub-frame sequence, the first sub-frame sequence being the target frame and all frames before the target frame;

[0027] Based on the packet loss distribution characteristics, determining the target countermeasure mode.

[0028] In an optional implementation, the method of determining the target countermeasure mode based on the packet loss distribution characteristics comprises:

[0029] Based on the packet loss distribution characteristics, predicting packet loss information of a second sub-frame sequence, determining a prediction failure rate, the second sub-frame sequence being a plurality of subsequent frames of the target frame;

[0030] In the case that the prediction failure rate is greater than or equal to a second threshold value, determining the target countermeasure mode as the second countermeasure mode;

[0031] In the case that the prediction failure rate is less than the second threshold value, determining the target countermeasure mode as the first countermeasure mode.

[0032] In an optional implementation, the target countermeasure mode is the first countermeasure mode, and the method of performing XOR operation on the plurality of first data packets according to the target countermeasure mode to generate a plurality of error correction data comprises:

[0033] Grouping the plurality of first data packets to obtain a plurality of first data pools, each of the first data pools containing a plurality of first data packets;

[0034] performing exclusive OR operation on the first data packets in each of the first data pools to generate the error correction data corresponding to the first data pools.

[0035] In an alternative embodiment, after the combination of the error correction data and the first data packets into the second data packet is sent to the server, the method further comprises:

[0036] In response to the feedback message returned by the server for the second data packet, grouping the first data packets obtained in the next frame to obtain a plurality of third data pools arranged in sequence, each of the third data pools containing a plurality of first data packets, and at least one same first data packet being contained in adjacent third data pools;

[0037] performing exclusive OR operation on the first data packets in each of the third data pools to generate the optimized error correction data corresponding to the third data pools;

[0038] combining the optimized error correction data and the first data packets in the next frame into a second data packet of the next frame and sending the second data packet to the server.

[0039] In an alternative embodiment, the target adversarial mode is the second adversarial mode, and the performing exclusive OR operation on the first data packets in the plurality of first data pools to generate the error correction data comprises:

[0040] grouping the first data packets to obtain a plurality of first data pools, each of the first data pools containing a plurality of first data packets;

[0041] reassigning the first data packets in the first data pools to obtain a plurality of second data pools, each of the second data pools containing at least one first data packet from each of the first data pools, and different second data pools containing different first data packets;

[0042] performing exclusive OR operation on the first data packets in each of the second data pools to generate the error correction data corresponding to the second data pools.

[0043] In an alternative embodiment, after the combination of the error correction data and the first data packets into the second data packet is sent to the server, the method further comprises:

[0044] In response to the feedback message returned by the server for the second data packet, grouping the first data packets obtained in the next frame to obtain a plurality of fourth data pools, each of the fourth data pools containing a plurality of first data packets, and at least two fourth data pools containing at least one same first data packet;

[0045] redistributing the first data packets of the plurality of fourth data pools to obtain a plurality of fifth data pools, each fifth data pool containing at least one first data packet from each fourth data pool;

[0046] performing exclusive OR operation on the first data packets in each fifth data pool to generate a plurality of optimized error correction data corresponding to the plurality of fifth data pools;

[0047] combining the plurality of optimized error correction data and the first data packets of the next frame into the second data packets of the next frame and sending the second data packets of the next frame to a server.

[0048] In an optional implementation, after the plurality of first data packets are obtained, the method further comprises:

[0049] generating a preset picture group frame value based on the first data packets, the preset picture group frame value being used to represent a rated frame number of each picture group;

[0050] sending the preset picture group frame value to the server, so that the server performs a rectification operation on a target data stream corresponding to the second data packets based on the preset picture group frame value.

[0051] In an optional implementation, the method is applied to a server and comprises:

[0052] receiving second data packets sent by a first terminal device, the second data packets comprising a plurality of error correction data and a plurality of first data packets;

[0053] processing the second data packets to restore the plurality of first data packets based on the plurality of error correction data and generate third data packets;

[0054] in a case where there is data loss in the third data packets, generating a feedback message and sending the feedback message to the first terminal device.

[0055] In an optional implementation, after the third data packets are generated, the method further comprises:

[0056] receiving a preset picture group frame value sent by the first terminal device;

[0057] decoding the third data packets to generate a target data stream;

[0058] performing a rectification operation on the target data stream according to the preset picture group frame value, so that each picture group frame value in the target data stream is equal to the preset picture group frame value, to obtain an optimized data stream, wherein the rectification operation at least comprises frame filtering or frame copying;

[0059] encoding the optimized data stream to obtain fourth data packets.

[0060] A third aspect of the embodiments of the present disclosure provides a data processing system, the system comprising a first device terminal and a server;

[0061] The first device terminal is configured to acquire a plurality of first data packets, perform sniffing processing on the plurality of first data packets, and determine a target adversarial mode, wherein the target adversarial mode comprises a first adversarial mode or a second adversarial mode, the first adversarial mode is used to generate error correction data for random packet loss, and the second adversarial mode is used to generate error correction data for burst packet loss; perform XOR operation on the plurality of first data packets according to the target adversarial mode to generate a plurality of error correction data; and combine the plurality of error correction data and the plurality of first data packets into a second data packet and send the second data packet to the server.

[0062] The server is configured to receive the second data packet sent by the first terminal device, perform processing on the second data packet, restore the plurality of first data packets based on the plurality of error correction data to generate a third data packet, and in the case that there is data loss in the third data packet, generate a feedback message and send the feedback message to the first terminal device.

[0063] A fourth aspect of the embodiments of the present disclosure provides a data processing apparatus applied to a first terminal device, the apparatus comprising:

[0064] An original data acquisition module is configured to acquire a plurality of first data packets.

[0065] A mode determination module is configured to perform sniffing processing on the plurality of first data packets, and determine a target adversarial mode, wherein the target adversarial mode comprises a first adversarial mode or a second adversarial mode, the first adversarial mode is used to generate error correction data for random packet loss, and the second adversarial mode is used to generate error correction data for burst packet loss.

[0066] An XOR operation module is configured to perform XOR operation on the plurality of first data packets according to the target adversarial mode to generate a plurality of error correction data.

[0067] A first sending module is configured to combine the plurality of error correction data and the plurality of first data packets into a second data packet and send the second data packet to a server.

[0068] A fifth aspect of the embodiments of the present disclosure provides a data processing apparatus applied to a server, the apparatus comprising:

[0069] A data packet receiving module is configured to receive a second data packet sent by the first terminal device, wherein the second data packet comprises a plurality of error correction data and a plurality of first data packets.

[0070] a reduction module, configured to process the second data packet to reduce the first data packets based on the error correction data, to generate a third data packet;

[0071] a feedback module, configured to generate a feedback message and send the feedback message to the first terminal device in a case that there is data loss in the third data packet.

[0072] The sixth aspect of the embodiments of the present disclosure provides a computing processing device, comprising:

[0073] a memory, in which computer readable code is stored; and

[0074] one or more processors, when the computer readable code is executed by the one or more processors, the computing processing device performs the data processing method according to any one of the first aspect, or the computing processing device performs the data processing method according to any one of the second aspect.

[0075] The seventh aspect of the embodiments of the present disclosure provides a computer program, comprising computer readable code, when the computer readable code is run on a computing processing device, causing the computing processing device to perform the data processing method according to any one of the first aspect, or causing the computing processing device to perform the data processing method according to any one of the second aspect.

[0076] The eighth aspect of the embodiments of the present disclosure provides a computer readable medium, in which the computer program according to the seventh aspect is stored.

[0077] The above description is only a summary of the technical solutions of the present disclosure, in order to more clearly understand the technical means of the present disclosure, the specific embodiments of the present disclosure can be implemented according to the content of the description, and in order to make the above and other purposes, characteristics and advantages of the present disclosure more obvious and easy to understand, the following specific embodiments of the present disclosure are described. Beneficial effects:

[0078] The present disclosure provides a data processing method, device, equipment, program and readable medium, applied to a first terminal device, the method comprising: obtaining a plurality of first data packets; sniffing and processing the plurality of first data packets to determine a target countermeasure mode, the target countermeasure mode comprising a first countermeasure mode or a second countermeasure mode, the first countermeasure mode being used to generate error correction data for random packet loss, and the second countermeasure mode being used to generate error correction data for burst packet loss; performing exclusive or operation on the plurality of first data packets according to the target countermeasure mode to generate a plurality of error correction data; and combining the plurality of error correction data and the plurality of first data packets into a second data packet and sending the second data packet to a server.

[0079] The disclosure determines different weak network countermeasures through sniffing processing, so that when multimedia data is transmitted in a weak network environment, different countermeasures are used to perform XOR operation on the original data for different packet loss types, so that the server can effectively recover and send the lost packet data to the monitoring terminal according to the error correction data generated by different countermeasures in various weak network environments, and realize effective monitoring and broadcasting of multimedia data in a weak network environment.

[0080] The above description is only a summary of the technical solutions of the disclosure. In order to more clearly understand the technical means of the disclosure, the disclosure can be implemented according to the content of the specification, and in order to make the above and other purposes, characteristics and advantages of the disclosure more obvious and easy to understand, the following specific embodiments of the disclosure are described. BRIEF DESCRIPTION OF DRAWINGS

[0081] In order to more clearly illustrate the technical solutions in the embodiments of the disclosure or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or the prior art description. Obviously, the drawings in the following description are some embodiments of the disclosure, and those skilled in the art can also obtain other drawings according to these drawings without creative labor.

[0082] Fig. 1 schematically shows a method flowchart for executing the data processing method of the disclosure on the first terminal device side;

[0083] Fig. 2 schematically shows a processing flowchart for executing the data processing method of the disclosure on the first terminal device side;

[0084] Fig. 3 schematically shows a processing flowchart for determining the target countermeasure mode for executing the data processing method of the disclosure on the first terminal device side;

[0085] Fig. 4 schematically shows a packet loss distribution characteristic diagram in the data processing method proposed by the disclosure;

[0086] Fig. 5 schematically shows a principle diagram for generating error correction data based on the first countermeasure mode in the data processing method proposed by the disclosure;

[0087] Fig. 6 schematically shows a principle diagram for generating error correction data based on the second countermeasure mode in the data processing method proposed by the disclosure;

[0088] Fig. 7 schematically shows a principle diagram for generating error correction data based on the improved first countermeasure mode in the data processing method proposed by the disclosure;

[0089] Fig. 8 schematically shows a principle diagram for generating error correction data based on the improved second countermeasure mode in the data processing method proposed by the disclosure;

[0090] Fig. 9 schematically shows a method flow chart for executing the data processing method of the present disclosure on the server side;

[0091] Fig. 10 schematically shows a processing flow chart for executing the data processing method of the present disclosure on the server side;

[0092] Fig. 11 schematically shows a frame per second statistics chart of a video original stream data with still pictures constantly repeated;

[0093] Fig. 12 schematically shows a frame per second statistics chart of a video after rectification operation in the data processing method of the present disclosure executed on the server side;

[0094] Fig. 13 schematically shows a block diagram of a computing processing device for executing the method according to the present disclosure;

[0095] Fig. 14 schematically shows a storage unit for holding or carrying program codes for implementing the method according to the present disclosure. Specific Embodiments

[0096] For the purpose, technical solutions and advantages of the embodiments of the present disclosure to be clearer, the technical solutions in the embodiments of the present disclosure will be described clearly and completely below with reference to the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are part of the embodiments of the present disclosure, rather than all the embodiments. Based on the embodiments in the present disclosure, all other embodiments obtained by those of ordinary skill in the art without any creative work fall within the scope of protection of the present disclosure.

[0097] At present, there are many products based on Android on the market, and audio-visual products are one of them. We often see various forms of multimedia advertisements in elevators, department stores and buses, and also see large-screen displays of audio-visual programs in many occasions (such as government posters, public welfare propaganda, department stores, etc.). These multimedia data need to be monitored and played, that is, monitored and broadcasted.

[0098] However, in a weak network environment (such as an underground garage, a remote area, an elevator, etc.), the signal is weak and the network speed is slow. In the transmission layer of video communication, due to unstable wireless network signal or unreliable Internet link, etc., network packet loss and network delay may occur in a weak network environment, which causes data loss and delay when multimedia data such as audio and video is sent from a terminal device in a weak network environment to a server, so that it becomes extremely difficult for a monitoring terminal to monitor based on the multimedia data stream distributed by the server. This is because the negative impact of too high packet loss rate on the transmission of original data packets is disastrous. For example, for an 80% packet loss rate, a raw data packet is transmitted 10 times, and the final arrival rate is only 90%. However, for video multimedia data, each frame of video usually contains at least 3 to 5 original data packets, and it is still very difficult to require all of them to be transmitted every time to arrive at the server for complete decoding and rendering.

[0099] A commonly used solution in the prior art is to set an edge gateway to increase a step of buffering, so that the gateway and the monitoring service communicate, and for a device application deployment complex and high delay scene, this method has a certain effect. However, on the one hand, due to the need for one-step buffering before sending by setting the gateway, the monitoring service in a weak network environment has a large delay; on the other hand, the gateway method needs to deploy many additional devices, such as hotspots in a weak network environment, which greatly increases the cost.

[0100] Therefore, the embodiment of the present disclosure proposes a data processing method applied to a first terminal device. FIG. 1 schematically shows a method flowchart for executing the data processing method of the present disclosure on the first terminal device side. As shown in FIG. 1, the method comprises the following steps:

[0101] S101, obtaining a plurality of first data packets.

[0102] S102, performing sniffing processing on the plurality of first data packets to determine a target adversarial mode.

[0103] S103, performing XOR operation on the plurality of first data packets according to the target adversarial mode to generate a plurality of error correction data.

[0104] S104, combining the plurality of error correction data and the plurality of first data packets into a second data packet and sending the second data packet to a server.

[0105] In the embodiments of the present disclosure, the first terminal device is a display device with display function and data processing function, and exemplarily, the first terminal device includes but is not limited to an LED display screen, an LCD display screen, an OLED display screen, etc., and the first terminal device is configured to display multimedia data. The first data packet is an encoded data packet corresponding to multimedia data of the first terminal device, and contains encoded data corresponding to original data streams of a plurality of different application programs and system services in the first terminal device.

[0106] In the process of data transmission from the first terminal device to the server, the weak network environment or the hardware and software problems of the terminal device itself may affect the data transmission process, resulting in packet loss in the data transmission process, and thus causing data loss in the data generated by the server after decoding and recovering the first data packet. The types of the packet loss may include random packet loss and burst packet loss. The random packet loss refers to that, in the process of transmission of the first data packet from the first terminal device to the server, the first data packet is lost randomly and continuously with relatively fixed packet loss information due to noise or collision. The burst packet loss refers to that, at a certain time in the network, a large number of adjacent first data packets are suddenly lost due to certain time or conditions. The burst packet loss phenomenon is usually caused by certain specific events or conditions (for example, link breakage), and has a great influence on network performance and service quality, and may cause the service quality of real-time services (such as monitoring and broadcasting services) to be seriously affected.

[0107] To cope with data loss caused by packet loss, a forward error correction (FEC) mode is used to generate redundant error correction codes corresponding to the first data packets as error correction data, and the first data packets and the error correction data are sent to the server together, so that the server recovers the lost data packets based on the error correction data, reducing the adverse effects of packet loss on network performance and service quality. However, due to the large difference in the distribution of lost data packets between random packet loss and burst packet loss in the data transmission process, the error correction data generated based on a single forward error correction mode for recovering lost data cannot be effectively applied to both types of packet loss phenomena at the same time. Therefore, the embodiments of the present disclosure determine a target countermeasure mode by sniffing the plurality of first data packets, the target countermeasure mode including a first countermeasure mode or a second countermeasure mode, the first countermeasure mode being used to generate error correction data for random packet loss, and the second countermeasure mode being used to generate error correction data for burst packet loss. Based on the differences between random packet loss and burst packet loss, the first countermeasure mode and the second countermeasure mode generate error correction data in different ways. Therefore, the error correction data generated by the server based on the first countermeasure mode can effectively recover data packets lost due to random packet loss, and the error correction data generated by the server based on the second countermeasure mode can effectively recover data packets lost due to burst packet loss, thereby effectively improving the weak network countermeasure capability for various packet loss phenomena.

[0108] In the embodiments of the present disclosure, after determining the target countermeasure mode to be applied in the current data transmission process, an exclusive OR operation is performed on the plurality of first data packets based on the determined target countermeasure mode (e.g., the first countermeasure mode or the second countermeasure mode) to generate a plurality of error correction data, and then the plurality of error correction data and the plurality of first data packets are combined into second data packets and sent to the server, so that the server recovers the lost data based on the error correction data generated by the target countermeasure mode. The error correction data is generated by performing an exclusive OR operation on the first data packets. The exclusive OR operation is a binary operation that refers to an operation of the same position of a piece of data and another piece of data. In the field of encoding and decoding, the exclusive OR operation is used for setting the symbol bit. It is easy to understand that the exclusive OR operation can be used to restore the original data through reverse operation, for example, (same as 0, different as 1), so In this way

[0109] The embodiments of the present disclosure determine the target countermeasures mode for different packet loss phenomena through sniffing processing, and based on the target countermeasures mode, error correction data for effectively recovering data packets can be generated in each type of packet loss, so that the first terminal device can effectively prevent data loss when transmitting data to the server in a weak network environment and / or in the case that the terminal device itself has software and hardware problems, thereby improving transmission efficiency and reducing network delay, and macroscopically showing good weak network environment countermeasures capability.

[0110] FIG. 2 schematically shows a processing flowchart for executing the data processing method of the present disclosure on the first terminal device side. In combination with FIG. 2, in order to better enable those skilled in the art to better understand the scheme of the present disclosure, the data processing method applied to the first terminal device provided by the embodiments of the present disclosure is described in detail as follows.

[0111] In the specific implementation step S101, the first terminal device includes one or more application programs or system services that generate multimedia data, and the first terminal device synthesizes and displays the multimedia data on the screen to form a corresponding display effect. The first terminal device first acquires an original data stream, which is to-be-transmitted data transmitted to the server and at least includes data in one or more application programs or system services in the first terminal device. After acquiring the original data stream, the original data stream is encoded at the video coding layer (VCL) to generate original encoded data corresponding to the original data stream, and the original encoded data is used to represent the data content of the original data stream. The encoding process can be performed in an asynchronous manner, or in a synchronous manner, or in other types of encoding manner to generate corresponding original encoded data. The specific encoding processing manner can be determined according to actual conditions, as long as it conforms to the technical concept of the present disclosure, which is not limited in the present disclosure.

[0112] In some optional embodiments, after acquiring the original data stream, an encoding object is first created, and the encoding type (such as video encoding) and the corresponding encoding parameters are specified. Then, the attribute information of the to-be-encoded original data stream is configured, and the attribute information at least includes the data format, the encoder parameters for guiding the encoding process, etc. Based on the encoding type, the encoding parameters and the attribute information, the data of the original data stream is encoded to generate the original encoded data corresponding to the original data stream.

[0113] It should be noted that the encoding format of the original encoded data generated in the embodiments of the present disclosure is determined according to the data type of the original data stream. For example, when the original data stream is video data, the encoding format of the original encoded data includes but is not limited to H264, H265 (HEVC), etc. When the original data stream is audio data, the encoding format of the original encoded data includes but is not limited to mp3, AAC, etc.

[0114] After the original encoded data corresponding to the original data stream is generated at the video encoding layer, the original encoded data is formatted and provided with header information at the network abstract layer (NAL), so as to encapsulate the original encoded information into a first data packet in NAL format, to ensure the effective transmission of the first data packet on various channels and storage media. After the first data packet in NAL format is generated, the first data packet in NAL format is encapsulated into a plurality of first data packets in a corresponding transmission format based on the transmission format. For example, the first data packet in NAL format is encapsulated into a plurality of first data packets in avc format of mp4, or the first data packet in NAL format is encapsulated into a plurality of first data packets in rtp format for socket transmission.

[0115] In some optional embodiments, the first terminal device is an Android terminal device, the Android terminal device is equipped with an Android system, and the Android system is a free and open source mobile operating system based on a Linux kernel and mainly applied to mobile devices such as televisions, digital cameras, game consoles, smart watches and the like. Among a plurality of system services included in the Android terminal device, a SurfaceFlinger service is included, the SurfaceFlinger service is a core system service in the Android, and the SurfaceFlinger service is mainly responsible for the composition of screen display content. Specifically, the SurfaceFlinger service receives image buffers from a plurality of application programs and system services, composes the image buffers into a final buffer according to attributes such as positions, sizes, transparencies and Z-axis sequences of the image buffers, and then sends the final buffer to a terminal display device for display. Therefore, the multimedia data of each application program and system service in the first terminal device can be sampled, rendered and integrated by the SurfaceFlinger service, so that the basic function of original recording and broadcasting is realized. In other words, by calling the SurfaceFlinger service in the Android terminal device, the original data stream of a plurality of applications and system services in the terminal device can be obtained. Therefore, the SurfaceFlinger service is called first to obtain the original data stream, the original data stream at least includes data of a plurality of applications in the first terminal device, then a system-level codec library MediaCodec of the Android is called to sample and encode the original data stream to generate original encoded data, and finally the original encoded data is data-encapsulated to obtain the plurality of first data packets.

[0116] By using the SurfaceFlinger service, the application isolation between the applications in the Android terminal device is broken through, the original data stream of a plurality of application programs and system services can be uniformly obtained, the data acquisition efficiency in the data transmission process is improved, and the processing delay is reduced.

[0117] It is easy to understand that the process of obtaining the first data packet by the first terminal device in the embodiments of the present disclosure includes but is not limited to the process of obtaining the first data packet by the Android terminal device described above, as long as the process of obtaining the first data packet is applicable to the scheme concept of the present disclosure.

[0118] In some optional embodiments, after the first data packet is obtained by collecting the original data stream and performing encoding processing, since the collection of the original data stream can cause the problem of frame freezing due to the "lazy" mode of the collection process, for example, for the process of collecting the original data stream by the SurfaceFlinger service of the Android terminal device, the SurfaceFlinger service will not collect images when the picture is not moving. For the data transmission of live streaming, problems such as timeout, missing key data recovery, and stream interruption and resumption can occur, and the user's intuitive feeling is that the picture is frozen. FIG. 11 schematically shows a frame per second statistical diagram of a video original stream data with a still picture repeatedly. As shown in FIG. 11, when the original data stream collected based on the SurfaceFlinger service contains a still picture repeatedly, the frame number of the video original stream data fluctuates greatly when output from the Android terminal device to the second terminal device through the server, generally between 40 to 15 frames, which greatly reduces the user experience. Therefore, in the embodiments of the present disclosure, after obtaining the plurality of first data packets, a preset picture group frame value is generated based on the first data packets, and the preset picture frame value is used to represent the rated frame number of each picture group. Specifically, a custom NAL packet is generated based on the first data packets, and the custom NAL packet contains a parameter for representing the preset picture frame value of the picture group, and the parameter format is 0001 (NAL delimiter) 000D (NAL type) + (0xFF) nal payload, wherein the payload is a 2-bit value representing the frame number of the current picture group. The NAL type of the custom NAL packet is a reserved bit, for example, the NAL type of the reserved bit can be any one of 13-23, for example, the NAL type of the reserved bit can be 13, and if the original data stream has a private protocol occupied, the NAL type is modified to other reserved bits. Finally, the custom NAL packet corresponding to the preset picture frame value is sent to the server, so that the server performs rectification operation on the target data stream corresponding to the second data packet based on the preset picture frame value.

[0119] In the implementation of step S102, after generating the plurality of first data packets, the first terminal device needs to generate error correction data based on the plurality of first data packets, so that the server recovers the lost data packets in transmission through the error correction data. During the process of data transmission from the first terminal device to the server, the packet loss phenomenon that occurs includes at least random packet loss phenomenon and burst packet loss phenomenon. Due to the significant difference in packet loss distribution and time node between the two, in order to enable the server to effectively recover data based on error correction data regardless of data loss caused by various types of packet loss phenomena in a weak network scenario, the embodiments of the present disclosure need to first determine a target countermeasure mode for countering weak network packet loss. The target countermeasure mode includes a first countermeasure mode and a second countermeasure mode. The first countermeasure mode is used to generate error correction data for random packet loss. The second countermeasure mode is used to generate error correction data for burst packet loss. Different target countermeasure modes are used to adaptively counter different packet loss phenomena in a weak network scenario.

[0120] Figure 3 schematically shows a processing flow chart for determining the target countermeasure mode on the first terminal device side for executing the data processing method of the present disclosure. As shown in Figure 3, in the embodiments of the present disclosure, the target countermeasure mode is determined by sniffing the plurality of first data packets. The sniffing process can determine the target countermeasure mode based on the redundancy of the first data packets, or based on the packet loss distribution characteristics. In the case of determining the target countermeasure mode based on any one of the redundancy and the packet loss distribution characteristics in the sniffing process, the sniffing process is ended. Specifically, in the initial frame sequence, the first data packets are sequentially transmitted to the server according to the first countermeasure mode, and the feedback message returned by the server is tried to be received. The initial frame sequence is a plurality of consecutive frames when the first data packets start to be transmitted to the server. In each frame of the initial frame sequence, a certain redundancy of first data packets is sequentially obtained according to the frame order. The redundancy is the number of first data packets for transmission in each frame. The first data packets of the target frame are subjected to exclusive OR operation according to the first countermeasure mode, to generate error correction data corresponding to the target frame. The target frame is any one frame in the initial frame sequence. The error correction data of the target frame and the plurality of first data packets corresponding to the target frame are combined into a second data packet and sent to the server. In the initial frame sequence, the specific steps of performing exclusive OR operation on the first data packets of the target frame based on the first countermeasure mode can refer to the description of exclusive OR operation when the target countermeasure mode is the first countermeasure mode below. The present disclosure will not be repeated here.

[0121] In some optional embodiments, the first terminal device performs exclusive-OR operation on the plurality of first data packets according to the first countermeasure mode based on packet loss information of the target frame, to generate second data packets of the target frame, wherein the packet loss information is information representing the first data packets that are not lost after being transmitted to the server, for example, the packet loss information can be the number of each first data packet that is not lost after being transmitted to the server, the packet loss information can be a packet loss rate calculated based on each first data packet that is not lost after being transmitted to the server, etc.; the second data packets of the target frame are transmitted to the server, and the first terminal device attempts to receive a first feedback message returned by the server, wherein the first feedback message is a feedback message for the target frame.

[0122] In the embodiments of the present disclosure, in the process of performing exclusive-OR operation on the target frame based on the first countermeasure mode or the second countermeasure mode, the first data packets need to be grouped first, wherein the number of first data packets in each group depends on the packet loss information of the previous frame, so as to ensure that the error correction data generated by each group can at least represent the data characteristics of the plurality of first data packets satisfying the packet loss information, for example, the packet loss information is a packet loss rate, and in the case that the packet loss rate is 30%, at least 3 first data packets are grouped to perform exclusive-OR operation to generate corresponding error correction data. In the case that the target frame is the first frame of the initial frame sequence, the packet loss information of the target frame is initial packet loss information, and the initial packet loss information is a preset value, for example, the packet loss information is a packet loss rate, and the initial packet loss information is 10%. In the case that the target frame is a non-first frame of the initial frame sequence, the packet loss information of the target frame is packet loss information determined based on the first feedback message.

[0123] The server recovers and restores the data lost in the transmission process based on the first data packets and the error correction data in the received second data packets. In the case that the data packets after recovery and restoration still have data loss compared with the second data packets, the server generates a corresponding first feedback message and returns the first terminal device. If the first terminal device does not receive the first feedback message returned by the server in the target frame, it indicates that the packet loss problem can be solved based on the first countermeasure mode in the current sniffing process, and the current packet loss phenomenon is a random packet loss phenomenon, that is, the target countermeasure mode is determined as the first countermeasure mode.

[0124] If the first terminal device receives the first feedback message returned by the server in the target frame, it indicates that the packet loss problem cannot be completely solved based on the first countermeasure mode in the current sniffing process, and there is still partial data loss. At this time, the reason can be that the packet loss phenomenon is not a random packet loss phenomenon, or that the redundancy of the target frame is too small to cause insufficient error correction data to completely recover the lost data. Therefore, in the case where the target frame receives the first feedback message, the embodiment of the disclosure determines the reason for receiving the first feedback message by a first threshold, the first threshold being the maximum value of the redundancy under a random packet loss phenomenon. If the redundancy of the target frame is less than the first threshold, it is considered that the redundancy of the target frame is too small to cause insufficient error correction data to completely recover the lost data, and the redundancy is expanded in the next frame, and the data transmission is continued in the first countermeasure mode.

[0125] In some optional embodiments, in the case where the target frame receives the first feedback message returned by the server, if the redundancy of the target frame is less than the first threshold, the packet loss information of the target frame is determined in response to the first feedback message of the target frame. Subsequently, the redundancy of the first data packet is expanded according to a preset proportion or a preset redundancy increment, and the first data packet after expansion is subjected to an exclusive OR operation based on the packet loss information of the target frame and the first countermeasure mode, to generate a second data packet of the next frame of the target frame. The second data packet of the next frame of the target frame is sent to the server, and a second feedback message is attempted to be received, the second feedback message being a feedback message for the next frame of the target frame. Similarly, if the first terminal device does not receive the second feedback message returned by the server in the next frame of the target frame, it indicates that the packet loss problem can be solved based on the first countermeasure mode in the current sniffing process, and it is determined that the current packet loss phenomenon is a random packet loss phenomenon, that is, the target countermeasure mode is determined as the first countermeasure mode.

[0126] If the first terminal device receives the second feedback message returned by the server in the next frame of the target frame, it indicates that the packet loss problem cannot be completely solved based on the first countermeasure mode in the current sniffing process, and there is still partial data loss. At this time, the reason can be that the packet loss phenomenon is not a random packet loss phenomenon, or that the redundancy of the next frame of the target frame is too small to cause insufficient error correction data to completely recover the lost data. Therefore, in the case where the second feedback message returned by the server is received, the step of expanding the redundancy of the first data packet and performing data transmission to the server in the first countermeasure mode in the next frame of the target frame is continued until the redundancy of the first data packet is greater than or equal to the first threshold. At this time, it is determined that the data loss is not caused by a random packet loss phenomenon, and the target countermeasure mode is determined as the second countermeasure mode.

[0127] In some optional embodiments, since the random packet loss phenomenon is lost with relatively fixed packet loss information randomly and continuously, the distribution of the packet loss information is relatively uniform; the burst packet loss is a large number of adjacent first data packets suddenly lost, and the distribution of the packet loss information is more concentrated. Based on this, the target countermeasure mode can also be determined according to the packet loss distribution characteristics in the sniffing process. In the sniffing process, the target countermeasure mode is determined based on any one of the redundancy and the packet loss distribution characteristics, the sniffing process is ended, and the first data packet of the subsequent frame is transmitted according to the determined target countermeasure mode. Specifically, after the target frame receives the feedback message returned by the server, the packet loss distribution characteristics of the first subframe sequence are obtained in response to the feedback message of the target frame, and the first subframe sequence is the target frame and all frames before the target frame. Since the target frame and all frames before the target frame are transmitted in the first countermeasure mode, it can be considered that the packet loss distribution characteristics of the first subframe sequence can reflect the distribution characteristics of the random packet loss.

[0128] If the packet loss information of the subsequent frame is consistent with the packet loss distribution characteristics of the first subframe sequence, it can be considered that the packet loss type of the subsequent frame is random packet loss; if the packet loss information of the subsequent frame is not consistent with the packet loss distribution characteristics of the first subframe sequence, it can be considered that the packet loss type of the subsequent frame is burst packet loss. Therefore, after determining the packet loss distribution characteristics, the target countermeasure mode can be determined based on the packet loss distribution characteristics. Specifically, the packet loss information of the second subframe sequence is predicted based on the packet loss distribution characteristics, and the prediction failure rate is determined, and the second subframe sequence is a plurality of frames after the target frame. In the embodiment of the disclosure, the packet loss distribution characteristics of the first subframe sequence are used as the prediction result, the first data packet of each frame in the second subframe sequence is transmitted to the server based on the first countermeasure mode, and the actual packet loss distribution characteristics of each frame in the second subframe sequence are determined based on the feedback message sent by the server. In the case where the actual packet loss distribution characteristics and the prediction result are consistent, it is considered that the frame prediction is successful; otherwise, in the case where the actual packet loss distribution characteristics and the prediction result are not consistent, it is considered that the frame prediction fails. The prediction failure rate of all frames in the second subframe sequence is counted. In the case where the prediction failure rate is greater than or equal to a second threshold, it is considered that the packet loss type after the target frame is burst packet loss phenomenon, and the target countermeasure mode is determined as the second countermeasure mode; in the case where the prediction failure rate is less than the second threshold, it is considered that the packet loss type after the target frame is random packet loss phenomenon, and the target countermeasure mode is determined as the first countermeasure mode.

[0129] Optionally, the distribution feature can be characterized by a Gaussian data sampling distribution, in response to a feedback message for the target frame, obtaining the packet loss rate of the first subframe sequence as packet loss information, performing Gaussian sampling distribution on the packet loss information of the first subframe sequence, determining a target Gaussian distribution interval, and taking the target Gaussian distribution interval as the packet loss distribution feature. FIG. 4 schematically shows a packet loss distribution feature in the data processing method proposed in the disclosure. As shown in FIG. 4, if the packet loss information in the first subframe sequence falls within [μ-σ, μ+σ] in the Gaussian distribution interval, then [μ-σ, μ+σ] is taken as the target Gaussian distribution interval. Since the first subframe sequence reflects the distribution feature of random packet loss, the target Gaussian distribution interval can be used to predict the packet loss type of each subsequent frame. If the packet loss information of the subsequent frame is within the target Gaussian distribution interval, it can be considered that the packet loss type of the subsequent frame is random packet loss; if the packet loss information of the subsequent frame is outside the target Gaussian distribution interval, it can be considered that the packet loss type of the subsequent frame is burst packet loss.

[0130] Subsequently, based on the target Gaussian distribution interval, a target interval corresponding to the packet loss information of the first subframe sequence is determined as a prediction interval of the second subframe sequence; in the second subframe sequence, the first data packet is sequentially transmitted to the server according to the first adversarial mode, and based on the feedback message sent by the server, the actual packet loss information of each frame in the second subframe sequence is determined. In the case where the actual packet loss information is within the prediction interval [μ-σ, μ+σ], it is determined that the frame prediction is successful; otherwise, in the case where the actual packet loss information is outside the prediction interval [μ-σ, μ+σ], it is determined that the frame prediction fails. The prediction failure rate of all frames in the second subframe sequence is counted, and the prediction failure rate is used to represent the probability that the multiple packet loss information in the second subframe sequence does not fall within the prediction interval. In the case where the prediction failure rate is greater than or equal to a second threshold value, it is considered that the packet loss type after the target frame is a burst packet loss phenomenon, and the target adversarial mode is determined as the second adversarial mode; in the case where the prediction failure rate is less than the second threshold value, it is considered that the packet loss type after the target frame is a random packet loss phenomenon, and the target adversarial mode is determined as the first adversarial mode.

[0131] In the step S103, after determining the target adversarial pattern based on the sniffing processing, exclusive OR operation is performed on the plurality of first data packets based on the target adversarial pattern to generate error correction data for recovering the missing data. FIG. 5 schematically shows a principle diagram of generating error correction data based on the first adversarial pattern in the data processing method proposed by the present disclosure. As shown in FIG. 5, in the case that the target adversarial pattern is the first adversarial pattern, the plurality of error correction data is generated in the following manner. First, the plurality of first data packets is grouped according to the arrangement order to obtain a plurality of first data pools, each of which contains a plurality of first data packets, and the plurality of first data packets in each first data pool are adjacent first data packets. The redundancy of the first data packets contained in each first data pool can be determined based on the packet loss information corresponding to the feedback message returned by the server in the last frame. In the case that no feedback message is received in the last frame, the packet loss information corresponding to the feedback message returned by the server in the last time is determined. Then, exclusive OR operation is performed on the plurality of first data packets in each first data pool to generate the plurality of error correction data corresponding to the plurality of first data pools.

[0132] For example, in FIG. 5, the current frame image includes a plurality of first data packets p1-p10 arranged in sequence, the packet loss rate corresponding to the feedback message returned by the server in the last time is 30%, the plurality of first data packets is grouped in groups of at least 3 first data packets to obtain three first data pools P1, P2 and P3, the first data pool P1 contains three adjacent first data packets (p1, p2, p3), the first data pool P2 contains three adjacent first data packets (p4, p5, p6), and the first data pool P3 contains four adjacent first data packets (p7, p8, p9, p10). Exclusive OR operation is performed on the first data packets in each first data pool to generate a plurality of error correction data f1, f2 and f3. Specifically, for the first data pool P1, exclusive OR operation is performed on the first data packets p1, p2 and p3 to generate the error correction data f1. The error correction data f1 is used to recover the data p1, p2 and p3 in the first data pool P1. For the first data pool P2, exclusive OR operation is performed on the first data packets p4, p5 and p6 to generate the error correction data f2. The error correction data f2 is used to recover the data p4, p5 and p6 in the first data pool P2. For the first data pool P3, exclusive OR operation is performed on the first data packets p7, p8, p9 and p10 to generate the error correction data f3. The error correction data f3 is used to recover the data p7, p8, p9 and p10 in the first data pool P2. It should be noted that the above example is only one specific implementation to help the skilled in the art better understand the scheme of the present disclosure, and in actual implementation, only the technical concept consistent with the scheme of the present disclosure is required, and the specific scheme is not limited by the present disclosure.

[0133] In some optional embodiments, Fig. 7 schematically shows a schematic diagram of generating error correction data based on the improved first adversarial mode in the data processing method according to the present disclosure. As shown in Fig. 7, in the case that the target adversarial mode is the first adversarial mode, in order to prevent all the first data packets in the data pool corresponding to the error correction data from being lost, a plurality of error correction data can also be generated in the following manner: first, the plurality of first data packets are grouped to obtain a plurality of fifth data pools, each of the fifth data pools contains a plurality of first data packets, and adjacent fifth data pools contain at least one same first data packet, so that for each fifth data pool, there is another fifth data pool sharing at least one first data packet, i.e., there are at least two first data packets generating error correction data. The redundancy of the first data packets contained in each fifth data pool can be determined based on the packet loss information corresponding to the feedback message returned by the server in the last frame, and in the case that no feedback message is received in the last frame, the packet loss information corresponding to the feedback message returned by the server in the last time is determined. Then, the plurality of first data packets in each fifth data pool are subjected to XOR operation to generate the plurality of error correction data corresponding to the plurality of fifth data pools. In the present embodiment of the present disclosure, by sharing at least one first data packet between adjacent fifth data pools, the situation that all the first data packets in a data pool are lost and the data packets in the data pool cannot be recovered is prevented, and when all the first data packets in a fifth data pool are lost, the first data packets in the fifth data pool can be recovered based on the adjacent fifth data pool.

[0134] For example, in Fig. 7, the current frame image includes a plurality of first data packets p1-p7 arranged in sequence, the packet loss rate corresponding to the feedback message returned by the server in the last time accepted is 30%, and the plurality of first data packets are grouped in groups of at least three first data packets to obtain three fifth data pools P1, P2 and P3. The fifth data pool P1 contains three adjacent first data packets (p1, p2, p3), the fifth data pool P2 contains three adjacent first data packets (p3, p4, p5), and the fifth data pool P3 contains three adjacent first data packets (p5, p6, p7). The first data packets in each fifth data pool are subjected to XOR operation to generate a plurality of error correction data f1, f2 and f3. Specifically, for the fifth data pool P1, XOR operation is performed For the fifth data pool P2, XOR operation is performed For the fifth data pool P3, XOR operation is performed When the first data packets p3, p4 and p5 contained in the fifth data pool P2 are all lost in the transmission process, there are shared first data packets in the fifth data pool P2 and the adjacent P1 and P3, respectively, so that The first data packet p3 is recovered by means of... The first data packet p5 is recovered in this way, and then... The first data packet p4 is recovered in this way, thereby achieving effective recovery of all first data packets in the fifth data pool P2. It should be noted that the above example is only a specific implementation method given to enable those skilled in the art to better understand the solution of this disclosure. In actual implementation, it is only necessary to conform to the technical concept of the solution of this disclosure. This disclosure does not limit the specific solution.

[0135] In some optional implementations, Figure 6 schematically illustrates the principle of generating error correction data based on the second adversarial mode in the data processing method proposed in this disclosure. As shown in Figure 6, when the target adversarial mode is the second adversarial mode, multiple error correction data are generated as follows: First, the multiple first data packets are grouped according to their arrangement order to obtain multiple first data pools. Since the loss of first data packets in sudden packet loss is concentrated in the large-scale loss of adjacent data packets, and the first data pool is the error correction data generated by directly XORing adjacent first data packets, directly generating error correction data from the first data pool can easily lead to the direct loss of all first data packets in the first data pool corresponding to one error correction data in the event of sudden packet loss. Therefore, the second adversarial mode redistributes the data packets in the first data pool to ensure that the first data packets corresponding to each generated error correction data are multiple first data packets that are evenly distributed, thereby avoiding the inability to effectively recover the error correction data and improving the weak network adversarial effect under sudden packet loss. Specifically, the first data packets of the plurality of first data pools are redistributed to obtain a plurality of second data pools. Each second data pool contains at least one first data packet from each of the first data pools, and the first data packets contained in different second data pools are different. Subsequently, the plurality of first data packets in each second data pool are XORed to generate the plurality of error correction data corresponding one-to-one with the plurality of second data pools.

[0136] Taking FIG. 6 as an example, the current frame image includes a plurality of first data packets p1-p10 arranged in sequence, the packet loss rate corresponding to the feedback message returned by the server accepted last time is 30%, the plurality of first data packets are grouped into three first data pools P1, P2 and P3 by taking at least three first data packets as a group, the first data pool P1 contains three adjacent first data packets (p1, p2, p3), the first data pool P2 contains three adjacent first data packets (p4, p5, p6), and the first data pool P3 contains four adjacent first data packets (p7, p8, p9, p10). One data packet is collected from each first data pool to generate three second data pools P4, P5 and P6, the second data pool P4 contains four first data packets (p1, p4, p7, p10), the second data pool P5 contains three first data packets (p2, p5, p8), and the second data pool P6 contains three first data packets (p3, p6, p9). The first data packets in each second data pool are subjected to exclusive OR operation respectively to generate a plurality of error correction data f1, f2 and f3. Specifically, for the second data pool P4, the exclusive OR operation is performed on p1, p4, p7 and p10 to generate the error correction data f1 The error correction data f1 is used to recover the data p1, p4, p7 and p10 in the second data pool P4; for the second data pool P5, the exclusive OR operation is performed on p2, p5 and p8 to generate the error correction data f2 The error correction data f2 is used to recover the data p2, p5 and p8 in the second data pool P5; for the second data pool P6, the exclusive OR operation is performed on p3, p6 and p9 to generate the error correction data f3 The error correction data f3 is used to recover the data p3, p6 and p9 in the second data pool P6. It should be noted that the above example is only one specific implementation given to make the technical personnel better understand the scheme of the present disclosure, and in actual implementation, only the technical concept consistent with the scheme of the present disclosure is required, and the specific scheme is not limited by the present disclosure.

[0137] In the embodiment of the present disclosure, the error correction data corresponding to the second data pool is generated based on the second adversarial mode, which ensures that the first data packets contained in each second data pool are uniformly distributed in all first data packets corresponding to the frame. When burst packet loss occurs, the data packets in the second data pool corresponding to each error correction data will not all be lost, which ensures the effective recovery ability of the error correction data to the data packets in the second data pool.

[0138] In some optional embodiments, Fig. 8 schematically shows a schematic diagram of generating error correction data based on the improved second adversarial mode in the data processing method according to the present disclosure. As shown in Fig. 8, in the case that the target adversarial mode is the second adversarial mode, in order to prevent all the first data packets in the data pool corresponding to the error correction data from being lost, the plurality of error correction data can also be generated in the following manner: first, the plurality of first data packets are grouped in sequence to obtain a plurality of sixth data pools, each of the sixth data pools contains a plurality of adjacent first data packets, and at least two sixth data pools contain at least one same first data packet. The redundancy of the first data packets contained in each sixth data pool can be determined based on the packet loss information corresponding to the feedback message returned by the server in the last frame. In the case that no feedback message is received in the last frame, the packet loss information corresponding to the feedback message returned by the server in the last time is determined. Then, the first data packets in the plurality of sixth data pools are re-allocated to obtain a plurality of seventh data pools, each of the seventh data pools contains at least one first data packet from each of the sixth data pools; then, the plurality of first data packets in each of the seventh data pools are subjected to an exclusive OR operation to generate the plurality of error correction data. In the present embodiment of the present disclosure, by the mutual redundancy between the seventh data pools, at least one first data packet is shared in a manner to prevent the case that all the first data packets in a data pool are lost to cause the data packets in the data pool to be unrecoverable. When all the first data packets in a seventh data pool are lost, the other seventh data pools having the same first data packet can be used for recovery.

[0139] For example, in Fig. 8, the current frame image includes a plurality of first data packets p1-p11 arranged in sequence, the packet loss rate corresponding to the feedback message returned by the server in the last time accepted is 30%, the plurality of first data packets are grouped in groups of at least three first data packets to obtain four sixth data pools P1, P2, P3 and P4, wherein P4 and P3 have the same first data packet p7. One data packet is collected from each of the three different sixth data pools to generate four seventh data pools P5, P6, P7 and P8, the seventh data pool P5 contains three first data packets (p1, p4, p7), the seventh data pool P6 contains three first data packets (p2, p5, p8), the seventh data pool P7 contains three first data packets (p3, p6, p9), and the seventh data pool P8 contains three first data packets (p7, p10, p11). The first data packets in each of the seventh data pools are subjected to an exclusive OR operation to generate a plurality of error correction data f1, f2, f3 and f4. It should be noted that the above example is only a specific embodiment given to enable those skilled in the art to better understand the scheme of the present disclosure, and in actual implementation, only the technical concept consistent with the scheme of the present disclosure is required, and the specific scheme is not limited by the present disclosure.

[0140] In the step S104, after the error correction data corresponding to the plurality of first data packets generated based on the target adversarial pattern is generated, the error correction data and the plurality of first data packets corresponding to the error correction data are combined into a second data packet, which is sent to the server through a network protocol. Optionally, in order to ensure the reachability of the second data packet, when the second data packet is transmitted, the network transmission time delay (RTT) of the error correction data is set according to the node network condition, for example, when the network transmission time delay RTT of the node network is less than 20 ms, the network transmission time delay of the error correction data is set to 20 ms.

[0141] In some optional embodiments, when the target adversarial pattern is the first adversarial pattern, after the second data packet generated based on the first adversarial pattern is sent to the server, there may still be a random packet loss phenomenon, at which time the second data packet generated by the improved first adversarial pattern can be sent to the server when data transmission is performed in the next frame. Specifically, as shown in FIG. 7, after the plurality of error correction data and the first data packet are combined into a second data packet and sent to the server, in response to the feedback message returned by the server for the second data packet, a plurality of third data pools in sequence are obtained by grouping a plurality of first data packets obtained in the next frame, each of the third data pools contains a plurality of first data packets, and adjacent third data pools contain at least one same first data packet; the plurality of first data packets in each third data pool are subjected to an exclusive OR operation to generate a plurality of optimized error correction data corresponding to the plurality of third data pools; and the plurality of optimized error correction data and the first data packet of the next frame are combined into a second data packet of the next frame and sent to the server.

[0142] In some optional embodiments, in the case that the target adversarial mode is the second adversarial mode, after the second data packet generated based on the second adversarial mode is sent to the server, there may still be burst packet loss phenomenon, at which time the second data packet generated by the improved second adversarial mode can be sent to the server when data transmission is performed in the next frame. Specifically, as shown in FIG. 8, after the plurality of error correction data and the first data packet are combined into the second data packet and sent to the server, in response to the feedback message returned by the server for the second data packet, the plurality of first data packets obtained in the next frame are grouped to obtain a plurality of fourth data pools, each of the fourth data pools contains a plurality of first data packets, and at least two fourth data pools contain at least one same first data packet; the first data packets of the plurality of fourth data pools are re-allocated to obtain a plurality of fifth data pools, each of the fifth data pools contains at least one first data packet from each of the fourth data pools; the plurality of first data packets in each of the fifth data pools are subjected to XOR operation to generate a plurality of optimization error correction data corresponding to the plurality of fifth data pools; and the plurality of optimization error correction data and the first data packet of the next frame are combined into the second data packet of the next frame and sent to the server.

[0143] In some optional embodiments, as shown in FIG. 3, in the case that the number of lost data represented by the packet loss information in the feedback message returned by the server is small or the packet loss rate is low, in order to improve the transmission rate, the disclosure embodiment directly obtains the lost first data packet of the current frame and directly sends the lost first data packet to the server. Specifically, in response to the feedback message returned by the server for the second data packet, the packet loss information of the current frame is determined; the packet loss information of the current frame is judged based on a third threshold value, in the case that the packet loss information of the current frame is greater than the third threshold value, it is considered that the target adversarial mode needs to be started for error correction data generation, based on the target adversarial mode determined by sniffing processing, the corresponding second data packet is generated and sent to the server; in the case that the packet loss information of the current frame is less than or equal to the third threshold value, it is considered that the target adversarial mode does not need to be started, and the lost first data packet is directly obtained based on the feedback message for the second data packet, and the lost first data packet is re-sent to the server.

[0144] The disclosure provides a data processing method, device, equipment, program and readable medium, which are applied to a first terminal device. The method comprises the following steps: obtaining a plurality of first data packets; performing sniffing processing on the plurality of first data packets to determine a target countermeasure mode, wherein the target countermeasure mode comprises a first countermeasure mode or a second countermeasure mode, the first countermeasure mode is used to generate error correction data for random packet loss, and the second countermeasure mode is used to generate error correction data for burst packet loss; performing exclusive or operation on the plurality of first data packets according to the target countermeasure mode to generate a plurality of error correction data; and combining the plurality of error correction data and the plurality of first data packets into a second data packet and sending the second data packet to a server. The disclosure determines different weak network countermeasure modes through sniffing processing, so that when multimedia data is transmitted in a weak network environment, different countermeasure modes are used to perform exclusive or operation on the original data for different packet loss types, so that the server can effectively recover and send the packet loss data to the monitoring terminal according to the error correction data generated by different countermeasure modes, and the effective monitoring and broadcasting of multimedia data in a weak network environment is realized.

[0145] Based on the same inventive concept, the disclosure provides a data processing method applied to a server. FIG. 9 schematically shows a method flowchart for executing the data processing method of the disclosure on the server side. As shown in FIG. 9, the method comprises the following steps:

[0146] S201, receiving a second data packet sent by a first terminal device.

[0147] In the implementation of step S201, the second data packet sent by the first terminal device is received, and the second data packet comprises a plurality of error correction data and a plurality of first data packets. Based on the second data packet, the plurality of error correction data and the plurality of first data packets are extracted.

[0148] S202, processing the second data packet to restore the plurality of first data packets based on the plurality of error correction data to generate a third data packet.

[0149] In the implementation of step S202, since the error correction data contains the characteristics of the plurality of first data packets, it is first determined whether the plurality of first data packets corresponding to the error correction data exist data loss based on the plurality of error correction data. In the case where the first data packets exist data loss, the lost first data packets are restored by reverse exclusive or operation based on the error correction data and the first data packets without loss in the data pool corresponding to each error correction data, and the first data packets transmitted to the server and the first data packets restored based on the error correction data are combined into a third data packet.

[0150] In some optional embodiments, after obtaining the third data packet, since the third data packet has the same encoding format as the first data packet, the third data packet is taken as a first output, and the first output is sent to a developer or an internal client through a real-time communication technology (such as webrtc), and the first output is used for the developer to view the data collected on the first terminal device.

[0151] In some optional embodiments, in order to avoid the data packet sent by the first terminal device being decoded into corresponding multimedia data causing picture freezing to be poor due to a large fluctuation in the frame number of each picture group, the first terminal device generates and sends a preset picture group frame value to the server, and the server receives the preset picture group frame value sent by the first terminal device and performs a smoothing operation on the frame value of each picture group of the multimedia data based on the preset picture group frame value, so that the frame value of each picture group of the multimedia data tends to be stable. Specifically, after accepting the preset picture group frame value, the third data packet is decoded to generate a target data stream, and the target data stream is subjected to a smoothing operation according to the preset picture frame value, so that the frame value of each picture group in the target data stream is equal to the preset picture group frame value, to obtain an optimized data stream. The smoothing operation at least includes frame filtering or frame copying. The frame filtering operation filters out the jittered images in the picture group, so as to reduce the picture group frame value less than the preset picture group frame value to the preset picture group frame value. The frame copying operation copies and inserts only a few images, so as to supplement the picture group frame value less than the preset picture group frame value to the preset picture group frame value. FIG. 12 schematically shows a video frame per second statistical diagram after the smoothing operation in the data processing method of the present disclosure performed on the server side. As shown in FIG. 12, after the server performs the smoothing operation on the target data stream, each frame fluctuates in a small range of 25 frames, so that the change of the picture group frame value is more stable, and the user experience is effectively improved. Optionally, the server receives the preset picture group frame value sent by the first terminal device as 25 frames.

[0152] In some optional embodiments, after obtaining the optimized data stream, the optimized data stream is encoded and encapsulated to generate a fourth data packet corresponding to the optimized data stream, the fourth data packet is a NAL format data encapsulation packet, and the fourth data packet is taken as a second output and sent to a network storage module (such as I / O Storage, CDN Cache) or adjusted to support a http network stream and sent to a second terminal device. Exemplarily, the http network stream includes but is not limited to an hls format network stream, an mpeg-dash format network stream, and the like.

[0153] In some optional embodiments, after obtaining the fourth data packet, the fourth data packet is fragmented to generate a plurality of fifth data packets, and the plurality of fifth data packets are transmitted to the third terminal device as a third output through a Real Time Streaming Protocol (RTSP).

[0154] S203, in the case where the third data packet has data loss, a feedback message is generated and transmitted to the first terminal device.

[0155] In the implementation of S203, although the lost first data packet is recovered based on the error correction data, there can still be a case where part of the lost first data packet cannot be recovered. In this case, based on the information of the first data packet transmitted by the first terminal device represented by the error correction data and the information of the first data packet corresponding to the third data packet, it is determined whether there is still data loss after data recovery of the data transmission result of the current frame, and a corresponding feedback message is generated based on the determination result. The feedback message includes an acknowledgement (ACK) and a negative acknowledgement (NACK). The ACK is a feedback message generated in the case where the third data packet obtained after the server recovers the data contains all the first data packets transmitted by the first terminal device. The ACK is used to represent that there is no loss in the data transmitted in the current frame. The NACK is a feedback message generated in the case where the third data packet obtained after the server recovers the data does not contain all the first data packets transmitted by the first terminal device. The NACK is used to represent that there is loss in the data transmitted in the current frame.

[0156] Based on the same inventive concept, the embodiments of the present disclosure provide a data processing system, the system comprising a first terminal device and a server;

[0157] The first terminal device is configured to obtain a plurality of first data packets, perform sniffing processing on the plurality of first data packets, determine a target countermeasure mode, the target countermeasure mode comprising a first countermeasure mode or a second countermeasure mode, the first countermeasure mode being used to generate error correction data for random packet loss, the second countermeasure mode being used to generate error correction data for burst packet loss, perform exclusive or operation on the plurality of first data packets according to the target countermeasure mode to generate a plurality of error correction data, and combine the plurality of error correction data and the plurality of first data packets into a second data packet and transmit the second data packet to the server.

[0158] The server is configured to receive the second data packet sent by the first terminal device, process the second data packet to restore the first data packets based on the error correction data, and generate a third data packet; and in the case that there is data loss in the third data packet, generate a feedback message and send the feedback message to the first terminal device.

[0159] Based on the same inventive concept, the embodiments of the present disclosure provide a data processing apparatus applied to a first terminal device, the apparatus comprising:

[0160] An original data obtaining module is configured to obtain a plurality of first data packets.

[0161] A mode determining module is configured to sniff the plurality of first data packets and determine a target countermeasure mode, wherein the target countermeasure mode comprises a first countermeasure mode or a second countermeasure mode, the first countermeasure mode is used to generate error correction data for random packet loss, and the second countermeasure mode is used to generate error correction data for burst packet loss.

[0162] An exclusive OR operation module is configured to perform exclusive OR operation on the plurality of first data packets according to the target countermeasure mode to generate a plurality of error correction data.

[0163] A first sending module is configured to combine the plurality of error correction data and the plurality of first data packets into a second data packet and send the second data packet to a server.

[0164] In an optional embodiment, the mode determining module comprises:

[0165] A second sending submodule is configured to sequentially perform data transmission on the first data packets according to the first countermeasure mode to the server and attempt to receive a feedback message returned by the server in an initial frame sequence.

[0166] A first mode determining submodule is configured to determine the target countermeasure mode as the first countermeasure mode in the case that a target frame does not receive the feedback message returned by the server, wherein the target frame is any frame in the initial frame sequence.

[0167] In an optional embodiment, the second sending submodule comprises:

[0168] A first exclusive OR operation unit is configured to perform exclusive OR operation on the plurality of first data packets according to the first countermeasure mode to generate a second data packet of a target frame based on packet loss information of the target frame.

[0169] A first receiving unit is configured to send the second data packet of the target frame to the server and attempt to receive a first feedback message returned by the server, wherein the first feedback message is a feedback message for the target frame.

[0170] In an optional implementation, the mode determining module further comprises:

[0171] a redundancy expansion submodule, configured to expand the redundancy of the first data packet, and perform data transmission to the server according to the first adversarial mode in a next frame of the target frame;

[0172] a second mode determining submodule, configured to, in a case where a second feedback message returned by the server is received, continue to perform the step of expanding the redundancy of the first data packet, and performing data transmission to the server according to the first adversarial mode in a next frame of the target frame until the redundancy of the first data packet is greater than or equal to a first threshold, determine the target adversarial mode as the second adversarial mode, the second feedback message being a feedback message for the next frame of the target frame.

[0173] In an optional implementation, the redundancy expansion submodule comprises:

[0174] a target frame packet loss information determining unit, configured to determine target frame packet loss information in response to a feedback message for the target frame;

[0175] a second exclusive OR operation unit, configured to expand the redundancy of the first data packet, and perform exclusive OR operation on the expanded first data packet according to the first adversarial mode based on the target frame packet loss information, to generate a second data packet of a next frame of the target frame;

[0176] a second receiving unit, configured to send the second data packet of the next frame of the target frame to the server, and attempt to receive the second feedback message.

[0177] In an optional implementation, the mode determining module further comprises:

[0178] a distribution feature submodule, configured to acquire a packet loss distribution feature of a first subframe sequence in response to a feedback message for the target frame, the first subframe sequence being the target frame and all frames before the target frame;

[0179] a third mode determining submodule, configured to determine the target adversarial mode based on the packet loss distribution feature.

[0180] In an optional implementation, the third mode determining submodule comprises:

[0181] a prediction failure rate unit, configured to predict packet loss information of a second subframe sequence based on the packet loss distribution feature, and determine a prediction failure rate, the second subframe sequence being a plurality of subsequent frames of the target frame;

[0182] a second adversarial mode determination unit, configured to determine the target adversarial mode as a second adversarial mode when the prediction failure rate is greater than or equal to a second threshold value;

[0183] a first adversarial mode determination unit, configured to determine the target adversarial mode as a first adversarial mode when the prediction failure rate is less than the second threshold value.

[0184] In an optional implementation, the exclusive OR operation module comprises:

[0185] a first data pool submodule, configured to group the plurality of first data packets to obtain a plurality of first data pools, each of the first data pools comprising a plurality of first data packets;

[0186] an exclusive OR operation submodule, configured to perform exclusive OR operation on the plurality of first data packets in each of the first data pools to generate the plurality of error correction data corresponding to the plurality of first data pools.

[0187] In an optional implementation, the apparatus further comprises:

[0188] a third data pool module, configured to group a plurality of first data packets obtained in a next frame to obtain a plurality of third data pools arranged in sequence in response to the feedback message returned by the server for the second data packet, each of the third data pools comprising a plurality of first data packets, and adjacent third data pools comprising at least one same first data packet;

[0189] a second exclusive OR operation module, configured to perform exclusive OR operation on the plurality of first data packets in each of the third data pools to generate the plurality of optimized error correction data corresponding to the plurality of third data pools;

[0190] a third sending module, configured to combine the plurality of optimized error correction data and the first data packets of the next frame into the second data packet of the next frame and send the second data packet of the next frame to the server.

[0191] In an optional implementation, the exclusive OR operation module comprises:

[0192] a grouping submodule, configured to group the plurality of first data packets to obtain a plurality of first data pools, each of the first data pools comprising a plurality of first data packets;

[0193] a second data pool submodule, configured to re-allocate the first data packets in the plurality of first data pools to obtain a plurality of second data pools, each of the second data pools comprising at least one first data packet from each of the first data pools, and different second data pools comprising different first data packets;

[0194] a third exclusive OR operation sub-module, configured to perform exclusive OR operation on the first data packets in each of the second data pools to generate the error correction data corresponding to the second data pools;

[0195] In an alternative embodiment, the apparatus further comprises:

[0196] a fourth data pool module, configured to group the first data packets obtained in a next frame in response to the feedback message returned by the server to obtain a plurality of fourth data pools, each of the fourth data pools containing a plurality of first data packets, and at least two of the fourth data pools containing at least one same first data packet;

[0197] a fifth data pool module, configured to re-allocate the first data packets in the fourth data pools to obtain a plurality of fifth data pools, each of the fifth data pools containing at least one first data packet from each of the fourth data pools;

[0198] a fourth exclusive OR operation module, configured to perform exclusive OR operation on the first data packets in each of the fifth data pools to generate the optimized error correction data corresponding to the fifth data pools;

[0199] a fourth sending module, configured to combine the optimized error correction data and the first data packets in the next frame into the second data packets in the next frame and send the second data packets in the next frame to the server.

[0200] In an alternative embodiment, the apparatus further comprises:

[0201] a preset frame value module, configured to generate a preset picture group frame value based on the first data packets, the preset picture group frame value being used to represent a rated frame number of each picture group;

[0202] a fifth sending module, configured to send the preset picture group frame value to the server, so that the server performs rectification operation on a target data stream corresponding to the second data packets based on the preset picture group frame value.

[0203] In an alternative embodiment, the original data obtaining module comprises:

[0204] an original data stream obtaining sub-module, configured to call a SurfaceFlinger service to obtain an original data stream, the original data stream at least including data of a plurality of applications in the first terminal device;

[0205] an encoding sub-module, configured to encode the original data stream to generate original encoded data;

[0206] a packaging sub-module, configured to package the original encoded data to obtain the first data packets.

[0207] Based on the same inventive concept, the embodiment of the present disclosure provides a data processing device applied to a server, the device comprising:

[0208] a data packet receiving module configured to receive a second data packet sent by the first terminal device, the second data packet comprising a plurality of error correction data and a plurality of first data packets;

[0209] a restoring module configured to process the second data packet to restore the plurality of first data packets based on the plurality of error correction data, and generate a third data packet;

[0210] a feedback module configured to generate a feedback message and send the feedback message to the first terminal device in the case that there is data loss in the third data packet.

[0211] In an alternative embodiment, the device further comprises:

[0212] a preset frame value receiving module configured to receive a preset picture group frame value sent by the first terminal device;

[0213] a decoding module configured to decode the third data packet and generate a target data stream;

[0214] a rectifying module configured to perform a rectifying operation on the target data stream according to the preset picture group frame value, so that each picture group frame value in the target data stream is equal to the preset picture group frame value, and obtain an optimized data stream, wherein the rectifying operation at least comprises frame filtering or frame copying;

[0215] a fourth data packet module configured to encode the optimized data stream and obtain a fourth data packet.

[0216] The device embodiments described above are only schematic, wherein the units described as separate components may or may not be physically separate, and the components displayed as units may or may not be physical units, i.e., may be located in one place, or may be distributed on a plurality of network units. Part or all of the modules can be selected according to actual needs to achieve the purpose of the present embodiment. Those skilled in the art can understand and implement without creative labor.

[0217] Various component embodiments of the present disclosure can be implemented in hardware, or as software modules running in one or more processors, or combinations thereof. Those skilled in the art will appreciate that some or all of the functions of some or all of the components of the computing processing device according to embodiments of the present disclosure can be implemented using a microprocessor or a digital signal processor (DSP) in practice. The present disclosure can also be implemented as a device or apparatus program (e.g., a computer program and a computer program product) for performing part or all of the methods described herein. Such a program implementing the present disclosure can be stored on a computer readable medium or can have one or more signals. Such signals can be downloaded from an Internet website, or provided on a carrier medium, or in any other form.

[0218] For example, Fig. 13 schematically shows a block diagram of a computing processing device for performing a method according to the present disclosure. The computing processing device traditionally comprises a processor 1010 and a computer program product or a computer readable medium in the form of a memory 1020. The memory 1020 can be an electronic memory such as a flash memory, an EEPROM (electrically erasable programmable read-only memory), an EPROM, a hard disk, or a ROM. The memory 1020 has a storage space 1030 for program code 1031 for performing any of the method steps in the methods described above. For example, the storage space 1030 for program code can comprise individual program codes 1031 for implementing the various steps in the above methods, respectively. These program codes can be read from or written to one or more computer program products. These computer program products include program code carriers such as hard disks, compact disks (CDs), memory cards, or floppy disks. Such computer program products are often portable or stationary memory units as described with reference to Fig. 14. The memory unit can have a storage section, a storage space, etc., arranged similarly to the memory 1020 in the computing processing device of Fig. 13. The program code can be compressed, for example, in a suitable form. Typically, the memory unit comprises computer readable code 1031', i.e., code that can be read by a processor such as 1010, which, when run by the computing processing device, causes the computing processing device to perform the various steps in the methods described above.

[0219] Reference herein to "one embodiment", "an embodiment" or "one or more embodiments" means that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment of the disclosure. The appearances of the phrase "in one embodiment" in various places in this specification are not necessarily all referring to the same embodiment.

[0220] In the description provided herein, numerous specific details are set forth. However, it is understood that embodiments of the disclosure can be practiced without these specific details. In some instances, well-known methods, structures and techniques have not been shown in detail in order not to obscure an understanding of this description.

[0221] In the claims, any reference signs placed between parentheses shall not be construed as limiting the claim. The word 'comprising' does not exclude the presence of elements or steps other than those listed in a claim. The word 'a' or 'an' preceding an element does not exclude the presence of a plurality of such elements. The disclosure can be implemented by means of both hardware and software, and any combination thereof. In a unitary claim, several devices, apparatuses or means can be listed, comprising means for carrying out a certain task. The use of the term'means' in a claim is intended to refer to a combination of means for performing a task, even if such means are not explicitly recited in the claim. The word 'first','second', 'third', etc. do not imply any order. The terms 'first','second', 'third', etc. are to be interpreted according to the context in which they are used.

[0222] It has to be noted that the above-mentioned embodiments illustrate rather than limit the application, since various modifications are possible within the scope of the appended claims. As such, the particular embodiments provided are meant to be illustrative only and not meant to be limiting as to the scope of the disclosure.

Claims

1. A data processing method, characterized by, Applied to a first terminal device, the method comprises: Obtaining a plurality of first data packets; Sniffing the plurality of first data packets to determine a target countermeasure mode, the target countermeasure mode comprising a first countermeasure mode or a second countermeasure mode, the first countermeasure mode being used to generate error correction data for random packet loss, and the second countermeasure mode being used to generate error correction data for burst packet loss; Performing an exclusive OR operation on the plurality of first data packets according to the target countermeasure mode to generate a plurality of error correction data; Combining the plurality of error correction data and the plurality of first data packets into second data packets and sending the second data packets to a server.

2. The data processing method according to claim 1, characterized in that, The step of sniffing the plurality of first data packets to determine a target countermeasure mode comprises: In an initial frame sequence, sequentially performing data transmission on the first data packets to the server according to the first countermeasure mode, and attempting to receive feedback messages returned by the server; In a case where a target frame does not receive the feedback messages returned by the server, determining the target countermeasure mode as the first countermeasure mode, the target frame being any frame in the initial frame sequence.

3. The data processing method according to claim 2, characterized in that, The step of performing data transmission on the first data packets to the server according to the first countermeasure mode, and attempting to receive feedback messages returned by the server comprises: Based on packet loss information of a target frame, performing an exclusive OR operation on the plurality of first data packets according to the first countermeasure mode to generate second data packets of the target frame; Sending the second data packets of the target frame to the server, and attempting to receive first feedback messages returned by the server, the first feedback messages being feedback messages for the target frame.

4. The data processing method according to claim 3, characterized in that, In a case where the target frame is a first frame of the initial frame sequence, the packet loss information of the target frame is initial packet loss information; In a case where the target frame is a non-first frame of the initial frame sequence, the packet loss information of the target frame is packet loss information determined based on the first feedback messages.

5. The data processing method according to claim 2, characterized in that, In a case where the target frame receives the feedback messages returned by the server, the method further comprises: Expanding a redundancy of the first data packets, and performing data transmission on a next frame of the target frame to the server according to the first countermeasure mode; In a case where second feedback messages returned by the server are received, continuing to perform the step of expanding the redundancy of the first data packets, and performing data transmission on the next frame of the target frame to the server according to the first countermeasure mode until the redundancy of the first data packets is greater than or equal to a first threshold value, and determining the target countermeasure mode as the second countermeasure mode, the second feedback messages being feedback messages for the next frame of the target frame.

6. The data processing method according to claim 5, characterized in that, The step of expanding the redundancy of the first data packets, and performing data transmission on the next frame of the target frame to the server according to the first countermeasure mode comprises: Determining packet loss information of a target frame in response to feedback messages for the target frame; Expanding the redundancy of the first data packets, and performing an exclusive OR operation on the expanded first data packets according to the first countermeasure mode based on the packet loss information of the target frame to generate second data packets of the next frame of the target frame; sending a second data packet of a next frame of the target frame to the server and attempting to receive a second feedback message.

7. The data processing method according to claim 2, characterized in that, In a case where the target frame receives a feedback message returned by the server, the method further includes: in response to the feedback message for the target frame, obtaining a packet loss distribution feature of a first sub-frame sequence, the first sub-frame sequence being the target frame and all frames before the target frame; determining the target adversarial mode based on the packet loss distribution feature.

8. The data processing method according to claim 7, characterized in that, The determining the target adversarial mode based on the packet loss distribution feature includes: predicting packet loss information of a second sub-frame sequence based on the packet loss distribution feature, determining a prediction failure rate, the second sub-frame sequence being a plurality of subsequent frames of the target frame; in a case where the prediction failure rate is greater than or equal to a second threshold value, determining the target adversarial mode as the second adversarial mode; in a case where the prediction failure rate is less than the second threshold value, determining the target adversarial mode as the first adversarial mode.

9. The data processing method of claim 1, wherein, The target adversarial mode is the first adversarial mode, and the performing exclusive or operation on the plurality of first data packets according to the target adversarial mode to generate a plurality of error correction data includes: grouping the plurality of first data packets to obtain a plurality of first data pools, each of the first data pools containing a plurality of first data packets; performing exclusive or operation on the plurality of first data packets in each first data pool to generate the plurality of error correction data corresponding to the plurality of first data pools one by one.

10. The data processing method according to claim 9, characterized in that, After the plurality of error correction data and the first data packets are combined into a second data packet and sent to the server, the method further includes: in response to a feedback message returned by the server for the second data packet, grouping a plurality of first data packets obtained in a next frame to obtain a plurality of third data pools arranged in sequence, each of the third data pools containing a plurality of first data packets, and adjacent third data pools containing at least one same first data packet; performing exclusive or operation on the plurality of first data packets in each third data pool to generate a plurality of optimized error correction data corresponding to the plurality of third data pools one by one; combining the plurality of optimized error correction data and the first data packets of the next frame into a second data packet of the next frame and sending the second data packet to the server.

11. The data processing method of claim 1, wherein, The target adversarial mode is the second adversarial mode, and the performing exclusive or operation on the plurality of first data packets according to the target adversarial mode to generate a plurality of error correction data includes: grouping the plurality of first data packets to obtain a plurality of first data pools, each of the first data pools containing a plurality of first data packets; re-distributing the first data packets of the plurality of first data pools to obtain a plurality of second data pools, each of the second data pools containing at least one first data packet from each of the first data pools, and different second data pools containing different first data packets; performing exclusive or operation on the plurality of first data packets in each second data pool to generate the plurality of error correction data corresponding to the plurality of second data pools one by one.

12. The data processing method according to claim 11, characterized in that, After the plurality of error correction data and the first data packets are combined into a second data packet and sent to the server, the method further includes: In response to the feedback message returned by the server for the second data packet, a plurality of first data packets obtained in a next frame are grouped to obtain a plurality of fourth data pools, each of the fourth data pools containing a plurality of first data packets, and at least two fourth data pools containing at least one same first data packet; The first data packets in the plurality of fourth data pools are re-distributed to obtain a plurality of fifth data pools, each of the fifth data pools containing at least one first data packet from each of the fourth data pools; The plurality of first data packets in each of the fifth data pools are subjected to an exclusive OR operation to generate a plurality of optimized error correction data corresponding to the plurality of fifth data pools; The plurality of optimized error correction data and the first data packets of the next frame are combined into the second data packet of the next frame and sent to the server.

13. The data processing method of claim 1, wherein, After obtaining the plurality of first data packets, the method further comprises: Based on the first data packets, a preset picture group frame value is generated, the preset picture frame value being used to represent a rated frame number of each picture group; The preset picture frame value is sent to the server, so that the server performs a rectification operation on a target data stream corresponding to the second data packet based on the preset picture frame value.

14. The data processing method according to any one of claims 1 to 13, characterized by, The first terminal device is an Android terminal device, and the obtaining of the plurality of first data packets comprises: A SurfaceFlinger service is called to obtain an original data stream, the original data stream at least including data of a plurality of applications in the first terminal device; The original data stream is encoded to generate original encoded data; The original encoded data is data-encapsulated to obtain the plurality of first data packets.

15. A data processing method, characterized by, Applied to a server, the method comprises: Receiving a second data packet sent by a first terminal device, the second data packet including a plurality of error correction data and a plurality of first data packets; Processing the second data packet to restore the plurality of first data packets based on the plurality of error correction data to generate a third data packet; In the case that there is data loss in the third data packet, a feedback message is generated and sent to the first terminal device.

16. The data processing method according to claim 15, characterized in that, After generating the third data packet, the method further comprises: Receiving a preset picture group frame value sent by the first terminal device; Decoding processing the third data packet to generate a target data stream; Performing a rectification operation on the target data stream according to the preset picture frame value, so that each picture group frame value in the target data stream is equal to the preset picture group frame value to obtain an optimized data stream, wherein the rectification operation at least includes frame filtering or frame copying; Encoding processing the optimized data stream to obtain a fourth data packet.

17. A data processing system, characterized by The system comprises a first terminal device and a server; The first terminal device is configured to obtain a plurality of first data packets, perform sniffing processing on the plurality of first data packets, and determine a target countermeasure mode, the target countermeasure mode including a first countermeasure mode or a second countermeasure mode, the first countermeasure mode being used to generate error correction data for random packet loss, and the second countermeasure mode being used to generate error correction data for burst packet loss; The plurality of first data packets are subjected to an exclusive OR operation according to the target countermeasure mode to generate a plurality of error correction data; Combine the plurality of error correction data and the plurality of first data packets into a second data packet and send to a server; The server is configured to receive the second data packet sent by the first terminal device; Process the second data packet to restore the plurality of first data packets based on the plurality of error correction data and generate a third data packet; In the case of data loss in the third data packet, generate a feedback message and send to the first terminal device.

18. A data processing apparatus, characterized by The apparatus applied to a first terminal device comprises: An original data acquisition module configured to acquire a plurality of first data packets; A mode determination module configured to sniff the plurality of first data packets and determine a target countermeasure mode, the target countermeasure mode comprising a first countermeasure mode or a second countermeasure mode, the first countermeasure mode being configured to generate error correction data for random packet loss, and the second countermeasure mode being configured to generate error correction data for burst packet loss; An exclusive or operation module configured to perform exclusive or operation on the plurality of first data packets according to the target countermeasure mode and generate a plurality of error correction data; A first sending module configured to combine the plurality of error correction data and the plurality of first data packets into a second data packet and send to a server.

19. A data processing apparatus, characterized by The apparatus applied to a server comprises: A data packet receiving module configured to receive a second data packet sent by the first terminal device, the second data packet comprising a plurality of error correction data and a plurality of first data packets; A restoration module configured to process the second data packet to restore the plurality of first data packets based on the plurality of error correction data and generate a third data packet; A feedback module configured to, in the case of data loss in the third data packet, generate a feedback message and send to the first terminal device.

20. A computing processing device, comprising: comprise: a memory having computer readable code stored therein; and one or more processors, when the computer readable code is executed by the one or more processors, the computing processing device performs the data processing method as claimed in any one of claims 1-14, or the computing processing device performs the data processing method as claimed in claim 15 or 16.

21. A computer program, characterized in that, comprise computer readable code which, when run on a computing processing device, causes the computing processing device to perform the data processing method as claimed in any one of claims 1-14, or causes the computing processing device to perform the data processing method as claimed in claim 15 or 16.

22. A computer readable medium characterized by a computer program as claimed in claim 21 is stored.

Images