Video data processing method and device, storage medium and electronic equipment

Abstract

The invention discloses a video data processing method and device, a storage medium and electronic equipment. The method comprises the following steps: in response to a video playing request from a target client, determining to-be-played video data and a first frame playing condition, determining first frame video data and a target sending sequence according to the to-be-played video data and the first frame playing condition, packaging the first frame video data into a first group of data packets according to the target sending sequence, and sending the first group of data packets to the target client according to the target sending sequence, and sending the non-first-frame video data to the target client under the condition that the first group of data packets are confirmed to be sent to the target client and are confirmed to be received by the target client. According to the method and the device, the technical problem of relatively low processing efficiency of the video data caused by poor optimization of the first frame data of the video data and overlong processing time is solved. The embodiment of the invention can be applied to various scenes such as cloud technology and artificial intelligence.

Description

Technical Field

[0001] The present application relates to the field of computers, and in particular to a method and device for processing video data, a storage medium, and an electronic device. Background Art

[0002] In the related art, the technical solution for processing video data mainly improves the sending rate of video data by setting initial sending parameters. In other words, during the transmission of video data, the same initial sending parameters are set to improve the video data transmission rate and shorten the time for video data processing. However, due to the large differences in the software and hardware equipment levels of the recipients of the video data, the method of setting the initial sending parameters in the related art is not universal. This technical solution cannot be applied to some video data processing scenarios, resulting in low video data processing efficiency.

[0003] To address the above-mentioned problems, no effective solution has been proposed yet. Summary of the invention

[0004] The embodiments of the present application provide a method and device for processing video data, a storage medium, and an electronic device to at least solve the technical problem of low processing efficiency of video data due to poor optimization of the first frame data of the video data and excessive processing time.

[0005] According to one aspect of an embodiment of the present application, a method for processing video data is provided, comprising: in response to a video playback request from a target client, determining video data to be played and a first frame playback condition, wherein the first frame playback condition includes the number of audio frames and the number of video frames required for the video data to be played to complete the first frame playback on the target client; determining first frame video data and a target sending order according to the video data to be played and the first frame playback condition, wherein the first frame video data includes target audio frames corresponding to the number of audio frames in the video data to be played and target video frames corresponding to the number of video frames in the video data to be played, and the target sending order indicates the target audio frames. The target video frame and the target video frame are sent in a sending order; the first frame video data is encapsulated into a first group of data packets according to the target sending order, and the first group of data packets are sent to the target client, wherein, in the case that the first group of data packets contain a second group of data packets that have been confirmed to be sent to the target client and have not yet been confirmed to be received by the target client, the second group of data packets are sent in reverse order of the target sending order; in the case that the first group of data packets have all been confirmed to be sent to the target client and have been confirmed to be received by the target client, the non-first frame video data are sent to the target client, wherein the video data to be played includes the first frame video data and the non-first frame video data.

[0006] According to another aspect of an embodiment of the present application, a video data processing device is also provided, including: a first determination module, used to determine the video data to be played and the first frame playback condition in response to a video playback request from a target client, wherein the first frame playback condition includes the number of audio frames and the number of video frames required for the video data to be played to complete the first frame playback on the target client; a second determination module, used to determine the first frame video data and a target sending order according to the video data to be played and the first frame playback condition, wherein the first frame video data includes the target audio frames in the video data to be played corresponding to the number of audio frames and the target video frames in the video data to be played corresponding to the number of video frames, and the target sending order indicates the target audio The target video frame and the target video frame are sent in a sending order; a first transmission module is used to encapsulate the first frame video data into a first group of data packets according to the target sending order, and send the first group of data packets to the target client, wherein, in the case that there is a second group of data packets in the first group of data packets that have been confirmed to be sent to the target client and have not been confirmed to be received by the target client, the second group of data packets are sent in the reverse order of the target sending order; a second transmission module is used to send non-first frame video data to the target client when the first group of data packets have been confirmed to be sent to the target client and have been confirmed to be received by the target client, wherein the video data to be played includes the first frame video data and the non-first frame video data.

[0007] Optionally, the device is used to respond to a video playback request from a target client to determine the video data to be played and the first frame playback condition in the following manner: in response to the video playback request, obtain the first frame data information of the target client, wherein the first frame data information includes a first moment and a second moment corresponding to the played video data cached by the target client, the first moment indicating the moment when the target client first receives the data packet corresponding to the played video data, and the second moment indicating the moment when the target client starts decoding and rendering the played video data; determine the first frame playback condition based on the first frame data information, wherein the first frame data information includes the number of audio frames and the number of video frames.

[0008] Optionally, the device is used to determine the first frame video data and the target sending order according to the video data to be played and the first frame playback condition in the following manner: performing a parsing operation on the video data to be played to determine the initial audio frame and the initial video frame in the video data to be played, wherein the initial audio frame and the initial video frame are identified and determined by a specific type field; extracting the target audio frame from the initial audio frame according to the number of audio frames and the number of video frames, and extracting the target video frame from the initial video frame, generating the first frame video data, and setting the first frame video data to be sent according to the target sending order.

[0009] Optionally, the device is used to extract the target audio frame from the initial audio frame according to the number of audio frames and the number of video frames, and extract the target video frame from the initial video frame, generate the first frame video data, and set the first frame video data to be sent according to the target sending order in the following manner: when the first frame playback condition includes the first frame cache duration of the played video data, extract the first data frame from the video data to be played according to the first frame cache duration, wherein the first data frame includes the first video frame or the first audio frame, the first video frame represents a video frame that meets the first frame cache duration, the first audio frame represents an audio frame that meets the first frame cache duration, and the first frame cache duration represents the first frame video duration of the played video data; determine the target audio frame, the target video frame and the first data frame as the first frame video data, and set the target audio frame and the target video frame to be sent according to the target sending order, and set the first data frame to be sent according to a first sending order, wherein the first sending order is located after the target sending order.

[0010] Optionally, the device is used to extract the target audio frame from the initial audio frame according to the number of audio frames and the number of video frames, and extract the target video frame from the initial video frame to generate the first frame video data and the target sending order in the following manner: when the first frame playback condition includes the first frame cache data size of the played video data, extract the second data frame from the video data to be played according to the first frame cache data size, wherein the second data frame includes a second video frame or a second audio frame, the second video frame represents a video frame that meets the first frame cache data size, the second audio frame represents an audio frame that meets the first frame cache data size, and the first frame cache data size represents the first frame video data size of the played video data; determine the target audio frame, the target video frame and the second data frame as the first frame video data, and set the target audio frame and the target video frame to be sent in the target sending order, and set the second data frame to be sent in a second sending order, wherein the second sending order is located after the target sending order.

[0011] Optionally, the device is used to extract the target audio frame from the initial audio frame according to the number of audio frames and the number of video frames, and extract the target video frame from the initial video frame to generate the first frame video data and the target sending order in the following manner: when the first frame playback condition includes the total number of first frame cache frames of the played video data, extract a third data frame from the video data to be played according to the total number of first frame cache frames, wherein the third data frame includes a third video frame or a third audio frame, the third video frame represents a video frame that meets the total number of first frame cache frames, the third audio frame represents an audio frame that meets the total number of first frame cache frames, and the total number of first frame cache frames represents the number of frames of the first frame of the played video data; determine the target audio frame, the target video frame and the third data frame as the first frame video data, and set the target audio frame and the target video frame to be sent in the target sending order, and set the third data frame to be sent in a third sending order, wherein the third sending order is located after the target sending order.

[0012] Optionally, the device is used to encapsulate the first frame video data into a first group of data packets according to the target sending order, and send the first group of data packets to the target client in the following manner: encapsulate the first frame video data into multiple data packets; set a corresponding data packet sequence number for each of the multiple data packets according to the target sending order, wherein the value of the data packet sequence number is used to indicate the sending order of the corresponding data packets; and send the multiple data packets to the target client in sequence according to the value of the data packet sequence number.

[0013] Optionally, the device is also used to: obtain a first data packet sequence number, wherein the first data packet sequence number is used to identify a data packet that has been sent to the target client; receive a second data packet sequence number returned by the target client, wherein the second data packet sequence number is used to identify a data packet confirmed to be received by the target client; determine the second group of data packets based on the first data packet sequence number and the second data packet sequence number, and send the second group of data packets in reverse order of the target sending order.

[0014] Optionally, the device is used to determine the second group of data packets based on the first data packet sequence number and the second data packet sequence number in the following manner, and send the second group of data packets in reverse order of the target sending order: determine the number of data packets currently confirmed to be received by the target client based on the second data packet sequence number; extract retransmitted data packets corresponding to the number of data packets from the second group of data packets in reverse order of the target sending order, send the retransmitted data packets to the target client, and delete the retransmitted data packets from the second group of data packets.

[0015] Optionally, the device is also used to: respond to a video playback request from a target client, determine the video data to be played, and initialize a target variable for the video data to be played, wherein the target variable is used to indicate whether the first frame of video data has been sent; obtain a third data packet sequence number, wherein the third data packet sequence number is used to identify a data packet that needs to be sent to the target client; when the second data packet sequence number and the third data packet sequence number are different, determine that the first frame of video data has not been sent, send the first group of data packets to the target client in the target sending order, and send the second group of data packets in reverse order of the target sending order; when the second data packet sequence number and the third data packet sequence number are the same, determine that the first frame of video data has been sent, stop sending the second group of data packets, and send non-first frame video data to the target client.

[0016] Optionally, the device is used to respond to a video playback request from a target client and determine the video data to be played and the first frame playback condition in the following manner: detect the video playback request and determine the video data to be played and the first frame playback condition field; when the first frame playback condition field indicates that the video playback request includes the first frame playback condition, set the first frame playback condition according to the video playback request; when the first frame playback condition field indicates that the video playback request does not include the first frame playback condition, obtain the pre-set first frame playback condition.

[0017] According to another aspect of the embodiments of the present application, a computer-readable storage medium is provided, in which a computer program is stored, wherein the computer program is configured to execute the above-mentioned video data processing method when running.

[0018] According to another aspect of the embodiments of the present application, a computer program product or a computer program is provided, the computer program product or the computer program includes computer instructions, the computer instructions are stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the above video data processing method.

[0019] According to another aspect of the embodiments of the present application, there is further provided an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to execute the video data processing method through the computer program.

[0020] In an embodiment of the present application, in response to a video playback request from a target client, the video data to be played and the first frame playback condition are determined, wherein the first frame playback condition includes the number of audio frames and the number of video frames required for the video data to be played to complete the first frame playback on the target client; the first frame video data and the target sending order are determined according to the video data to be played and the first frame playback condition, wherein the first frame video data includes target audio frames corresponding to the number of audio frames in the video data to be played and target video frames corresponding to the number of video frames in the video data to be played, and the target sending order represents the sending order of the target audio frames and the target video frames; the first frame video data is encapsulated into a first group of data packets according to the target sending order, and the first group of data packets are sent to the target client, wherein, in the first group of data packets, there is a second group of data packets that have been confirmed to be sent to the target client and have not yet been confirmed to be received by the target client , send the second group of data packets in the reverse order of the target sending order; when the first group of data packets have been confirmed to be sent to the target client and confirmed to be received by the target client, send the non-first frame video data to the target client, wherein the video data to be played includes the first frame video data and the non-first frame video data. By obtaining the video data to be played and the first frame playback condition, the ability of the client's first frame playback condition is determined according to the first frame playback condition, and then the sending order of the audio and video frames in the first frame video data is adjusted according to the first frame data of the video data to be played and the first frame playback condition, so as to achieve the purpose of optimizing the delay of the first frame video data, thereby realizing the technical effect of improving the viewing experience of the object, optimizing network transmission, and ensuring the continuity and smoothness of video playback, thereby solving the technical problem of low processing efficiency of video data due to poor optimization of the first frame data of video data and too long processing time.

[0021] In addition, the data packets of the first frame of video data are sent in the target sending order. When the video sender fails to promptly receive the message of successful data packet reception sent by the video receiver or some data packets are lost, the data packets are actively sent to the video receiver in the reverse order of the target sending order of the first frame of video data, thereby improving the reliability of video playback, reducing video transmission delay, and achieving the technical effect of improving network utilization and increasing the stability of data transmission. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0023] Figure 1 is a schematic diagram of an application environment of an optional video data processing method according to an embodiment of the present application;

[0024] Figure 2 is a flowchart of an optional method for processing video data according to an embodiment of the present application;

[0025] Figure 3 is a schematic diagram of an optional method for processing video data according to an embodiment of the present application;

[0026] Figure 4 is a schematic diagram of another optional method for processing video data according to an embodiment of the present application;

[0027] Figure 5 is a schematic diagram of another optional method for processing video data according to an embodiment of the present application;

[0028] Figure 6 is a schematic diagram of another optional method for processing video data according to an embodiment of the present application;

[0029] Figure 7 is a schematic diagram of another optional method for processing video data according to an embodiment of the present application;

[0030] Figure 8 is a schematic diagram of another optional method for processing video data according to an embodiment of the present application;

[0031] Fig. 9 is a schematic diagram of another optional method for processing video data according to an embodiment of the present application;

[0032] Fig.10 is a schematic structural diagram of an optional video data processing device according to an embodiment of the present application;

[0033] Fig.11 is a schematic structural diagram of an optional video data processing product according to an embodiment of the present application;

[0034] Fig.12 It is a schematic diagram of the structure of an optional electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0035] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.

[0036] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0037] First, some nouns or terms that appear in the description of the embodiments of the present application are subject to the following interpretation:

[0038] TCP: Transport Control Protocol, the basic protocol for Internet communications, divides data into small data packets for transmission on the Internet.

[0039] QUIC: Quick UDP Internet Connection, a new type of Internet connection protocol based on the UDP protocol, aims to provide faster connection speeds and lower latency, without sacrificing security. It avoids delays such as handshakes and slow start in the TCP protocol by implementing connection establishment, encryption, and data transmission functions at the transport layer. QUIC also uses multiplexing technology to transmit multiple data streams at the same time, improving network utilization. At the same time, it also has functions such as congestion control, error recovery, and flow control to ensure the reliability and stability of data transmission.

[0040] UDP protocol: User Datagram Protocol, object datagram protocol, a connectionless, unreliable data transmission protocol, which transmits data in the form of data packets, but does not guarantee the order and reliability of data packets. It is suitable for applications with high real-time requirements, such as streaming media and voice communications.

[0041] QoE: Quality of Experience, the degree of satisfaction and enjoyment that the subject feels when using a product or service. It includes the subject's evaluation of the product or service's function, performance, ease of use, reliability, security, and effect. The quality of experience directly affects the subject's satisfaction and loyalty to the product or service, and also has an important impact on the company's reputation and competitiveness.

[0042] QoS: Quality of Services, the characteristics and capabilities of a product or service to meet customer needs. Service quality can be evaluated from many aspects, such as reliability, timeliness, positivity, credibility, empathy, customer satisfaction, etc.

[0043] RTT: Round-Trip Time, the time required from sending a request to receiving a response.

[0044] 0-RTT strategy: A network optimization technology that aims to reduce network latency and improve object experience. The core idea of ​​the 0-RTT strategy is to reduce the round-trip latency of subsequent requests by sending data in advance or performing authentication when establishing the initial connection. Normally, a three-way handshake is required to establish a connection, while the 0-RTT strategy can start transmitting data at the first request, thus saving the time of a handshake. The 0-RTT strategy can be applied to various network communication protocols and application scenarios, such as web browsing, video streaming, online games, etc.

[0045] IP: Internet Protocol, a series of rules and standards used for data transmission and communication on the Internet. Internet Protocol consists of multiple protocols, including but not limited to: TCP / IP protocol, UDP protocol (User Datagram Protocol), HTTP protocol (Hypertext Transfer Protocol), FTP protocol (File Transfer Protocol), SMTP protocol (Simple Mail Transfer Protocol), POP3 protocol (Post Office Protocol version 3), IMAP protocol (Internet Message Access Protocol), Telnet protocol (Telnet Protocol), SSH protocol (Secure Shell Protocol), etc.

[0046] DNS: Domain Name System, a domain name resolution protocol used to convert domain names into IP addresses to achieve domain name addressing on the Internet. It is used to convert domain names (such as example.com) into IP addresses (such as 192.0.2.1) or other network service records (such as mail server records, text records, etc.).

[0047] The present application is described below in conjunction with embodiments:

[0048] According to one aspect of an embodiment of the present application, a method for processing video data is provided. Optionally, in this embodiment, the method for processing video data can be applied to Figure 1 In the hardware environment composed of the server 101 and the terminal device 103 shown in FIG. Figure 1As shown, the server 101 is connected to the terminal device 103 via a network, and can be used to provide services for the terminal device or an application installed on the terminal device. The application can be a video application, an instant messaging application, a browser application, an educational application, a game application, etc. A database 105 may be set up on the server or independently of the server to provide data storage services for the server 101, for example, a game data storage server. The above-mentioned network may include, but is not limited to, a wired network and a wireless network, wherein the wired network includes, a local area network, a metropolitan area network and a wide area network; the wireless network includes, Bluetooth, WIFI and other networks that implement wireless communication; the terminal device 103 may be a terminal configured with an application, and may include, but is not limited to, at least one of the following: a mobile phone (such as an Android phone, an iOS phone, etc.), a laptop computer, a tablet computer, a PDA, a MID (Mobile Internet Devices), a PAD, a desktop computer, a smart TV, an intelligent voice interaction device, a smart home appliance, a vehicle-mounted terminal, an aircraft, a virtual reality (VR) terminal, an augmented reality (AR) terminal, a mixed reality (MR) terminal and other computer devices; the above-mentioned server may be a single server, a server cluster consisting of multiple servers, or a cloud server.

[0049] Combination Figure 1 As shown, the above-mentioned video data processing method can be implemented in the terminal device 103, and can also be implemented by a server, for example, Figure 1 It is implemented in the server 101 shown; or it is implemented by the terminal device and the server together.

[0050] The above is only an example and is not specifically limited in this embodiment.

[0051] Optionally, as an optional implementation, as Figure 2 As shown, the method for processing the above video data includes:

[0052] S202, in response to a video playback request from a target client, determining the video data to be played and a first frame playback condition, wherein the first frame playback condition includes the number of audio frames and the number of video frames required for the video data to be played to complete the first frame playback on the target client;

[0053] Optionally, in this embodiment, the target client may include but is not limited to smart phones, tablet computers, laptop computers, desktop computers, smart speakers, smart watches, etc. The target client issues a video playback request, which can be understood as the target client requesting to obtain the video data to be played, wherein the video data to be played may include but is not limited to obtaining from an online media platform server, obtaining from a live broadcast platform, obtaining from an online education platform, obtaining from social media, obtaining from a game, obtaining from a smart home, etc. Correspondingly, the video to be played may include but is not limited to real-time video of a video conference, historical recorded video of a video conference, real-time video of a live broadcast, historical recorded video of a live broadcast, on-demand video, voice call video, voice recognition video, audio playback video, etc.

[0054] Exemplarily, the online media platform server can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The terminal can be a smart phone, tablet computer, laptop computer, desktop computer, smart speaker, smart watch, etc. The terminal and the server can be directly or indirectly connected by wired or wireless communication, which is not limited in this application.

[0055] It should be noted that in this application, cloud technology refers to a hosting technology that unifies a series of resources such as hardware, software, and network in a wide area network or a local area network to realize the calculation, storage, processing, and sharing of data. Cloud technology is a general term for network technology, information technology, integration technology, management platform technology, application technology, etc. based on the cloud computing business model application, which can form a resource pool and be used on demand, flexible and convenient. Cloud computing technology will become an important support. The background services of the technical network system require a large amount of computing and storage resources, such as video websites, picture websites, and more portal websites. With the high development and application of the Internet industry, each item may have its own identification mark in the future, and all need to be transmitted to the background system for logical processing. Data of different levels will be processed separately, and all kinds of industry data require strong system backing support, which can only be achieved through cloud computing.

[0056] Specifically, the first frame playback condition is obtained according to the video playback request of the above-mentioned target client, wherein the first frame playback request can be understood as the playback condition that needs to be met to play the first frame of video data of the above-mentioned video to be played, including but not limited to the number of audio frames required for the first frame video playback, the number of video frames required for the first frame video playback, the cache data duration of the historical playback video data, the cache data size in the historical playback video data, the total number of cache frames of the historical playback video data, etc.

[0057] In an exemplary embodiment, the present application does not specifically limit the above-mentioned number of audio frames and the number of video frames. In other words, the number of audio frames required to complete the playback of the first frame of the video data to be played on the target client may be 1 or more, and the number of video frames required to complete the playback of the first frame of the video data to be played on the target client may be 1 or more. For example, no video frame is required to complete the playback of the first frame of the video data to be played on the target client, that is, only audio frames are required to complete the playback of the first frame of the video data to be played on the target client, and only the number of audio frames exists in the above-mentioned first frame playback condition. For another example, no audio frame is required to complete the playback of the first frame of the video data to be played on the target client, that is, only video frames are required to complete the playback of the first frame of the video data to be played on the target client, and only the number of video frames exists in the above-mentioned first frame playback condition.

[0058] Exemplarily, the above-mentioned audio frame, that is, a continuous time segment in a piece of audio data, can be divided into fixed time intervals but is not limited to it. The audio frame may include but is not limited to the sampling rate, bit depth and number of channels of the audio data. The type of audio frame may include but is not limited to mono audio frame, dual-channel audio frame, multi-channel audio frame, non-interlaced audio frame, interlaced audio frame, etc., wherein a mono audio frame means that each audio frame contains sampling values ​​of only one channel; a dual-channel audio frame means that each audio frame contains sampling values ​​of two channels, which respectively represent the audio signals of the left and right channels; each audio frame in the multi-channel audio frame contains sampling values ​​of multiple channels, which can represent audio signals of multiple channels; the sampling values ​​in the non-interlaced audio frame are arranged in channel order, that is, the sampling values ​​of the first channel are first sampled, and then the sampling values ​​of the second channel, and the audio frames are obtained by sampling in sequence; the interlaced audio frame means that the sampling values ​​of the first time point are first sampled, and then the sampling values ​​of the second time point are sampled in sequence to obtain the audio frame.

[0059] Furthermore, the above-mentioned video frame, i.e., a continuous time segment in a video data, may be divided into fixed time intervals, but not limited to, and the video frame may include but not limited to pixels, frame rate, compression format, audio and metadata, etc. The frame rate can be understood as the number of frames contained in the video data per second, and the metadata may include but not limited to shooting device information, image resolution, etc. This information can be used to decode and play the video. The types of video frames may include but not limited to key frames, predicted frames, bi-directional frames, and discarded frames, etc., wherein a key frame is a complete frame in the encoded video sequence, which does not rely on other frames for decoding, represents the beginning or key scene of the video sequence, and contains complete image information; a predicted frame is a frame in the encoded video sequence, which is predicted and encoded by referring to the previous key frame or predicted frame, and may include the difference information between the current frame and the reference frame; a bi-directional predicted frame is predicted and encoded by referring to the previous key frame or predicted frame, and may include the difference information between the current frame and the previous and next reference frames; a discarded frame is not encoded and transmitted, and can be used to improve the compression performance of the video sequence, but does not require decoding operation during decoding.

[0060] In an exemplary embodiment, Figure 3 is a schematic diagram of an optional video data processing method according to an embodiment of the present application, sending the first frame of video data, that is, sending the target audio frame and the target video frame in the above-mentioned target sending order, and the target client receives the first frame of video data for playback, such as Figure 3 As shown: the server 302 acts as a transmitter, sending a first frame of video data 304, which includes an audio frame 306 and a video frame 308, and transmits it to a target client 310 through a network for playback, and the target client acts as a receiver of the first frame of video data.

[0061] In yet another exemplary embodiment, Figure 3 As shown: the above-mentioned video data to be played includes multiple frames of video data and multiple frames of audio data, wherein, according to the first frame playback condition corresponding to the target client 310, 3 video frames are required in the multiple frames of video data, and 2 video frames are required in the multiple frames of audio data, that is, the above-mentioned first frame video data 304 includes 3 video frames 306 and 2 audio frames 306.

[0062] Exemplarily, different target clients mentioned above have different first frame playback conditions. For example, a video conferencing software is installed on a mobile phone. The target opens the video conferencing software, enters the conference number to join the target conference, and then sends a video playback request to the background server of the video conferencing software. The background server determines the first frame playback condition based on the video playback request. At this time, the first frame playback condition is 1 video frame and 3 audio frames.

[0063] In another exemplary embodiment, a streaming media player is installed on a computer. The subject opens the streaming media player, searches for a movie to watch, and then sends a video playback request to the background server of the streaming media player. The background server determines the first frame playback condition based on the video playback request. At this time, the first frame playback condition is 10 video frames and 4 audio frames.

[0064] Furthermore, the target client determines the first frame playback condition and issues the video playback request, which can be performed through the following steps:

[0065] S1, the target client counts the information of the played video data received in its cache from time T0 to time T1, where T0 represents the time when the first packet of the played video data is received from the player, and T1 represents the time when the player starts decoding and rendering the first frame of the played video data;

[0066] S2, the target client obtains the above-mentioned first frame playback condition from the information of the played video data counted in S1, including the number of audio frames required for the first frame of video playback and the number of video frames required for the first frame of video playback;

[0067] S3, the target client sends a video play request, and the receiving module receives the video play request, wherein the video play request includes a first frame play condition;

[0068] S4, the receiving module transmits the first frame playing condition and the flow identifier FlowID to the policy control module;

[0069] S5, the policy control module obtains the first frame playback condition.

[0070] It should be noted that the stream identifier can be composed of a five-tuple of source address, destination address, source port, destination port and protocol number in the TCP protocol, or it can be represented by a connection identifier in the QUIC protocol. In the present application, if the first frame playback condition does not exist, the policy control module will load the first frame playback condition configured by the administrator for the video data to be played, and the first frame playback condition configured by the administrator will be declared in the configuration file in advance.

[0071] For example, Figure 4is a schematic diagram of another optional video data processing method according to an embodiment of the present application, such as Figure 4 As shown:

[0072] S1, the server (sender) obtains the video data to be played;

[0073] S2, the parsing module 402 parses the video data to be played, and obtains information such as the frame type 404, the frame duration 406, and the frame size 408 contained in the video data to be played;

[0074] S3, identifying, according to relevant fields in the audio and video protocol, that the video data to be played includes 300 audio frames and 267 video frames, the frame length of the video data to be played is 300 ms, and the frame size is 800 pixels in width and 600 pixels in height;

[0075] S4, sending the to-be-played video data frame type 404, the to-be-played video data frame duration 406, and the to-be-played video data frame size 408 to the sending module 410, ready for sending;

[0076] S5, the sending module determines whether the target client has completed receiving the video data to be played. If not, the first frame playback condition of the target client is obtained.

[0077] It should be noted that the audio and video protocols in the above S3 may include but are not limited to the HTTP-FLV protocol (Hypertext Transfer Protocol-Flash Video Protocol), where the Flash video protocol refers to a set of protocols used to play videos in Adobe Flash Player. The Flash video protocol can transfer video files to Adobe Flash Player and decode and display them in the player. Among the Flash video protocols, the most commonly used protocol is RTMP (Real-Time Messaging Protocol). RTMP is a real-time communication protocol that can transmit audio, video and data between the client and the server. The RTMP protocol is commonly used for streaming media transmission and can support real-time audio and video playback and interaction. It is a TCP-based protocol that uses a single persistent connection to transmit data. The RTMP protocol supports multiple video encoding and audio encoding formats.

[0078] Exemplarily, the type field in the Tag header can be used to identify the type 304 of the video data frame to be played. When the type field is 0x08, it indicates that the frame is an audio frame. When the type field is 0x09, it indicates that the frame is a video frame. Then, the size 308 of the video data frame to be played is obtained according to the PreviousTagSize field. The frame duration of the video data to be played can be understood as the gap between the playback time PTS of a data frame and the start time of the video stream. For example, the PTS of a data frame is pts_i, and the playback time of the first frame of the video data to be played is pts_0. Then, the duration of the audio and video can be calculated by the following formula:

[0079] Frame_time_len=pts_i–pts_0

[0080] In this application, the play time pts_0 of the first frame of the video data to be played can be obtained in the following way:

[0081] pts_0=max{pts_first_video,pts_first_audio}

[0082] Among them, the PTS (Presentation Time Stamp) of the frame refers to the display timestamp corresponding to each frame in the video stream. It is used to determine the playback order and time position of each frame in the video. The unit of PTS is usually milliseconds (ms), which means the time interval from the start of video playback to the current frame. pts_first_video means the PTS of the first video frame, and pts_first_audio means the PTS of the first audio frame. That is to say, the playback time pts_0 of the first frame of the video data to be played is the larger one of the PTS of the first video frame and the PTS of the first audio frame.

[0083] S204, determining the first frame video data and the target sending order according to the video data to be played and the first frame playing condition, wherein the first frame video data includes the target audio frames corresponding to the number of audio frames in the video data to be played and the target video frames corresponding to the number of video frames in the video data to be played, and the target sending order indicates the sending order of the target audio frames and the target video frames;

[0084] Optionally, in this embodiment, the above-mentioned first frame video data is the first frame video data of the video to be played, which may include but is not limited to real-time video and pre-recorded video, which may be high-definition video, 4K video, etc., and the audio may include but is not limited to real-time voice and pre-recorded audio, etc.

[0085] It should be noted that the above-mentioned target audio frame is the specified continuous time segment in the above-mentioned video data to be played, the above-mentioned target video frame is the specified continuous time segment in the above-mentioned video data to be played, and the number of target audio frames is the same as the number of audio frames in the above-mentioned first frame playback condition, and the number of target video frames is the same as the number of video frames in the above-mentioned first frame playback condition, that is, the above-mentioned first frame video data contains the same number of target audio frames as the number of audio frames in the above-mentioned first frame playback condition, and the same number of target video frames as the number of video frames in the above-mentioned first frame playback condition.

[0086] Exemplarily, the sending order of the target audio frame and the target video frame is determined according to the above target sending order, for example, Figure 5 is a schematic diagram of another optional video data processing method according to an embodiment of the present application. If the number of video frames in the above first frame playback condition is 3 and the number of audio frames is 1, such as Figure 5 As shown, the target sending order can be determined as follows:

[0087] S1, the video data to be played received by the sending module 502 are audio frame (A1), video frame (V1), video frame (V2), video frame (V3), video frame (V4), audio frame (A2), video frame (V5), video frame (V6), video frame (V7), audio frame (A2), etc.;

[0088] S2, adjust the sending order of the video data to be played to: A1, A2, A3, V1, V2, V3, V4, V5, V6, V7;

[0089] S3, adjusting the number of audio and video frames to be sent according to the first frame cache duration 504 of the target client's played video data, the first frame cache data size 506 of the played video data, the total number of first frame cache frames 508 of the played video data and other information included in the first frame playback condition, wherein the played video data is the historical video data that has been played by the target client before the video playback request is issued, and the historical played video data can come from the same sending module or from different sending modules;

[0090] S4, determining the video frame as V2 according to the first frame buffering time 504 of the played video data;

[0091] S5, determining the video frame as V4 according to the first frame buffer data size 506 of the played video data;

[0092] S6, determining the video frame as V5 according to the total number of cached frames of the first frame of the played video data 508;

[0093] S7, according to S4 to S6, intercepting the first 8 frames of the video data to be played after the first sorting in S2, namely A1, A2, A3, V1, V2, V3, V4, V5;

[0094] S8, the sending module sends 8 frames of data A1, A2, A3, V1, V2, V3, V4, and V5 to the target client, and temporarily stores V6 and V7, waiting for subsequent sending.

[0095] S206, encapsulating the first frame of video data into a first group of data packets according to the target sending order, and sending the first group of data packets to the target client, wherein, if there is a second group of data packets in the first group of data packets that have been confirmed to be sent to the target client and have not been confirmed to be received by the target client, sending the second group of data packets in reverse order of the target sending order;

[0096] Optionally, in this embodiment, the first group of data packets may include but are not limited to multiple data packets, and the data packets may include but are not limited to data packets encoded by audio frames, video frames, etc.

[0097] Specifically, the first frame of video data can be encapsulated into the above-mentioned first group of data packets, and only the first frame of video data exists in the first group of data packets. For example, the first group of data packets includes 3 data packets, the first data packet includes audio frame A1, the second data packet includes video frame V2, and the third data packet includes video frames A2 and V3. Corresponding to the above-mentioned first frame playback condition, two audio frames and two video frames are required. The first group of data packets are sent in sequence according to the above-mentioned target sending order. After sending the data packets, it is determined that the above-mentioned target client receives the first data packet. If it is impossible to determine whether the target client receives the second and third data packets, the second group of data packets includes the second and third data packets. The second group of data packets will be sent in reverse order of the target sending order, that is, after resending the third data packet to the target client again, the second data packet will be resent to the target client.

[0098] It should be noted that Figure 6 is a schematic diagram of another optional video data processing method according to an embodiment of the present application, such as Figure 6As shown, assuming that the first group of data packets includes data packet 601, data packet 602, data packet 603, data packet 604, and data packet 605, and the second group of data packets includes data packet 602, data packet 603, and data packet 604, and the target sending order is data packet 601, data packet 602, data packet 603, and data packet 604, and data packet 605 is sent in sequence. In the case of sending the second group of data packets, the first data packet sent is 604. At this time, a 602 confirmation message sent from the target client is received, indicating that data packet 602 has been received. At this time, data packet 602 in the second group of data packets will be removed, that is, at this time, only data packets 604 and 603 need to be resent. Similarly, if a 602 confirmation message and a 603 confirmation message are received from the target client at the same time, indicating that data packets 602 and 603 have been received, data packets 602 and 603 in the second group of data packets will be removed, that is, at this time, only data packets 604 need to be resent.

[0099] S208, when the first group of data packets have been confirmed to be sent to the target client and confirmed to be received by the target client, the non-first frame video data is sent to the target client, wherein the video data to be played includes the first frame video data and the non-first frame video data.

[0100] Optionally, in this embodiment, the non-first frame video data may include but is not limited to audio frames and video frames, and the non-first frame video data is also in the video data to be played.

[0101] Exemplarily, the video data to be played includes 30 video frames and 20 audio frames, and according to the first frame playback condition, it is determined that there are 2 video frames and 3 audio frames in the first frame video data, then there are 28 video frames and 17 audio frames in the non-first frame video data. After confirming that the first group of data packets have been sent and the target client confirms receipt, the non-first frame video data is sent.

[0102] In an exemplary embodiment, Figure 7 is a schematic diagram of another optional video data processing method according to an embodiment of the present application. The video data processing method proposed in the present application can be applied to the real-time video delivery scenario of the live broadcast platform, such as Figure 7 As shown:

[0103] S702, the background server of the live broadcast platform receives a video playback request sent by the target client;

[0104] S704, the background server of the live broadcast platform checks whether the video playback request carries the first frame playback condition;

[0105] S706, the video playback request carries the first frame playback condition, and the first frame playback condition is written into the policy control module;

[0106] S708: The video playback request does not carry the first frame playback condition, and the administrator configures the policy to obtain the first frame playback condition configured by the administrator;

[0107] S710, a parsing module parses live audio and video data;

[0108] S712, determining whether the first frame of live audio and video data has been parsed;

[0109] S714: The first frame of live audio and video data has not been sent yet, and the first frame playback condition is loaded;

[0110] S716, the first frame of live audio and video data is sent.

[0111] S718, adjusting the sending order of the first frame data of the live audio and video data according to the live audio and video frame sending strategy;

[0112] S720, sending the first frame data of the live audio and video data to the target client;

[0113] S722, the receiving module receives an ACK message sent by the target client, indicating that the sending of the first frame of the live audio and video data is completed;

[0114] S724, the sending module retransmits the first frame data of the live audio and video data in reverse order according to the sending order of the first frame data of the live audio and video data.

[0115] In another exemplary embodiment, the video data processing method proposed in the present application can be applied to the scenario of an online video conferencing platform, such as Figure 7 As shown:

[0116] S1, the backend server of the online video conferencing platform receives a video playback request from the target client;

[0117] S2, the backend server of the online video conferencing platform checks whether the video playback request carries the first frame playback condition;

[0118] S3: The video playback request carries the first frame playback condition, and the first frame playback condition is written into the policy control module;

[0119] S4: The video playback request does not carry the first frame playback condition. The administrator configures the policy and obtains the first frame playback condition configured by the administrator.

[0120] S5, the parsing module parses the conference audio and video data;

[0121] S6, determining whether the first frame of the conference audio and video data has been sent;

[0122] S7: The first frame of the conference audio and video data has not been sent yet, and the first frame playback conditions are loaded;

[0123] S8, the first frame of conference audio and video data is sent.

[0124] S9, adjusting the sending order of the first frame data of the conference audio and video data according to the conference audio and video frame sending strategy;

[0125] S10, sending the first frame of conference audio and video data to the target client;

[0126] S11, the receiving module receives an ACK message sent by the target client, indicating that the first frame of conference audio and video data has been sent.

[0127] S12, the sending module retransmits the first frame data of the conference audio and video data in reverse order according to the sending order of the first frame data of the conference audio and video data.

[0128] Through this embodiment, in response to a video playback request from a target client, the video data to be played and the first frame playback condition are determined, wherein the first frame playback condition includes the number of audio frames and the number of video frames required for the video data to be played to complete the first frame playback on the target client; the first frame video data and the target sending order are determined according to the video data to be played and the first frame playback condition, wherein the first frame video data includes the target audio frames corresponding to the number of audio frames in the video data to be played and the target video frames corresponding to the number of video frames in the video data to be played, and the target sending order represents the sending order of the target audio frames and the target video frames; the first frame video data is encapsulated into a first group of data packets according to the target sending order, and the first group of data packets are sent to the target client, wherein, in the first group of data packets, there is a second group of data packets that have been confirmed to be sent to the target client and have not yet been confirmed to be received by the target client, A second group of data packets are sent in reverse order of the target sending order; when the first group of data packets have been confirmed to be sent to the target client and confirmed to be received by the target client, non-first frame video data are sent to the target client, wherein the video data to be played includes first frame video data and non-first frame video data. By obtaining the video data to be played and the first frame playback condition, the ability of the client's first frame playback condition is determined according to the first frame playback condition, and then the sending order of the audio and video frames in the first frame video data is adjusted according to the first frame data of the video data to be played and the first frame playback condition, so as to achieve the purpose of optimizing the delay of the first frame video data, thereby realizing the technical effect of improving the viewing experience of the object, optimizing network transmission, and ensuring the continuity and smoothness of video playback, thereby solving the technical problem of low processing efficiency of video data due to poor optimization of the first frame data of video data and too long processing time.

[0129] In addition, the data packets of the first frame of video data are sent in the target sending order. When the video sender fails to promptly receive the message of successful data packet reception sent by the video receiver or some data packets are lost, the data packets are actively sent to the video receiver in the reverse order of the target sending order of the first frame of video data, thereby improving the reliability of video playback, reducing video transmission delay, and achieving the technical effect of improving network utilization and increasing the stability of data transmission.

[0130] As an optional scheme, the above-mentioned response to a video playback request from a target client to determine the video data to be played and the first frame playback condition includes: in response to the above-mentioned video playback request, obtaining the first frame data information of the above-mentioned target client, wherein the above-mentioned first frame data information includes a first moment and a second moment corresponding to the played video data cached by the above-mentioned target client, the above-mentioned first moment indicates the moment when the above-mentioned target client first receives the data packet corresponding to the above-mentioned played video data, and the above-mentioned second moment indicates the moment when the above-mentioned target client starts to decode and render the above-mentioned played video data; determining the above-mentioned first frame playback condition according to the above-mentioned first frame data information, wherein the above-mentioned first frame data information includes the above-mentioned audio frame number and the above-mentioned video frame number.

[0131] Optionally, in the present embodiment, the above-mentioned first frame data information may include but is not limited to the moment when the target client receives the first data packet of the video data to be played, the moment when the target client starts to decode and render the first frame data of the video data to be played, and the data of the video data to be played that the target client has received and cached, wherein the data of the video data to be played that the target client has received and cached may include but is not limited to the duration of the cached video data to be played, the data size of the cached video data to be played, the number of video frames of the cached video data to be played, the number of audio frames of the cached video data to be played, etc.

[0132] Specifically, the first moment is the moment when the target client receives the first data packet of the video data to be played, and the second moment is the moment when the target client starts to decode and render the first frame data of the video data to be played. That is to say, assuming that there are a total of 10 data packets of the video data to be played, the first moment is the moment when the target client receives the first data packet. When the third data packet is received, the target client starts to decode and render the first frame data of the video data to be played, which is the second moment.

[0133] In an embodiment of the present application, by responding to the above-mentioned video playback request, the first frame data information of the above-mentioned target client is obtained, wherein the above-mentioned first frame data information includes a first moment and a second moment corresponding to the played video data cached by the above-mentioned target client, the above-mentioned first moment indicates the moment when the above-mentioned target client first receives the data packet corresponding to the above-mentioned played video data, and the above-mentioned second moment indicates the moment when the above-mentioned target client starts decoding and rendering the above-mentioned played video data; the above-mentioned first frame playback condition is determined according to the above-mentioned first frame data information, thereby achieving the purpose of improving the object experience and reducing the server load.

[0134] As an optional scheme, the above-mentioned determination of the first frame video data and the target sending order based on the above-mentioned video data to be played and the above-mentioned first frame playback conditions includes: parsing the above-mentioned video data to be played, determining the initial audio frame and the initial video frame in the above-mentioned video data to be played, wherein the above-mentioned initial audio frame and the above-mentioned initial video frame are identified and determined by specific type fields; extracting the above-mentioned target audio frame from the above-mentioned initial audio frame according to the above-mentioned audio frame quantity and the above-mentioned video frame quantity, and extracting the above-mentioned target video frame from the above-mentioned initial video frame, generating the above-mentioned first frame video data, and setting the above-mentioned first frame video data to be sent according to the above-mentioned target sending order.

[0135] Optionally, in this embodiment, the above-mentioned initial audio frame is the audio frame contained in the above-mentioned video data to be played, and the above-mentioned initial video frame is the video frame contained in the above-mentioned video data to be played. For example, there are 20 video frames and 25 audio frames in the video data to be played. At this time, the initial audio frame is these 20 audio frames, and the initial video frame is these 25 video frames.

[0136] Specifically, the audio frames and video frames can be identified through fields of specific types. For example, the video data to be played is transmitted according to the HTTP-FLV protocol, and the type field in the Tag header of the data packet is extracted to identify the frame type 304 of the video data to be played. When the type field is 0x08, it indicates that the frame is an audio frame, and when the type field is 0x09, it indicates that the frame is a video frame.

[0137] It should be noted that the present application does not limit the number of the above-mentioned initial audio frames and initial video frames. After obtaining the above-mentioned target audio frame from the above-mentioned initial audio frame, and obtaining the above-mentioned target video frame from the above-mentioned initial video frame, the target audio frame and the target video frame are determined as the first frame video data, and the first frame video data is sent to the target client in the above-mentioned target sending order.

[0138] In an embodiment of the present application, by performing a parsing operation on the above-mentioned video data to be played, the initial audio frame and the initial video frame in the above-mentioned video data to be played are determined, wherein the above-mentioned initial audio frame and the above-mentioned initial video frame are identified and determined by a specific type field; according to the above-mentioned number of audio frames and the above-mentioned number of video frames, the above-mentioned target audio frame is extracted from the above-mentioned initial audio frame, and the above-mentioned target video frame is extracted from the above-mentioned initial video frame, the above-mentioned first frame video data is generated, and the above-mentioned first frame video data is set to be sent in the above-mentioned target sending order, thereby saving bandwidth resources and achieving the technical effect of improving the smoothness of video playback.

[0139] As an optional scheme, the above-mentioned target audio frame is extracted from the above-mentioned initial audio frame according to the above-mentioned number of audio frames and the above-mentioned number of video frames, and the above-mentioned target video frame is extracted from the above-mentioned initial video frame, the above-mentioned first frame video data is generated, and the above-mentioned first frame video data is set to be sent according to the above-mentioned target sending order, including: when the above-mentioned first frame playback condition includes the first frame cache length of the played video data, the first data frame is extracted from the above-mentioned video data to be played according to the above-mentioned first frame cache length, wherein the above-mentioned first data frame includes the first video frame or the first audio frame, the above-mentioned first video frame represents a video frame that meets the above-mentioned first frame cache length, the above-mentioned first audio frame represents an audio frame that meets the above-mentioned first frame cache length, and the above-mentioned first frame cache length represents the first frame video length of the above-mentioned played video data; the above-mentioned target audio frame, the above-mentioned target video frame and the above-mentioned first data frame are determined as the above-mentioned first frame video data, and the above-mentioned target audio frame and the above-mentioned target video frame are set to be sent according to the above-mentioned target sending order, and the above-mentioned first data frame is set to be sent according to the first sending order, wherein the above-mentioned first sending order is located after the above-mentioned target sending order.

[0140] Optionally, in this embodiment, the above-mentioned played video data can be understood as historical video data that has been played by the above-mentioned target client before completing the playback of the video data to be played. For example, the object enters the live broadcast room A of the live broadcast platform to watch for three consecutive days from the 1st day to the 3rd day, and the object enters the live broadcast room A again on the 4th day to obtain the real-time video data of the live broadcast room A. At this time, the real-time video data of the live broadcast room A on the 4th day is the above-mentioned video data to be played, and the historical video data of the live broadcast room A from the 1st day to the 3rd day is the above-mentioned played video data.

[0141] Specifically, the duration of the playback of the first frame of the played video data is the first frame cache duration. For example, if the subject enters live broadcast room A of the live broadcast platform one day ago to watch, the first frame cache duration can be understood as the time consumed for the first frame of the video in live broadcast room A to complete the playback.

[0142] Exemplarily, a video frame is extracted from the video data to be played as the first data frame according to the first frame cache time length, or an audio frame is extracted from the video data to be played as the first data frame according to the first frame cache time length. In other words, the first data frame is in the video data to be played, and the playback of the first data frame can be completed within the first frame cache time length. The first data frame may include multiple audio frames or only one audio frame, or the first data frame may include multiple video frames or only one video frame. The present application does not limit the number of frames of the first data frame.

[0143] In an exemplary embodiment, the first frame cache duration is 20ms, the target video frames are 3, the target audio frames are 4, the first data frame has 2 audio frames, and it takes 2ms to send the first data frame. At this time, the first data frame is an audio frame that meets the first frame cache duration. The first frame video data includes 3 video frames and 6 audio frames. After sending the target video frame and the target audio frame in the target sending order, the first data frame is sent. That is to say, the first sending order is after the target sending order. Sending the first frame video data can be understood as sending the target video frame and the target audio frame in the target sending order, and sending the first data frame in the first sending order.

[0144] In an embodiment of the present application, when the first frame playback condition includes the first frame cache time of the played video data, the first data frame is extracted from the video data to be played according to the first frame cache time, wherein the first data frame includes the first video frame or the first audio frame, the first video frame represents a video frame that meets the first frame cache time, the first audio frame represents an audio frame that meets the first frame cache time, and the first frame cache time represents the first frame video time of the played video data; the target audio frame, the target video frame and the first data frame are determined as the first frame video data, and the target audio frame and the target video frame are set to be sent in the target sending order, and the first data frame is set to be sent in the first sending order, so that when the video playback starts, the purpose of being able to immediately display a complete frame of video or audio is achieved, thereby improving the viewing experience of the object.

[0145] As an optional scheme, the above-mentioned target audio frame is extracted from the above-mentioned initial audio frame according to the above-mentioned number of audio frames and the above-mentioned number of video frames, and the above-mentioned target video frame is extracted from the above-mentioned initial video frame to generate the above-mentioned first frame video data and the above-mentioned target sending order, including: when the above-mentioned first frame playback condition includes the first frame cache data size of the played video data, the second data frame is extracted from the above-mentioned video data to be played according to the above-mentioned first frame cache data size, wherein the above-mentioned second data frame includes a second video frame or a second audio frame, the above-mentioned second video frame represents a video frame that meets the above-mentioned first frame cache data size, the above-mentioned second audio frame represents an audio frame that meets the above-mentioned first frame cache data size, and the above-mentioned first frame cache data size represents the first frame video data size of the above-mentioned played video data; the above-mentioned target audio frame, the above-mentioned target video frame and the above-mentioned second data frame are determined as the above-mentioned first frame video data, and the above-mentioned target audio frame and the above-mentioned target video frame are set to be sent according to the above-mentioned target sending order, and the above-mentioned second data frame is set to be sent according to the second sending order, wherein the above-mentioned second sending order is located after the above-mentioned target sending order.

[0146] Exemplarily, the data volume of the first frame of video data played is the first frame cache data size. For example, if the object enters the live broadcast platform's live broadcast room A to watch one day ago, the first frame cache data size can be understood as the sum of the audio data volume and video data volume of the first frame of video in live broadcast room A that has been played.

[0147] It should be noted that, according to the size of the first frame cache data, a video frame is extracted from the video data to be played as the second data frame, or, according to the size of the first frame cache data, an audio frame is extracted from the video data to be played as the second data frame. In other words, the second data frame is in the video data to be played, and the amount of data contained in the second data frame is less than the amount of data of the first frame of video data of the played video data. The second data frame may include multiple audio frames or only one audio frame, or the first data frame may include multiple video frames or only one video frame. The present application does not limit the number of frames of the second data frame.

[0148] In an exemplary embodiment, the first frame cache data size is 20 bits, there are 2 target video frames, there are 3 target audio frames, and the second data frame has 1 audio frame, which contains 2 bits of information. At this time, the second data frame is an audio frame that meets the first frame cache data size. The first frame video data includes 2 video frames and 4 audio frames. After sending the target video frame and the target audio frame in the target sending order, the second data frame is sent. That is to say, the second sending order is after the target sending order. Sending the first frame video data can be understood as sending the target video frame and the target audio frame in the target sending order, and sending the second data frame in the second sending order.

[0149] In an embodiment of the present application, when the first frame playback condition includes the first frame cache data size of the played video data, a second data frame is extracted from the video data to be played according to the first frame cache data size, wherein the second data frame includes a second video frame or a second audio frame, the second video frame represents a video frame that meets the first frame cache data size, the second audio frame represents an audio frame that meets the first frame cache data size, and the first frame cache data size represents the first frame video data size of the played video data; the target audio frame, the target video frame and the second data frame are determined as the first frame video data, and the target audio frame and the target video frame are set to be sent in the target sending order, and the second data frame is set to be sent in the second sending order, thereby achieving the technical effect of improving the viewing experience of the object and ensuring the stability of video playback.

[0150] As an optional scheme, the above-mentioned target audio frame is extracted from the above-mentioned initial audio frame according to the above-mentioned audio frame number and the above-mentioned video frame number, and the above-mentioned target video frame is extracted from the above-mentioned initial video frame to generate the above-mentioned first frame video data and the above-mentioned target sending order, including: when the above-mentioned first frame playback condition includes the total number of first frame cache frames of the played video data, the third data frame is extracted from the above-mentioned video data to be played according to the above-mentioned total number of first frame cache frames, wherein the above-mentioned third data frame includes the third video frame or the third audio frame, the above-mentioned third video frame represents the video frame that meets the above-mentioned total number of first frame cache frames, the above-mentioned third audio frame represents the audio frame that meets the above-mentioned total number of first frame cache frames, and the above-mentioned total number of first frame cache frames represents the frame number of the first frame of the above-mentioned played video data; the above-mentioned target audio frame, the above-mentioned target video frame and the above-mentioned third data frame are determined as the above-mentioned first frame video data, and the above-mentioned target audio frame and the above-mentioned target video frame are set to be sent according to the above-mentioned target sending order, and the above-mentioned third data frame is set to be sent according to the third sending order, wherein the above-mentioned third sending order is located after the above-mentioned target sending order.

[0151] In an exemplary embodiment, the number of data frames contained in the first frame of video data of the played video data is the total number of cached frames of the first frame. For example, if the subject enters the live broadcast room A of the live broadcast platform to watch one day ago, the total number of cached frames of the first frame can be understood as the sum of the number of video frames and the number of audio frames required to complete the playback of the first frame video of the live broadcast room A.

[0152] Exemplarily, a video frame is extracted from the video data to be played as the third data frame according to the total number of first frame cache frames, or an audio frame is extracted from the video data to be played as the third data frame according to the total number of first frame cache frames. In other words, the third data frame is in the video data to be played, the amount of data contained in the second data frame is less than the amount of data of the first frame video data of the played video data, the third data frame may include multiple audio frames, or only one audio frame, or the first data frame may include multiple video frames, or only one video frame. The present application does not limit the number of frames of the third data frame.

[0153] Furthermore, the total number of cached frames in the first frame is 20 data frames, including 10 audio frames and 10 video frames, the target video frames are 2, the target audio frames are 3, and the third data frame is 11 audio frames. At this time, the third data frame is an audio frame that meets the total number of cached frames in the first frame. The first frame video data includes 2 video frames and 14 audio frames. After sending the target video frames and target audio frames in the target sending order, the third data frame is sent. That is to say, the third sending order is after the target sending order. Sending the first frame video data can be understood as sending the target video frames and target audio frames in the target sending order, and sending the third data frame in the third sending order.

[0154] In an embodiment of the present application, when the first frame playback condition includes the total number of first frame cache frames of the played video data, a third data frame is extracted from the video data to be played according to the total number of first frame cache frames, wherein the third data frame includes a third video frame or a third audio frame, the third video frame represents a video frame that meets the total number of first frame cache frames, the third audio frame represents an audio frame that meets the total number of first frame cache frames, and the total number of first frame cache frames represents the number of frames of the first frame of the played video data; the target audio frame, the target video frame and the third data frame are determined as the first frame video data, and the target audio frame and the target video frame are set to be sent in the target sending order, and the third data frame is set to be sent in the third sending order, thereby avoiding unnecessary resource consumption.

[0155] As an optional scheme, the above-mentioned first frame video data is encapsulated into a first group of data packets according to the above-mentioned target sending order, and the above-mentioned first group of data packets are sent to the above-mentioned target client, including: encapsulating the above-mentioned first frame video data into multiple data packets; setting a corresponding data packet sequence number for each of the above-mentioned multiple data packets according to the above-mentioned target sending order, wherein the value of the above-mentioned data packet sequence number is used to indicate the sending order of the corresponding data packets; and sending the above-mentioned multiple data packets to the above-mentioned target client in sequence according to the value of the above-mentioned data packet sequence number.

[0156] Optionally, in this embodiment, the above-mentioned data packet may include but is not limited to the source address of the data packet, the destination address of the data packet, the transmission protocol type, the length of the data packet, etc. The above-mentioned data packet sequence number corresponds to the data packet, and the value of the datagram sequence number is determined according to the sending order of the data packet. For example, there are 5 data packets to be sent in the first group of data packets, and the sending order of the first group of data packets is D1 data packet, D2 data packet, D3 data packet, D4 data packet, and D5 data packet. The data packet sequence of D1 data packet is 1, the data packet sequence of D2 data packet is 2, the data packet sequence of D3 data packet is 3, the data packet sequence of D4 data packet is 4, and the data packet sequence of D5 data packet is 5.

[0157] It should be noted that when the above-mentioned first frame video data is encapsulated into the first group of data packets for transmission, the above-mentioned target transmission order must be met. For example, the first frame video data includes 8 video frames and 2 audio frames. These 8 video frames and 2 audio frames are encapsulated in 5 data packets respectively, among which: D1 data packet encapsulates 2 video frames, D2 data packet encapsulates 2 video frames, D3 data packet encapsulates 2 video frames, D4 data packet encapsulates 2 video frames, and D5 data packet encapsulates 2 audio frames. The target transmission order is D1 data packet, D2 data packet, D3 data packet, D4 data packet, and D5 data packet are sent in sequence.

[0158] Exemplarily, there are 4 data packets to be sent in the first group of data packets, and the sending order of the first group of data packets is data packet D1, data packet D2, data packet D3, and data packet D4. The data packet order of data packet D1 is A, the data packet order of data packet D2 is B, the data packet order of data packet D3 is C, and the data packet order of data packet D4 is D. When the first group of data packets are sent to the target client, they will be sent according to the data packet sequence numbers corresponding to the data packets. That is, data packet D1 with data packet sequence number A is sent first, then data packet D2 with data packet sequence number B is sent, then data packet D3 with data packet sequence number C is sent, and finally data packet D4 with data packet sequence number D is sent.

[0159] It should be noted that the values ​​of the above-mentioned data packet sequence numbers are not limited to Arabic numerals, and this application does not limit the determination of the size relationship of the values ​​of the above-mentioned data packet sequence numbers.

[0160] In an embodiment of the present application, the first frame video data is encapsulated into a plurality of data packets; a corresponding data packet sequence number is set for each of the plurality of data packets according to the target sending order, wherein the value of the data packet sequence number is used to indicate the sending order of the corresponding data packets; and the plurality of data packets are sent to the target client in sequence according to the value of the data packet sequence number, thereby achieving a technical effect of ensuring the security of video data transmission.

[0161] As an optional solution, the method further includes: obtaining a first data packet sequence number, wherein the first data packet sequence number is used to identify a data packet that has been sent to the target client; receiving a second data packet sequence number returned by the target client, wherein the second data packet sequence number is used to identify a data packet confirmed to be received by the target client; determining the second group of data packets based on the first data packet sequence number and the second data packet sequence number, and sending the second group of data packets in reverse order of the target sending order.

[0162] Optionally, in this embodiment, the above-mentioned first data packet sequence number and the above-mentioned second data packet sequence number respectively represent one of the above-mentioned data packet sequence numbers. In other words, the total number of data packets that the above-mentioned target client should receive can be known according to the first data packet sequence number, and the total number of data packets actually received by the target client can be known according to the second data packet sequence number.

[0163] For example, Figure 8 is a schematic diagram of another optional video data processing method according to an embodiment of the present application, such as Figure 8 As shown, there are 4 data packets in the first group of data packets 802, namely, data packet D1, data packet D2, data packet D3, and data packet D4. Then, the first data packet sequence number is used to identify the sending of the first group of data packets. After sending the first group of data packets 802, the second data packet sequence number is received. From the second data packet sequence number, it is known that the target client has only received data packets D1 and D2, that is, the target client has not fed back whether data packets D3 and D4 are received. At this time, the second group of data packets 804 includes data packets D3 and D4, and the second group of data packets 804 will be sent again in the reverse order of the target sending order. Assuming that the target sending order indicates that the D3 data packet is sent first, and then the D4 data packet is sent, then the sending order of the second group of data packets is to send the D4 data packet first, and then the D3 data packet.

[0164] In an embodiment of the present application, by obtaining a first data packet sequence number, wherein the first data packet sequence number is used to identify a data packet that has been sent to the target client; receiving a second data packet sequence number returned by the target client, wherein the second data packet sequence number is used to identify a data packet confirmed to be received by the target client; determining the second group of data packets based on the first data packet sequence number and the second data packet sequence number, and sending the second group of data packets in reverse order of the target sending order, the purpose of improving the transmission efficiency of video data is achieved.

[0165] As an optional scheme, the above-mentioned second group of data packets is determined according to the above-mentioned first data packet sequence number and the above-mentioned second data packet sequence number, and the above-mentioned second group of data packets are sent in the reverse order of the above-mentioned target sending order, including: determining the number of data packets currently confirmed to be received by the above-mentioned target client according to the above-mentioned second data packet sequence number; extracting retransmission data packets corresponding to the above-mentioned number of data packets from the above-mentioned second group of data packets in the reverse order of the above-mentioned target sending order, sending the above-mentioned retransmission data packets to the above-mentioned target client, and deleting the above-mentioned retransmission data packets from the above-mentioned second group of data packets.

[0166] Optionally, in this embodiment, the above-mentioned retransmitted data packet can be understood as part of the data packets in the above-mentioned first group of data packets. After sending the first group of data packets, the target client does not feedback whether this part of the data packets is received. After the above-mentioned retransmitted data packet is sent to the target client, the corresponding retransmitted data packet will be deleted in the second group of data packets.

[0167] Exemplarily, the second group of data packets includes data packet D1, data packet D2, and data packet D3. At this time, it is determined according to the second data packet sequence number that the target client has received 2 data packets. Then, the number of the retransmitted data packets is 2. Data packet D2 and data packet D3 are determined as retransmitted data packets in reverse order of the target sequence and sent. After the sending is completed, data packet D2 and data packet D3 in the second group of data packets are deleted. Currently, only data packet D1 exists in the second group of data packets. If the target client is determined to have received data packet D2 but has not received data packet D3 according to the second data packet sequence number next time, data packet D3 is again determined as a data packet in the second group of data packets and needs to be retransmitted. If it is determined according to the second data packet sequence number that the target client currently confirms that it has received 2 data packets, then data packet D3 is determined as a retransmitted data packet together with data packet D1 and sent to the target client.

[0168] In an embodiment of the present application, by determining the number of data packets currently confirmed to be received by the above-mentioned target client according to the above-mentioned second data packet sequence number; extracting retransmission data packets corresponding to the above-mentioned number of data packets from the above-mentioned second group of data packets in the reverse order of the above-mentioned target sending order, sending the above-mentioned retransmission data packets to the above-mentioned target client, and deleting the above-mentioned retransmission data packets from the above-mentioned second group of data packets, the technical effect of enhancing the fault-tolerant processing capability of video data transmission is achieved.

[0169] As an optional scheme, the above method also includes: responding to a video playback request from a target client, determining the above video data to be played, and initializing a target variable for the above video data to be played, wherein the above target variable is used to indicate whether the above first frame video data has been sent; obtaining a third data packet sequence number, wherein the third data packet sequence number is used to identify a data packet that needs to be sent to the above target client; when the second data packet sequence number and the third data packet sequence number are different, determining that the above first frame video data has not been sent, sending the above first group of data packets to the above target client in accordance with the above target sending order, and sending the above second group of data packets in reverse order of the above target sending order; when the second data packet sequence number and the third data packet sequence number are the same, determining that the above first frame video data has been sent, stopping sending the above second group of data packets, and sending non-first frame video data to the above target client.

[0170] Optionally, in this embodiment, the present application does not specifically limit the value of the above-mentioned target variable, and uses the target variable to determine whether the transmission of the above-mentioned first frame of video data is completed. For example, when preparing to send the first data packet in the first group of data packets, the value of the target variable is set to 0, and when the above-mentioned target client receives the first group of data packets, the value of the target variable is set to 1, that is, the transmission of the above-mentioned first frame of video data is completed, or, when preparing to send the first data packet in the first group of data packets, the value of the target variable is set to 1, and when the above-mentioned target client receives the first group of data packets, the value of the target variable is set to 0, that is, the transmission of the above-mentioned first frame of video data is completed.

[0171] Specifically, the data packet that needs to be sent to the above-mentioned target client is determined by the above-mentioned third data packet sequence number. If the above-mentioned second data packet sequence number is different from the above-mentioned third data packet sequence number, it means that the first frame of video data has not been sent, and the value of the target variable is the initial value and no longer changes, or the value of the number of data packets required to send the first frame of video data is taken as the target variable according to the proportion of the number of data packets not received by the target client. Then the value of the target variable will be positively correlated with the number of data packets not received by the target client. For example, at the previous moment, the number of data packets not received by the target client is 2, and the number of data packets required to send the first frame of video data is fixed at 10. At this time, the value of the target variable is 0.2. At the next moment, the number of data packets not received by the target client becomes 1. At this time, the value of the target variable becomes 0.1. Finally, the number of data packets not received by the target client is 0, and the value of the target variable is 0, that is, the transmission of the above-mentioned first frame of video data is completed.

[0172] In an exemplary embodiment, the second data packet sequence number is the same as the third data packet sequence number, that is, the number of data packets currently confirmed to be received by the target client is the same as the number of data packets that need to be sent to the target client. In other words, the target client has received all the data packets in the first group of data packets, and the second group of data packets is no longer sent at this time, and the non-first frame video data is sent to the target client. At this time, the target client can play the first frame video data.

[0173] In an embodiment of the present application, by responding to a video playback request from a target client, the above-mentioned video data to be played is determined, and a target variable is initialized for the above-mentioned video data to be played, wherein the above-mentioned target variable is used to indicate whether the above-mentioned first frame of video data has been sent; a third data packet sequence number is obtained, wherein the above-mentioned third data packet sequence number is used to identify a data packet that needs to be sent to the above-mentioned target client; when the above-mentioned second data packet sequence number and the above-mentioned third data packet sequence number are different, it is determined that the above-mentioned first frame of video data has not been sent, and the above-mentioned first group of data packets are sent to the above-mentioned target client in accordance with the above-mentioned target sending order, and the above-mentioned second group of data packets are sent in reverse order of the above-mentioned target sending order; when the above-mentioned second data packet sequence number and the above-mentioned third data packet sequence number are the same, it is determined that the above-mentioned first frame of video data has been sent, the sending of the above-mentioned second group of data packets is stopped, and the non-first frame video data is sent to the above-mentioned target client, thereby achieving the technical effect of improving the efficiency of video data transmission.

[0174] As an optional scheme, the above-mentioned response to a video playback request from a target client to determine the video data to be played and the first frame playback condition includes: detecting the above-mentioned video playback request, determining the above-mentioned video data to be played and the first frame playback condition field; when the above-mentioned first frame playback condition field indicates that the above-mentioned video playback request includes the above-mentioned first frame playback condition, setting the above-mentioned first frame playback condition according to the above-mentioned video playback request; when the above-mentioned first frame playback condition field indicates that the above-mentioned video playback request does not include the above-mentioned first frame playback condition, obtaining the above-mentioned first frame playback condition which is set in advance.

[0175] Optionally, in this embodiment, the above-mentioned video playback request may include a first frame playback condition field. According to the first frame playback condition field, it can be known whether the above-mentioned target client has the first frame playback condition. For example, the first frame playback condition field is set to 1, indicating that the target client has the first frame playback condition, and the first frame playback condition field is set to 0, indicating that the target client does not have the first frame playback condition; or, the first frame playback condition field is set to 0, indicating that the target client has the first frame playback condition, and the first frame playback condition field is set to 1, indicating that the target client does not have the first frame playback condition.

[0176] Furthermore, assuming that the target client does not have the first frame playback condition, the pre-set first frame playback condition will be obtained. That is to say, the first frame playback condition obtained at this time does not come from the first frame playback condition in the video playback request, but the pre-set first frame playback condition. For example, the object opens the mobile phone to enter the social media application A and watch video B, that is, sends a video playback request to the background server of the social media application A. After the background server detects the video playback request, it finds that the first frame playback condition field is set to 0, that is, the video playback request does not have the first frame playback condition. The background server then obtains the first frame playback condition pre-set by the administrator as the first frame playback condition of the video B requested by the object.

[0177] In an embodiment of the present application, the video data to be played and the first frame playback condition field are determined by detecting the video playback request; when the first frame playback condition field indicates that the video playback request includes the first frame playback condition, the first frame playback condition is set according to the video playback request; when the first frame playback condition field indicates that the video playback request does not include the first frame playback condition, the pre-set first frame playback condition is obtained, thereby achieving a smoother and more stable video playback effect.

[0178] The above is only an example, and this application does not make any specific limitation.

[0179] Obviously, the embodiments described above are only part of the embodiments of the present application, rather than all the embodiments.

[0180] The present application is described in detail below with reference to specific embodiments:

[0181] Exemplarily, the present application proposes a method for optimizing the first frame of audio and video based on frame reordering, which is used to improve the service experience of the client and belongs to the field of computer networks. Among them, the method includes: after the server receives a request from the object terminal and obtains the data to be sent, the server obtains the audio and video frame type of the video data to be played through the parsing module, and passes the type and data to the sending module; the policy control module obtains the first frame playback condition of the client, and passes the above first frame playback condition to the sending module; the sending module adjusts the sending order of the audio and video frames according to the audio and video frame type and the first frame playback condition. In particular, in the present application, the sending module sends the audio frame that should be transmitted later in advance to meet the client's first frame playback requirements; when the sending module sends the first frame data, the sending module will perform a "reverse retransmission" operation, and actively retransmit according to the message number from large to small to improve the completion time of the first frame. In the present application, the first frame playback condition in the policy control module of the server (sender) can be obtained through the newly added end-cloud collaboration method, or it can be configured by the administrator himself. The audio and video first frame optimization method based on frame reordering proposed in this application can adjust the sending strategy in a targeted manner according to the actual first frame playback conditions of the client, thereby improving the completion time of the first frame, which is beneficial to improving the client's business experience and enhancing the core competitiveness of CDN (CONTENT DELIVERY NETWORK) manufacturers.

[0182] It should be noted that the first frame delay is an important criterion for measuring the performance of CDN (CONTENT DELIVERY NETWORK) manufacturers and a key indicator reflecting the client's service experience. From the perspective of CDN (CONTENT DELIVERY NETWORK) manufacturers, if the traffic transmission service they provide can complete the transmission of the first frame of audio and video in a shorter time, then they can gain recognition from more application providers. In this case, these application providers will switch more traffic to the CDN (CONTENT DELIVERY NETWORK) manufacturer, and at this time, the CDN manufacturer will also gain more revenue. From the perspective of the client, if the first frame of the video is played faster, the object is more inclined to watch these audio and video (the above-mentioned video data to be played). In this case, the application provider will obtain more traffic and other forms of payment (such as shopping); related, if the object opens a certain audio and video and the picture is delayed, these objects will tend to quit watching the audio and video, or even directly download the application, which is obviously unacceptable to the application provider. Therefore, how to reduce the first frame delay of audio and video services and optimize the service experience of target terminals has become the focus of continuous research by major CDN (Content Delivery Network) manufacturers and application providers.

[0183] Furthermore, there are many methods for optimizing the first frame delay, which can be generally divided into two categories. One is to increase the sending rate and reduce the additional time loss caused by packet loss repair by setting the initial sending parameters; for example, adjust the initial window size to 10 to further improve the sending efficiency; there are also many methods to configure a "more aggressive" window size through machine learning, in the hope of completing the transmission of the first frame of data in a shorter time. Another method focuses on protocol innovation, improving the service experience of the target terminal by designing, implementing, and deploying a transmission protocol that is more suitable for dubbing and video traffic; for example, the QUIC protocol improves the time loss of the original TCP protocol in the handshake process and supports the 0-RTT strategy; the multi-path transmission protocol overcomes the problem of unstable transmission efficiency caused by the diversity of network conditions and the dynamic nature of real-time status of the existing single path, and realizes the transmission of the same data resource through two paths, further shortening the data transmission completion time.

[0184] In an exemplary embodiment, under a multi-supplier strategy, the existing audio and video first frame optimization method has the following problems: CDN (Content Delivery Network) manufacturers cannot know the first frame playback conditions of third-party clients, and blindly improving the transmission efficiency of the first video frame may not lead to the optimization of the first frame delay. On the one hand, there are large differences in the software and hardware levels of the target clients. For example, for clients with low hardware levels, it is necessary to receive multiple audio and video frames before starting to decode and play the first frame; while for some target terminals with higher performance, it may be possible to play after receiving the first video frame. On the other hand, the playback configuration of the audio and video application of the third-party application provider is invisible to the CDN (CONTENTDELIVERY NETWORK) manufacturer. For example, in order to alleviate audio and video jams, the player will not play temporarily after receiving the first frame, and will start decoding and playback after a certain number of audio and video frames are cached.

[0185] For example, Figure 3 As shown in the figure, the first frame playback strategy of the client is to start the first screen decoding and playback operation only when receiving 1 video frame and 3 audio and video frames; after receiving the request message from the client, the server sends audio frame, video frame, video frame, video frame, audio frame, video frame, video frame, audio frame in sequence; and in this figure, when the client plays the first frame, it must receive the above 8 complete audio and video frames. Obviously, the 2nd, 3rd, 4th, and 5th video frames are redundant for the first frame playback, that is, without the above 2nd, 3rd, 4th, and 5th video frames, the client can still render the first frame. The packet loss of the tail data of the first frame is likely to become a bottleneck factor for the integrity of the first frame. In other words, the factor restricting the integrity of the first frame lies in the packet loss repair of its tail data, rather than the packet loss repair of the front data.

[0186] In an exemplary embodiment, Figure 8 As shown in the figure, during the transmission of the first frame data, the first and third audio data packets are lost, and the third video data packet is also lost. At this time, the data transmitter needs to retransmit these lost packets. As can be seen from the figure, the packet loss of the first audio data is repaired by retransmission at time t1, the packet loss of the third video data is repaired by retransmission at time t2, and the packet loss of the third audio data is repaired by retransmission at time t3. In this case, the entire first frame completion time is t3-t0. It can be seen that the factor affecting the integrity of the first frame lies in the transmission packet loss and retransmission repair of the later data in the first frame, rather than the packet loss and repair of the front data.

[0187] Exemplarily, to solve the above problems, this application will optimize the first frame delay of audio and video from the following aspects: enabling the sender to perceive the first frame playback conditions of the client, and adjust the audio and video frame sending order (the above target sending order) according to the above perceived conditions; enabling the sender to alleviate the problem of increased first frame completion time due to packet loss at the tail of the first frame through active retransmission. The audio and video first frame optimization method based on frame reordering designed in this application can accurately overcome the problems of unknown first frame playback strategy of third-party clients and high first frame completion time caused by data loss at the tail of the first frame, and enhance the core competitiveness of CDN (Content Delivery Network) manufacturers.

[0188] Specifically, the purpose of this application is to solve the problems of uncontrollable first frame optimization and low service experience caused by the unknown first frame playback conditions and high completion time caused by the loss of the tail data of the first frame in the existing audio and video services by third-party applications, and propose a method for optimizing the first frame of audio and video based on frame reordering. In this application, the parsing module of the server (sending end) performs frame parsing on the data to be sent, clarifies the frame type corresponding to the data, and passes these data and the frame type to the sending module; after receiving the above information, the sending module determines whether the current data is the first frame. If the current data is the first frame, the client's first frame playback conditions are obtained from the policy control module, and the sending order of the audio and video frames is adjusted according to the above conditions, and the audio and video frames that meet the first frame playback conditions are sent first; when the first frame data is all sent out, the sending module of the server performs a "reverse retransmission" operation on the message that has been sent but not yet confirmed, so as to reduce the phenomenon of increased first frame completion time caused by the loss of the tail data of the first frame.

[0189] For example, Fig. 9 is a schematic diagram of another optional video data processing method according to an embodiment of the present application, such as Fig. 9 As shown:

[0190] S1, after receiving the audio and video request message from the target terminal, the server (sender) writes the client's first frame playback condition (if there is a first frame playback condition) carried in the message into the newly added policy control module;

[0191] S2, the first frame playback condition can also be configured by the administrator and declared in the configuration file; the sending end obtains the data to be sent according to the audio and video identifier in the request message, and obtains the frame type of the audio and video data to be sent through the parsing module before sending the data, and passes the data and frame type to the sending module;

[0192] S3, after receiving the above information, the sending module determines whether the current audio and video stream has completed the transmission of the first frame; if the current audio and video stream has not completed the transmission of the first frame, the first frame playback condition of the current client is obtained from the policy control module, including the number of audio frames and video frames that need to be met;

[0193] S4, the sending module adjusts the sending order of the audio and video frames according to the frame type of the data to be sent and the first frame playing condition. The core idea is to give priority to sending the audio and video frames of the first frame of data;

[0194] S5, after receiving the requested audio and video frame data, the client sends a message confirmation message (ACK message) to the server. After receiving the above ACK message, the receiving module informs the sending module whether the first frame data has been transmitted and which numbered messages have been confirmed to be received. At the same time, the completion time of the first frame is fed back to the administrator for the administrator to configure the next stage;

[0195] S6, every time the sending end receives a confirmation message of a message, it selects one or more messages from the messages that have been sent but not yet confirmed to perform a retransmission operation. In this application, the message with a larger message number is selected (to perform a retransmission operation).

[0196] It should be noted that the technical solution proposed in this application enables the data sender to perceive the client's first frame playback conditions, and performs active retransmission operations on the tail packet loss that affects the first frame completion time. By prioritizing the transmission of the first frame data and actively retransmitting operations, the client's first frame delay is minimized and the client's service experience is improved.

[0197] S1, after receiving the request from the target terminal, the server (sender) extracts the first frame playback condition of the client (if the first frame playback condition exists) and passes it to the policy control module; if the first frame playback condition does not exist in the request message, the first frame playback condition in the policy control module is configured by the administrator, and the first frame playback condition is used by the sending module to obtain it when sending the first frame data later;

[0198] After receiving the request message from the target terminal, the receiving module of the server performs the following operations in sequence: detect whether the request message carries the first frame playback condition FF_con of the client player; in the present application, the request message adds a new FF_bool field, which occupies 1 bit. When FF_bool=1, it indicates that the request message carries the first frame playback condition FF_con; otherwise, it indicates that the request message does not carry the first frame playback condition FF_con; if the above-mentioned first frame playback condition is carried (i.e., FF_bool=1), the receiving module extracts the first frame playback condition FF_con of the client from the request message; in the present application, the first frame playback condition FF_con includes but is not limited to the following: the number of audio frames A_frame_amount; the number of video frames V_frame_amount; the cache duration Buffer_time; the cache data size Buffer_datasize; the number of cached audio and video frames Buffer_frame_amount;

[0199] It should be noted that in this application, the audio and video player detects its own first frame playback conditions, which are as follows:

[0200] S1-1, the audio and video player of the target terminal counts the audio and video data information received in its cache from time T0 to time T1; T0 represents the time when the first message requesting audio and video data is received from the player, and T1 represents the time when the player starts decoding and rendering (first frame) data;

[0201] S1-2, the first frame data information counted by the player includes the first frame playback condition FF_con, namely A_frame_amount, V_frame_amount, Buffer_time, Buffer_datasize, Buffer_frame_amount, etc.;

[0202] The receiving module passes the above-mentioned first frame playback condition and flow identifier FlowID to the policy control module. In this application, after receiving the request message containing the first frame playback condition, the receiving module passes the message containing FF_con and FlowID to the policy control module. The message is implemented through the pkt_ff_con message, as shown in the following formula:

[0203] Pkt_ff_con={FlowID,FF_con}

[0204] In this application, the flow identifier FlowID can be composed of a five-tuple including source address, destination address, source port, destination port and protocol number in the TCP protocol, or can be represented by a connection identifier in the QUIC protocol; in this application, if FF_con is empty, it means that the request message does not carry the first frame playback condition; if FF_con is empty, the policy control module will load the first frame playback condition FF_con_admin configured by the administrator for the flow; in this application, FF_con_admin is configured by the administrator and declared in the configuration file; FF_con_admin contains A_frame_amount, V_frame_amount, Buffer_time, Buffer_datasize, Buffer_frame_amount and other contents, which are all conditions set by the administrator through continuous testing and debugging;

[0205] S2, the server (sender) obtains the data to be sent and before sending the data, the parsing module performs audio and video parsing operations, and passes the obtained frame type, frame duration, frame size and other information as well as audio and video data to the sending module. The specific process is as follows:

[0206] S2-1, in this application, the type of the audio and video frame Frame_type can be obtained by identifying the relevant fields in the existing audio and video protocol. For example, in the HTTP-FLV protocol, the type of the audio and video frame can be identified by the type field in the Tag header. When type is 0x08, it indicates that the frame is an audio frame; when type is 0x09, it indicates that the frame is a video frame.

[0207] S2-2, in this application, the size of the audio and video frame Frame_size can be obtained by identifying the relevant fields in the existing audio and video protocol. For example, in the HTTP-FLV protocol, the size of the audio and video frame can be obtained through the PreviousTagSize field (occupying 4 bits);

[0208] S2-3, in this application, the duration of the audio and video frame Frame_time_len refers to the gap between the playback time PTS of the frame and the start time of the video stream. For example, the PTS of a frame is pts_i, and the playback time of the first frame of the video stream is pts_0, then the duration of the audio and video can be calculated by the following formula:

[0209] Frame_time_len=pts_i-pts_0

[0210] In this application, pts_0 is calculated by the following formula:

[0211]

[0212] Among them, pts_first_video represents the PTS of the first video frame, and pts_first_audio represents the PTS of the first audio frame;

[0213] S2-4, in this application, after the parsing module parses the received audio and video frames, it passes the audio and video data Frame_data as well as the frame type, frame duration, frame size, etc. to the sending module, such as Figure 4 shown.

[0214] S3, after receiving the above-mentioned audio and video frame data and information such as type, size, and duration, the sending module determines whether the first frame of audio and video data has been transmitted. If not, the first frame playback condition of the client is obtained from the policy control module, and the sending strategy of the audio and video frame is determined according to the condition, as follows:

[0215] S3-1, the sending module determines whether the first frame of the audio and video has been sent. After receiving the request of the target terminal, it creates a variable FF_Complete for the audio and video stream, with an initial value of 0; when the first frame of the audio and video data is transmitted, the value of FF_Complete is set to 1; when FF_Complete = 0, it indicates that the first frame of data has not been sent;

[0216] S3-2, after receiving the audio and video frames and parsing information from the parsing module, if the current FF_Complete=0, the sending module obtains the first frame playback condition FF_con of the client from the policy control module; and executes the first frame sending strategy of the following step S3-3;

[0217] S3-3, the sending module determines the sending strategy of the audio and video frames according to the obtained FF_con, as follows Figure 5As shown, the sending module adjusts the sending order of audio and video frames according to the number of audio frames A_frame_amount and the number of video frames V_frame_amount in FF_con; for example, if A_frame_amount=3 and V_frame_amount=1 in FF_con, and the data received by the sending module from the parsing module are audio frame (A1), video frame (V1), video frame (V2), video frame (V3), video frame (V4), audio frame (A2), video frame (V5), video frame (V6), video frame (V7), audio frame (A2), etc., then the sending module adjusts the sending order of audio and video to: A1, A2, A3, V1, V2, V3, V4, V5, V6, V7; the sending module adjusts the number of audio and video frames to be sent according to Buffer_time, Buffer_datasize, and Buffer_frame_amount in FF_con. The core idea is to ensure that the audio and video frame data required by FF_con can be sent out at one time; as shown in FIG. Figure 5 As shown, the sending module determines the "dividing point" of the audio and video frames that meet the conditions according to Buffer_time, Buffer_datasize, and Buffer_frame_amount in FF_con. For example, the audio and video frame that meets Buffer_time is V2, the audio and video frame that meets Buffer_datasize is V4, and the audio and video frame that meets Buffer_frame_amount is V5. In this case, the sending module intercepts A1, A2, A3, V1, V2, V3, V4, V5 of the audio and video (i.e., A1, A2, A3, V1, V2, V3, V4, V5) after the order is adjusted in step i and sends them out, and the remaining V6 and V7 will be temporarily stored in the sending module for subsequent sending; that is, the sending module adjusts the initial sending window init_cwnd to a value that can simultaneously meet the FF_con conditions; init_cwnd is configured to start from the audio and video stream (the above-mentioned video data to be played) and end at the cumulative data size of V5;

[0218] Exemplarily, the sending module counts the number (range) Pkt_num_set of the traffic message corresponding to the first frame data that meets the FF_con condition and transmits it to the receiving module, which is used to count whether the first frame data of the current audio and video stream has been transmitted to update the FF_complete parameter value;

[0219] S4, after receiving the audio and video data from the sending module, the target terminal sends a message confirmation message (ACK message) to the receiving module, which carries the message number that has been received (or the message number that has not been received). The receiving module determines whether the first frame data has been transmitted based on the above information, as shown in step S5;

[0220] S5, after receiving the ACK message from the target terminal, the receiving module determines whether the audio and video frame data that meets the FF_con condition has been transmitted. The receiving module summarizes and counts the message numbers confirmed by the target terminal to obtain the message number set Pkt_num_rec_set; the receiving module determines the relationship between Pkt_num_rec_set and Pkt_num_set: if the two are equal, it indicates that the audio and video data that meets the FF_con condition has been transmitted, and FF_complete is set to 1; otherwise, it is set to 0; when FF_complete is set to 1 for the first time, the receiving module notifies the sending module that FF_complete=1, and releases the previously cached information such as Pkt_num_rec_set and Pkt_num_set;

[0221] S6, after receiving the ACK message from the target terminal, the receiving module informs the sending module of the message number that the target terminal has received. The sending module performs the following operations according to the above information: the sending module performs a reverse retransmission operation from the "sent but unconfirmed messages" queue according to the number of messages recently received by the target terminal (represented as N), and the number of retransmitted messages is M; Figure 6 As shown, the sending module maintains a message queue that has been sent but not confirmed to be received for the audio and video data messages that meet the FF_con condition; after receiving the ACK message fed back from the target terminal, M messages are extracted from the above queue for retransmission; in this application, M = α*N, where N is the number of messages recently received by the client declared in the ACK message; by default, α = 1.0; this value supports administrator configuration and is reflected in the configuration file; in this application, after receiving the ACK message, the selection basis of the retransmission message is to retransmit according to the message number from large to small;

[0222] For example, the message number set recorded in the above queue is 8, 9, 10, 11, 12, 13, 14; and the latest ACK message declares that message numbers 8 and 9 have been recently received by the client; at this time, the sending module deletes 8 and 9 from the above queue; and retransmits message numbers 14 and 13 in sequence;

[0223] It should be noted that in the present application, when FF_complete=1, it indicates that the audio and video data that meets the FF_con condition has been transmitted. At this time, the sending module will no longer perform the above-mentioned reverse retransmission operation; the sending module only performs reverse retransmission operations on the audio and video data packets that meet the FF_con condition, that is, the message queue that has been sent but not confirmed to be received only records the audio and video data packets that meet the FF_con condition (the above-mentioned first frame video data), and does not include non-first frame data packets (the above-mentioned non-first frame video data).

[0224] In an exemplary embodiment, the entire process of this application is as follows Figure 7 As shown, the present application solves the problems of uncontrollable first frame optimization and low service experience caused by the unknown first frame playback conditions and high completion time caused by the loss of the tail data of the first frame caused by the third-party application in the existing audio and video services, and proposes an audio and video first frame optimization method based on frame reordering. In the present application, the parsing module of the server (sending end) performs frame parsing on the data to be sent, clarifies the frame type corresponding to the data, and passes these data and the frame type to the sending module; after receiving the above information, the sending module determines whether the current data is the first frame. If so, it obtains the conditions for the first frame playback of the client from the policy control module, and adjusts the sending order of the audio and video frames according to the above conditions, and gives priority to sending the audio and video frames that meet the first frame playback conditions; when the first frame data is all sent out, the sending module of the server performs a "reverse retransmission" operation on the message that has been sent but not yet confirmed, so as to reduce the phenomenon of increased first frame completion time caused by the loss of the tail data of the first frame.

[0225] It is understandable that in the specific implementation of this application, related data such as object information is involved. When the above embodiments of this application are applied to specific products or technologies, it is necessary to obtain the object's permission or consent, and the collection, use and processing of relevant data need to comply with relevant laws, regulations and standards of relevant countries and regions.

[0226] It should be noted that, for the aforementioned method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the described order of actions, because according to the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present application.

[0227] According to another aspect of the embodiment of the present application, a video data processing device for implementing the above-mentioned video data processing method is also provided. Fig.10 As shown, the device comprises:

[0228] The first determination module 1002 is used to determine the video data to be played and the first frame playback condition in response to the video playback request from the target client, wherein the first frame playback condition includes the number of audio frames and the number of video frames required for the first frame playback of the video data to be played on the target client;

[0229] The second determining module 1004 is used to determine the first frame video data and the target sending order according to the video data to be played and the first frame playing condition, wherein the first frame video data includes the target audio frames corresponding to the number of audio frames in the video data to be played and the target video frames corresponding to the number of video frames in the video data to be played, and the target sending order indicates the sending order of the target audio frames and the target video frames;

[0230] The first transmission module 1006 is used to encapsulate the first frame video data into a first group of data packets according to the target sending order, and send the first group of data packets to the target client, wherein, if there is a second group of data packets in the first group of data packets that have been confirmed to be sent to the target client and have not been confirmed to be received by the target client, send the second group of data packets in the reverse order of the target sending order;

[0231] The second transmission module 1008 is used to send non-first frame video data to the above-mentioned target client when the above-mentioned first group of data packets have been confirmed to be sent to the above-mentioned target client and confirmed to be received by the above-mentioned target client, wherein the above-mentioned video data to be played includes the above-mentioned first frame video data and the above-mentioned non-first frame video data.

[0232] As an optional scheme, the above-mentioned device is used to respond to a video playback request from a target client and determine the video data to be played and the first frame playback condition in the following manner: in response to the above-mentioned video playback request, obtain the first frame data information of the above-mentioned target client, wherein the above-mentioned first frame data information includes a first moment and a second moment corresponding to the played video data cached by the above-mentioned target client, the above-mentioned first moment indicates the moment when the above-mentioned target client first receives the data packet corresponding to the above-mentioned played video data, and the above-mentioned second moment indicates the moment when the above-mentioned target client starts to decode and render the above-mentioned played video data; determine the above-mentioned first frame playback condition based on the above-mentioned first frame data information, wherein the above-mentioned first frame data information includes the above-mentioned audio frame number and the above-mentioned video frame number.

[0233] As an optional scheme, the above-mentioned device is used to determine the first frame video data and the target sending order according to the above-mentioned video data to be played and the above-mentioned first frame playback conditions in the following manner: performing a parsing operation on the above-mentioned video data to be played, determining the initial audio frame and the initial video frame in the above-mentioned video data to be played, wherein the above-mentioned initial audio frame and the above-mentioned initial video frame are identified and determined by a specific type field; extracting the above-mentioned target audio frame from the above-mentioned initial audio frame according to the above-mentioned audio frame quantity and the above-mentioned video frame quantity, and extracting the above-mentioned target video frame from the above-mentioned initial video frame, generating the above-mentioned first frame video data, and setting the above-mentioned first frame video data to be sent according to the above-mentioned target sending order.

[0234] As an optional solution, the above-mentioned device is used to extract the above-mentioned target audio frame from the above-mentioned initial audio frame according to the above-mentioned number of audio frames and the above-mentioned number of video frames, and extract the above-mentioned target video frame from the above-mentioned initial video frame, generate the above-mentioned first frame video data, and set the above-mentioned first frame video data to be sent according to the above-mentioned target sending order: when the above-mentioned first frame playback condition includes the first frame cache length of the played video data, extract the first data frame from the above-mentioned video data to be played according to the above-mentioned first frame cache length, wherein the above-mentioned first data frame includes the first video frame or the first audio frame, the above-mentioned first video frame represents the video frame that meets the above-mentioned first frame cache length, the above-mentioned first audio frame represents the audio frame that meets the above-mentioned first frame cache length, and the above-mentioned first frame cache length represents the first frame video length of the above-mentioned played video data; determine the above-mentioned target audio frame, the above-mentioned target video frame and the above-mentioned first data frame as the above-mentioned first frame video data, and set the above-mentioned target audio frame and the above-mentioned target video frame to be sent according to the above-mentioned target sending order, and set the above-mentioned first data frame to be sent according to the first sending order, wherein the above-mentioned first sending order is located after the above-mentioned target sending order.

[0235] As an optional solution, the above-mentioned device is used to extract the above-mentioned target audio frame from the above-mentioned initial audio frame according to the above-mentioned number of audio frames and the above-mentioned number of video frames, and extract the above-mentioned target video frame from the above-mentioned initial video frame to generate the above-mentioned first frame video data and the above-mentioned target sending order in the following manner: when the above-mentioned first frame playback condition includes the first frame cache data size of the played video data, extract the second data frame from the above-mentioned video data to be played according to the above-mentioned first frame cache data size, wherein the above-mentioned second data frame includes the second video frame or the second audio frame, the above-mentioned second video frame represents a video frame that meets the above-mentioned first frame cache data size, the above-mentioned second audio frame represents an audio frame that meets the above-mentioned first frame cache data size, and the above-mentioned first frame cache data size represents the first frame video data size of the above-mentioned played video data; determine the above-mentioned target audio frame, the above-mentioned target video frame and the above-mentioned second data frame as the above-mentioned first frame video data, and set the above-mentioned target audio frame and the above-mentioned target video frame to be sent according to the above-mentioned target sending order, and set the above-mentioned second data frame to be sent according to the second sending order, wherein the above-mentioned second sending order is located after the above-mentioned target sending order.

[0236] As an optional solution, the above-mentioned device is used to extract the above-mentioned target audio frame from the above-mentioned initial audio frame according to the above-mentioned audio frame number and the above-mentioned video frame number, and extract the above-mentioned target video frame from the above-mentioned initial video frame to generate the above-mentioned first frame video data and the above-mentioned target sending order in the following manner: when the above-mentioned first frame playback condition includes the total number of first frame cache frames of the played video data, extract the third data frame from the above-mentioned video data to be played according to the above-mentioned total number of first frame cache frames, wherein the above-mentioned third data frame includes the third video frame or the third audio frame, the above-mentioned third video frame represents the video frame that meets the above-mentioned total number of first frame cache frames, the above-mentioned third audio frame represents the audio frame that meets the above-mentioned total number of first frame cache frames, and the above-mentioned total number of first frame cache frames represents the frame number of the first frame of the above-mentioned played video data; determine the above-mentioned target audio frame, the above-mentioned target video frame and the above-mentioned third data frame as the above-mentioned first frame video data, and set the above-mentioned target audio frame and the above-mentioned target video frame to be sent according to the above-mentioned target sending order, and set the above-mentioned third data frame to be sent according to the third sending order, wherein the above-mentioned third sending order is located after the above-mentioned target sending order.

[0237] As an optional solution, the above-mentioned device is used to encapsulate the above-mentioned first frame video data into a first group of data packets according to the above-mentioned target sending order in the following manner, and send the above-mentioned first group of data packets to the above-mentioned target client: encapsulate the above-mentioned first frame video data into multiple data packets; set a corresponding data packet sequence number for each of the above-mentioned multiple data packets according to the above-mentioned target sending order, wherein the value size of the above-mentioned data packet sequence number is used to indicate the sending order of the corresponding data packets; and send the above-mentioned multiple data packets to the above-mentioned target client in sequence according to the value size of the above-mentioned data packet sequence number.

[0238] As an optional solution, the above-mentioned device is also used to: obtain a first data packet sequence number, wherein the above-mentioned first data packet sequence number is used to identify a data packet that has been sent to the above-mentioned target client; receive a second data packet sequence number returned by the above-mentioned target client, wherein the above-mentioned second data packet sequence number is used to identify a data packet confirmed to be received by the above-mentioned target client; determine the above-mentioned second group of data packets based on the above-mentioned first data packet sequence number and the above-mentioned second data packet sequence number, and send the above-mentioned second group of data packets in the reverse order of the above-mentioned target sending order.

[0239] As an optional scheme, the above-mentioned device is used to determine the above-mentioned second group of data packets according to the above-mentioned first data packet sequence number and the above-mentioned second data packet sequence number in the following manner, and send the above-mentioned second group of data packets in the reverse order of the above-mentioned target sending order: determine the number of data packets currently confirmed to be received by the above-mentioned target client according to the above-mentioned second data packet sequence number; extract retransmission data packets corresponding to the above-mentioned number of data packets from the above-mentioned second group of data packets in the reverse order of the above-mentioned target sending order, send the above-mentioned retransmission data packets to the above-mentioned target client, and delete the above-mentioned retransmission data packets from the above-mentioned second group of data packets.

[0240] As an optional scheme, the above-mentioned device is also used to: respond to a video playback request from a target client, determine the above-mentioned video data to be played, and initialize a target variable for the above-mentioned video data to be played, wherein the above-mentioned target variable is used to indicate whether the above-mentioned first frame of video data has been sent; obtain a third data packet sequence number, wherein the above-mentioned third data packet sequence number is used to identify a data packet that needs to be sent to the above-mentioned target client; when the above-mentioned second data packet sequence number and the above-mentioned third data packet sequence number are different, determine that the above-mentioned first frame of video data has not been sent, send the above-mentioned first group of data packets to the above-mentioned target client in accordance with the above-mentioned target sending order, and send the above-mentioned second group of data packets in reverse order of the above-mentioned target sending order; when the above-mentioned second data packet sequence number and the above-mentioned third data packet sequence number are the same, determine that the above-mentioned first frame of video data has been sent, stop sending the above-mentioned second group of data packets, and send non-first frame video data to the above-mentioned target client.

[0241] As an optional scheme, the above-mentioned device is used to respond to a video playback request from a target client and determine the video data to be played and the first frame playback condition in the following manner: detect the above-mentioned video playback request and determine the above-mentioned video data to be played and the first frame playback condition field; when the above-mentioned first frame playback condition field indicates that the above-mentioned video playback request includes the above-mentioned first frame playback condition, set the above-mentioned first frame playback condition according to the above-mentioned video playback request; when the above-mentioned first frame playback condition field indicates that the above-mentioned video playback request does not include the above-mentioned first frame playback condition, obtain the above-mentioned first frame playback condition which is set in advance.

[0242] In the embodiments of the present application, the term "module" or "unit" refers to a computer program or a part of a computer program with a predetermined function, and works together with other related parts to achieve a predetermined goal, and can be implemented in whole or in part by using software, hardware (such as processing circuits or memories) or a combination thereof. Similarly, a processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be part of an overall module or unit that includes the function of the module or unit.

[0243] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0244] According to one aspect of the present application, a computer program product is provided, the computer program product comprising a computer program / instruction, the computer program / instruction comprising a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through the communication part 1109, and / or installed from a removable medium 1111. When the computer program is executed by the central processor 1101, various functions provided in the embodiments of the present application are executed.

[0245] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0246] Fig.11 The structure block diagram of a computer system for implementing an electronic device according to an embodiment of the present application is schematically shown.

[0247] It should be noted that Fig.11 The computer system 1100 of the electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0248] like Fig.11As shown, the computer system 1100 includes a central processing unit 1101 (CPU), which can perform various appropriate actions and processes according to the program stored in the read-only memory 1102 (ROM) or the program loaded from the storage part 1108 to the random access memory 1103 (RAM). Various programs and data required for system operation are also stored in the random access memory 1103. The central processing unit 1101, the read-only memory 1102 and the random access memory 1103 are connected to each other through a bus 1104. An input / output interface 1105 (Input / Output interface, i.e., I / O interface) is also connected to the bus 1104.

[0249] The following components are connected to the input / output interface 1105: an input section 1106 including a keyboard, a mouse, etc.; an output section 1107 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 1108 including a hard disk, etc.; and a communication section 1109 including a network interface card such as a LAN card, a modem, etc. The communication section 1109 performs communication processing via a network such as the Internet. A drive 1110 is also connected to the input / output interface 1105 as needed. A removable medium 1111, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 1110 as needed so that a computer program read therefrom is installed into the storage section 1108 as needed.

[0250] In particular, according to an embodiment of the present application, the process described in each method flow chart can be implemented as a computer software program. For example, an embodiment of the present application includes a computer program product, which includes a computer program carried on a computer readable medium, and the computer program contains a program code for executing the method shown in the flow chart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 1109, and / or installed from the removable medium 1111. When the computer program is executed by the central processor 1101, various functions defined in the system of the present application are executed.

[0251] According to another aspect of the embodiment of the present application, an electronic device for implementing the above-mentioned video data processing method is also provided. The electronic device may be Figure 1 The terminal device or server shown in the figure. This embodiment is described by taking the electronic device as a terminal device as an example. Fig.12As shown, the electronic device includes a memory 1202 and a processor 1204. The memory 1202 stores a computer program. The processor 1104 is configured to execute the steps in any of the above method embodiments through the computer program.

[0252] Optionally, in this embodiment, the electronic device may be located in at least one network device among a plurality of network devices of a computer network.

[0253] Optionally, in this embodiment, the processor may be configured to perform the following steps through a computer program:

[0254] S1, in response to a video playback request from a target client, determining the video data to be played and the first frame playback condition, wherein the first frame playback condition includes the number of audio frames and the number of video frames required for the video data to be played to complete the first frame playback on the target client;

[0255] S2, determining the first frame video data and the target sending order according to the video data to be played and the first frame playing condition, wherein the first frame video data includes the target audio frames corresponding to the number of audio frames in the video data to be played and the target video frames corresponding to the number of video frames in the video data to be played, and the target sending order indicates the sending order of the target audio frames and the target video frames;

[0256] S3, encapsulating the first frame of video data into a first group of data packets according to the target sending order, and sending the first group of data packets to the target client, wherein, if there is a second group of data packets in the first group of data packets that have been confirmed to be sent to the target client and have not been confirmed to be received by the target client, sending the second group of data packets in reverse order of the target sending order;

[0257] S4, when the first group of data packets have been confirmed to be sent to the target client and confirmed to be received by the target client, the non-first frame video data is sent to the target client, wherein the video data to be played includes the first frame video data and the non-first frame video data.

[0258] Alternatively, a person skilled in the art may understand that: Fig.12 The structure shown is for illustration only, and the electronic device may also be a smart phone (such as an Android phone, an iOS phone, etc.), a tablet computer, a PDA, a mobile Internet device (MID), a PAD, and other terminal devices. Fig.12 The electronic device and the electronic equipment described above are not limited in structure. Fig.12 More or fewer components (such as network interfaces, etc.) as shown in, or with Fig.12 Different configurations shown.

[0259] Among them, the memory 1202 can be used to store software programs and modules, such as the program instructions / modules corresponding to the method and device for processing video data in the embodiments of the present application. The processor 1104 executes various functional applications and data processing by running the software programs and modules stored in the memory 1202, that is, realizing the above-mentioned method for processing video data. The memory 1102 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 1202 may further include a memory remotely located relative to the processor 1204, and these remote memories may be connected to the terminal via a network. Examples of the above-mentioned networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof. Among them, the memory 1202 can be specifically, but not limited to, used for information such as video data to be played. As an example, such as Fig.10 As shown, the memory 1202 may include, but is not limited to, the first determination module 1002, the second determination module 1004, the first transmission module 1006, and the second transmission module 1008 in the video data processing device. In addition, other module units in the video data processing device may also be included but are not limited to, which will not be repeated in this example.

[0260] Optionally, the transmission device 1206 is used to receive or send data via a network. Specific examples of the network may include a wired network and a wireless network. In one example, the transmission device 1206 includes a network adapter (Network Interface Controller, NIC), which can be connected to other network devices and routers via a network cable so as to communicate with the Internet or a local area network. In one example, the transmission device 1206 is a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0261] In addition, the electronic device further includes: a display 1208 for displaying the video data to be played; and a connection bus 1210 for connecting various module components in the electronic device.

[0262] In other embodiments, the terminal device or server may be a node in a distributed system, wherein the distributed system may be a blockchain system, and the blockchain system may be a distributed system formed by connecting the multiple nodes through network communication. The nodes may form a peer-to-peer network, and any form of computing device, such as a server, terminal or other electronic device, may become a node in the blockchain system by joining the peer-to-peer network.

[0263] According to one aspect of the present application, a computer-readable storage medium is provided, and a processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the video data processing method provided in various optional implementations of the above-mentioned video data processing.

[0264] Optionally, in this embodiment, the computer-readable storage medium may be configured to store a computer program for performing the following steps:

[0265] S1, in response to a video playback request from a target client, determining the video data to be played and the first frame playback condition, wherein the first frame playback condition includes the number of audio frames and the number of video frames required for the video data to be played to complete the first frame playback on the target client;

[0266] S2, determining the first frame video data and the target sending order according to the video data to be played and the first frame playing condition, wherein the first frame video data includes the target audio frames corresponding to the number of audio frames in the video data to be played and the target video frames corresponding to the number of video frames in the video data to be played, and the target sending order indicates the sending order of the target audio frames and the target video frames;

[0267] S3, encapsulating the first frame of video data into a first group of data packets according to the target sending order, and sending the first group of data packets to the target client, wherein, if there is a second group of data packets in the first group of data packets that have been confirmed to be sent to the target client and have not been confirmed to be received by the target client, sending the second group of data packets in reverse order of the target sending order;

[0268] S4, when the first group of data packets have been confirmed to be sent to the target client and confirmed to be received by the target client, the non-first frame video data is sent to the target client, wherein the video data to be played includes the first frame video data and the non-first frame video data.

[0269] Optionally, in this embodiment, a person of ordinary skill in the art may understand that all or part of the steps in the various methods of the above embodiments may be completed by instructing hardware related to the terminal device through a program, and the program may be stored in a computer-readable storage medium, and the storage medium may include: a flash drive, a read-only memory (ROM), a random access memory (RAM), a disk or an optical disk, etc.

[0270] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0271] If the integrated units in the above embodiments are implemented in the form of software functional units and sold or used as independent products, they can be stored in the above computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for enabling one or more computer devices (which may be personal computers, servers, or network devices, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present application.

[0272] In the above embodiments of the present application, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.

[0273] In the several embodiments provided in the present application, it should be understood that the disclosed client can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0274] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0275] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0276] The above is only a preferred implementation of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A method for processing video data, characterized in that: include: In response to a video playback request from a target client, determine the video data to be played and a first frame playback condition, wherein the first frame playback condition includes the number of audio frames and the number of video frames required for the video data to be played to complete the first frame playback on the target client; Determining the first frame video data and the target sending order according to the video data to be played and the first frame playing condition, wherein the first frame video data includes the target audio frames corresponding to the number of audio frames in the video data to be played and the target video frames corresponding to the number of video frames in the video data to be played, and the target sending order indicates the sending order of the target audio frames and the target video frames; encapsulating the first frame of video data into a first group of data packets according to the target sending order, and sending the first group of data packets to the target client, wherein, if there is a second group of data packets in the first group of data packets that have been confirmed to be sent to the target client and have not yet been confirmed to be received by the target client, sending the second group of data packets in reverse order of the target sending order; When the first group of data packets have been confirmed to be sent to the target client and confirmed to be received by the target client, non-first frame video data is sent to the target client, wherein the video data to be played includes the first frame video data and the non-first frame video data.

2. The method according to claim 1, characterized in that The step of determining the video data to be played and the first frame playing condition in response to the video playing request from the target client includes: In response to the video playback request, first frame data information of the target client is obtained, wherein the first frame data information includes a first time and a second time corresponding to the played video data cached by the target client, the first time indicating the time when the target client first receives a data packet corresponding to the played video data, and the second time indicating the time when the target client starts decoding and rendering the played video data; The first frame playback condition is determined according to the first frame data information, wherein the first frame data information includes the number of audio frames and the number of video frames.

3. The method according to claim 1, characterized in that The determining the first frame video data and the target sending order according to the video data to be played and the first frame playing condition includes: Performing a parsing operation on the video data to be played, determining an initial audio frame and an initial video frame in the video data to be played, wherein the initial audio frame and the initial video frame are identified and determined by a specific type field; The target audio frame is extracted from the initial audio frame according to the number of audio frames and the number of video frames, and the target video frame is extracted from the initial video frame, the first frame video data is generated, and the first frame video data is set to be sent in the target sending order.

4. The method according to claim 3, characterized in that The step of extracting the target audio frame from the initial audio frame according to the number of audio frames and the number of video frames, extracting the target video frame from the initial video frame, generating the first frame of video data, and setting the first frame of video data to be sent in the target sending order includes: In a case where the first frame playback condition includes a first frame cache duration of the played video data, extracting a first data frame from the to-be-played video data according to the first frame cache duration, wherein the first data frame includes a first video frame or a first audio frame, the first video frame represents a video frame that satisfies the first frame cache duration, the first audio frame represents an audio frame that satisfies the first frame cache duration, and the first frame cache duration represents a first frame video duration of the played video data; The target audio frame, the target video frame and the first data frame are determined as the first frame of video data, and the target audio frame and the target video frame are set to be sent in the target sending order, and the first data frame is set to be sent in a first sending order, wherein the first sending order is located after the target sending order.

5. The method according to claim 3, characterized in that: The step of extracting the target audio frame from the initial audio frame according to the number of audio frames and the number of video frames, and extracting the target video frame from the initial video frame, and generating the first frame of video data and the target sending order includes: In a case where the first frame playback condition includes a first frame cache data size of the played video data, extracting a second data frame from the to-be-played video data according to the first frame cache data size, wherein the second data frame includes a second video frame or a second audio frame, the second video frame represents a video frame that satisfies the first frame cache data size, the second audio frame represents an audio frame that satisfies the first frame cache data size, and the first frame cache data size represents a first frame video data size of the played video data; The target audio frame, the target video frame and the second data frame are determined as the first frame of video data, and the target audio frame and the target video frame are set to be sent in the target sending order, and the second data frame is set to be sent in a second sending order, wherein the second sending order is located after the target sending order.

6. The method according to claim 3, characterized in that The step of extracting the target audio frame from the initial audio frame according to the number of audio frames and the number of video frames, and extracting the target video frame from the initial video frame, and generating the first frame of video data and the target sending order includes: In a case where the first frame playback condition includes the total number of first frame cache frames of the played video data, extracting a third data frame from the to-be-played video data according to the total number of first frame cache frames, wherein the third data frame includes a third video frame or a third audio frame, the third video frame represents a video frame that satisfies the total number of first frame cache frames, the third audio frame represents an audio frame that satisfies the total number of first frame cache frames, and the total number of first frame cache frames represents the number of frames of the first frame of the played video data; The target audio frame, the target video frame and the third data frame are determined as the first frame of video data, and the target audio frame and the target video frame are set to be sent in the target sending order, and the third data frame is set to be sent in a third sending order, wherein the third sending order is located after the target sending order.

7. The method according to claim 1, characterized in that The step of encapsulating the first frame of video data into a first group of data packets according to the target sending order, and sending the first group of data packets to the target client includes: Encapsulating the first frame of video data into multiple data packets; Setting a corresponding data packet sequence number for each of the multiple data packets according to the target sending order, wherein the value of the data packet sequence number is used to indicate the sending order of the corresponding data packet; The multiple data packets are sent to the target client in sequence according to the values ​​of the data packet sequence numbers.

8. The method according to claim 7, characterized in that The method further comprises: Obtaining a first data packet sequence number, wherein the first data packet sequence number is used to identify a data packet that has been sent to the target client; Receiving a second data packet sequence number returned by the target client, wherein the second data packet sequence number is used to identify a data packet confirmed to be received by the target client; The second group of data packets is determined according to the first data packet sequence number and the second data packet sequence number, and the second group of data packets is sent in reverse order of the target sending order.

9. The method according to claim 8, characterized in that The determining the second group of data packets according to the first data packet sequence number and the second data packet sequence number, and sending the second group of data packets in the reverse order of the target sending order, comprises: Determine the number of data packets currently confirmed to be received by the target client according to the second data packet sequence number; Retransmitted data packets corresponding to the number of data packets are extracted from the second group of data packets in reverse order of the target sending order, the retransmitted data packets are sent to the target client, and the retransmitted data packets are deleted from the second group of data packets.

10. The method according to claim 8, characterized in that The method further comprises: In response to a video playback request from a target client, determining the video data to be played, and initializing a target variable for the video data to be played, wherein the target variable is used to indicate whether the first frame of video data has been sent; Obtaining a third data packet sequence number, wherein the third data packet sequence number is used to identify a data packet that needs to be sent to the target client; When the second data packet sequence number is different from the third data packet sequence number, determining that the first frame of video data has not been completely sent, sending the first group of data packets to the target client in the target sending order, and sending the second group of data packets in reverse order of the target sending order; When the second data packet sequence number is the same as the third data packet sequence number, it is determined that the first frame of video data has been sent, the sending of the second group of data packets is stopped, and non-first frame video data is sent to the target client.

11. The method according to any one of claims 1 to 10, characterized in that The step of determining the video data to be played and the first frame playing condition in response to the video playing request from the target client includes: Detecting the video play request, and determining the video data to be played and the first frame play condition field; In a case where the first frame playback condition field indicates that the video playback request includes the first frame playback condition, setting the first frame playback condition according to the video playback request; When the first frame playback condition field indicates that the video playback request does not include the first frame playback condition, the preset first frame playback condition is acquired.

12. A video data processing device, characterized in that: include: A first determination module is used to determine the video data to be played and the first frame playback condition in response to a video playback request from a target client, wherein the first frame playback condition includes the number of audio frames and the number of video frames required for the video data to be played to complete the first frame playback on the target client; A second determination module is used to determine the first frame video data and the target sending order according to the video data to be played and the first frame playing condition, wherein the first frame video data includes the target audio frames corresponding to the number of audio frames in the video data to be played and the target video frames corresponding to the number of video frames in the video data to be played, and the target sending order indicates the sending order of the target audio frames and the target video frames; A first transmission module is used to encapsulate the first frame of video data into a first group of data packets according to the target sending order, and send the first group of data packets to the target client, wherein, if there is a second group of data packets in the first group of data packets that have been confirmed to be sent to the target client and have not been confirmed to be received by the target client, the second group of data packets are sent in reverse order of the target sending order; The second transmission module is used to send non-first frame video data to the target client when the first group of data packets have been confirmed to be sent to the target client and confirmed to be received by the target client, wherein the video data to be played includes the first frame video data and the non-first frame video data.

13. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored program, wherein the program can be executed by a terminal device or a computer to execute the method described in any one of claims 1 to 11.

14. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the steps of the method described in any one of claims 1 to 11 are implemented.

15. An electronic device comprising a memory and a processor, characterized in that: A computer program is stored in the memory, and the processor is configured to execute the method according to any one of claims 1 to 11 through the computer program.

Images