Video stream processing method and device and computer equipment

Abstract

The invention provides a video stream processing method and device and computer equipment, and belongs to the technical field of data processing. The method comprises the following steps: acquiring current collision information of each video stream; determining a to-be-adjusted target video stream according to the current collision information of each video stream and the frame feature variation of each video stream; according to the identifier of the to-be-adjusted target video stream, extracting an initial real-time transmission message sequence corresponding to the to-be-adjusted target video stream from a client cache region; and extracting basic video features of the target video stream to be adjusted from the initial real-time transmission message sequence, constructing a new real-time transmission message sequence according to the basic video features, and adding the new real-time transmission message sequence to a client cache region. According to the invention, the network congestion caused by the collision of the key frames in the video stream can be effectively solved, so that the user experience is improved.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a video stream processing method, apparatus and computer equipment. Background Art

[0002] As a way of transmitting video data, video streaming can make the video be processed as a stable and continuous stream through the network. Video streaming technology allows users to watch video content instantly without downloading the complete video file. Video streaming technology is widely used in video websites, instant video conferencing, interactive games, live broadcast services and other fields.

[0003] In the related art, video data of a video source is collected and encoded by an encoder to obtain encoded video data, the encoded video data is packaged to obtain a real-time transmission (Real-Time-Transport Protocol, RTP) message of the video data, and the RTP message is further encapsulated into a network layer (Internet Protocol, IP) message, and the IP message is sent to the client through the network. The client reassembles and unpacks the IP message to extract the RTP message of the video data, and sends the RTP message to the client decoder to restore the video content and play it on the client display interface.

[0004] However, when processing video streams based on related technologies, when multiple videos are concurrent, I-frame data from different video streams are prone to collision, resulting in a significant increase in the network bandwidth required for video stream service processing, which in turn leads to network congestion, data packet loss and other failures. Therefore, the solutions of related technologies have the problem that multiple video streams concurrently are prone to I-frame data collision, causing network congestion and poor user experience. Summary of the invention

[0005] The purpose of the present application is to provide a video stream processing method, apparatus and computer equipment, which can effectively solve the network congestion caused by key frame collision in the video stream, so as to improve the user experience.

[0006] The embodiment of the present application is implemented as follows:

[0007] According to a first aspect of an embodiment of the present application, a video stream processing method is provided, the method comprising:

[0008] Get the current collision information of each video stream;

[0009] Determine the target video stream to be adjusted according to the current collision information of each video stream and the frame feature change amount of each video stream;

[0010] According to the identifier of the target video stream to be adjusted, extracting the initial real-time transmission message sequence corresponding to the target video stream to be adjusted from the client buffer area;

[0011] The basic video features of the target video stream to be adjusted are extracted from the initial real-time transmission message sequence, a new real-time transmission message sequence is constructed according to the basic video features, and the new real-time transmission message sequence is added to the client buffer area.

[0012] As a possible implementation, according to the identifier of the target video stream to be adjusted, extracting the initial real-time transmission message sequence corresponding to the target video stream to be adjusted from the client buffer area includes:

[0013] Traverse each real-time transmission message in the client buffer area. For the current real-time transmission message traversed, if the video stream identifier corresponding to the current real-time transmission message is the identifier of the target video stream to be adjusted and the current real-time transmission message belongs to a real-time transmission message of a preset type, then add the current real-time transmission message to the initial real-time transmission message sequence.

[0014] As a possible implementation method, basic video features of the target video stream to be adjusted are extracted from the initial real-time transmission message sequence, and a new real-time transmission message sequence is constructed according to the basic video features, including:

[0015] Parsing the initial real-time transmission message sequence to obtain the number of messages in the initial real-time transmission message sequence, time information of each message, and a sequence number of a reference message, where the reference message is the first message or the last message in the initial real-time transmission message sequence;

[0016] Obtaining the message payload of each message in the initial real-time transmission message sequence;

[0017] The number of messages in the initial real-time transmission message sequence, the time information of each message, the sequence number of the reference message and the message payload of each message are used as the basic video features of the target video stream to be adjusted;

[0018] A new real-time transmission message sequence is constructed according to the basic video features, wherein the number of messages in the new real-time transmission message sequence is the same as the number of messages in the initial real-time transmission message sequence.

[0019] As a possible implementation method, a new real-time transmission message sequence is constructed according to basic video features, including:

[0020] Generate a sequence number of each message in a new real-time transmission message sequence according to the number of messages in the initial real-time transmission message sequence and the sequence number of the reference message;

[0021] Generate the time information of each message in a new real-time transmission message sequence according to the time information of each message;

[0022] Generate a message header of each message in a new real-time transmission message sequence according to a message header of each message in the initial real-time transmission message sequence, a sequence number of each message in the new real-time transmission message sequence, and time information of each message in the new real-time transmission message sequence;

[0023] The message payload of each message in a new real-time transmission message sequence is generated according to the message payload of each message.

[0024] As a possible implementation manner, generating a sequence number of each message in a new real-time transmission message sequence according to the number of messages in the initial real-time transmission message sequence and the sequence number of the reference message includes:

[0025] If the reference message is the first message, add the sequence number of the first message and the number of messages;

[0026] The sum of the added sum and i-1 is used as the sequence number of the i-th message in the new real-time transmission message sequence, where i is greater than or equal to 1, i is less than or equal to the number of messages, and i is an integer.

[0027] As a possible implementation method, a new real-time transmission message sequence is added to the client buffer, including:

[0028] It is determined whether there is a current real-time transmission message of the target video stream to be adjusted in the client buffer area. If not, a new real-time transmission message sequence is added to the client buffer area.

[0029] As a possible implementation method, according to the current collision information of each video stream and the frame feature change amount of each video stream, determining the target video stream to be adjusted includes:

[0030] Determine the target video stream to be adjusted according to the collision type and collision number of each video stream;

[0031] Or, extract multiple frame features of each video stream, and determine the target video stream to be adjusted according to the change amount of the target frame features of each video stream within a preset time period;

[0032] Or, the target video stream to be adjusted is determined according to the number of collisions of each video stream.

[0033] As a possible implementation method, the current collision information of each video stream is obtained, including:

[0034] Numbering each detected video stream to obtain an identification of each video stream;

[0035] According to the identifier of each video stream, respectively record the video information of each video stream, the video information includes: at least one key frame information, the key frame information includes: at least one of the key frame start time, the key frame end time, the number of data packets in the key frame and the size of each data packet;

[0036] According to the video information of each video stream, the current collision information of each video stream is recorded.

[0037] According to a second aspect of an embodiment of the present application, a video stream processing device is provided, the device comprising:

[0038] An acquisition module is used to obtain the current collision information of each video stream;

[0039] A determination module, used to determine the target video stream to be adjusted according to the current collision information of each video stream and the frame feature change amount of each video stream;

[0040] An extraction module, used for extracting an initial real-time transmission message sequence corresponding to the target video stream to be adjusted from a client buffer area according to an identifier of the target video stream to be adjusted;

[0041] The construction module is used to extract the basic video features of the target video stream to be adjusted from the initial real-time transmission message sequence, construct a new real-time transmission message sequence according to the basic video features, and add the new real-time transmission message sequence to the client buffer area.

[0042] According to a third aspect of an embodiment of the present application, a computer device is provided, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the computer program is executed by the processor, the video stream processing method described in the first aspect is implemented.

[0043] According to a fourth aspect of the embodiments of the present application, a computer-readable storage medium is provided, which stores a computer program. When the computer program is executed by a processor, the video stream processing method described in the first aspect is implemented.

[0044] The beneficial effects of the embodiments of the present application include:

[0045] A video stream processing method provided by an embodiment of the present application obtains current collision information of each video stream and extracts frame feature changes of consecutive frames of each video stream; selects a target video stream to be adjusted from multiple video streams in which a key frame collision occurs at the current moment according to the current collision information and frame feature changes of each video stream; extracts an initial real-time transmission message sequence corresponding to the target video stream to be adjusted from a client cache at the current moment according to an identifier of the target video stream to be adjusted; and extracts basic video information of the target video stream to be adjusted from the initial real-time transmission message sequence, constructs a new real-time transmission message sequence corresponding to the target video stream to be adjusted according to the basic video information of the target video stream to be adjusted, and adds the newly constructed real-time transmission message sequence to the client cache. Among them, when multiple video streams are concurrent at the same time and the key frames in the multiple video streams collide and cause network congestion, the real-time transmission message that is currently colliding can be discarded, and a new real-time transmission message sequence is constructed based on the basic video information of the video stream that discarded the real-time transmission message. During the packet loss period, the newly constructed real-time transmission message sequence is sent to the client decoder for decoding, which can solve the problem of network congestion caused by key frame collisions and ensure that there is no blank frame phenomenon on the user side during the packet loss period, thereby ensuring the continuity of the real-time video. In this way, the network congestion caused by the collision of key frames in the video stream can be effectively solved to improve the user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0047] Figure 1 A logic block diagram of an existing video data processing;

[0048] Figure 2 A flowchart of a first video stream processing method provided in an embodiment of the present application;

[0049] Figure 3 The present invention is a flow chart of message encapsulation of an existing video stream;

[0050] Figure 4 A schematic diagram of the structure of a message header of an existing NALU message;

[0051] Figure 5 A flowchart of a second video stream processing method provided in an embodiment of the present application;

[0052] Figure 6A flowchart of a third video stream processing method provided in an embodiment of the present application;

[0053] Figure 7 It is a structural schematic diagram of an existing real-time transmission message;

[0054] Figure 8 A schematic diagram of the structure of a message header of an existing real-time transmission message;

[0055] Fig. 9 A flowchart of a fourth video stream processing method provided in an embodiment of the present application;

[0056] Fig.10 A flowchart of a fifth video stream processing method provided in an embodiment of the present application;

[0057] Fig.11 A schematic diagram of the structure of a video stream processing device provided in an embodiment of the present application;

[0058] Fig.12 A schematic diagram of the structure of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0059] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. The components of the embodiments of the present application described and shown in the drawings here can be arranged and designed in various different configurations.

[0060] Therefore, the following detailed description of the embodiments of the present application provided in the accompanying drawings is not intended to limit the scope of the present application for which protection is sought, but merely represents selected embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without creative work are within the scope of protection of the present application.

[0061] At present, video data processing usually involves first collecting video data from a video source, encoding the collected video data through an encoder, and packaging the encoded video data to obtain RTP packets corresponding to each video data, further encapsulating the RTP packets into IP packets, and sending the IP packets to the client via the Internet. The client reassembles and unpacks the IP packets to extract the RTP packets of the video data, and inputs the extracted RTP packets into the client decoder to restore the video content. However, in this solution, when multiple videos are concurrent, I frame data from different video streams are prone to collision, resulting in a significant increase in the network bandwidth required for video stream service processing, which in turn leads to network congestion, data packet loss and other failures, which will cause a poor user experience for users.

[0062] To this end, the embodiment of the present application provides a video stream processing method, by obtaining the current collision information of each video stream, and according to the current collision information of each video stream and the frame feature variation of each video stream, determine the target video stream to be adjusted; according to the identifier of the target video stream to be adjusted, extract the initial real-time transmission message sequence corresponding to the target video stream to be adjusted from the client buffer area; extract the basic video features of the target video stream to be adjusted from the initial real-time transmission message sequence corresponding to the target video stream to be adjusted, and construct a new real-time transmission message sequence according to the basic video features, and if the new real-time transmission message sequence corresponding to the target video stream to be adjusted is not received at the current moment, the newly constructed real-time transmission message sequence is cached to the client buffer area, which can alleviate the video interruption caused by the packet loss of the target video stream to be adjusted. In addition, when there is an I frame collision, the real-time transmission message that collides can be discarded to reduce the network bandwidth requirement. In this way, it can effectively solve the network congestion caused by the collision of key frames in the video stream to improve the user experience.

[0063] Figure 1 For a logic block diagram of an existing video data processing, see Figure 1 The specific process of the existing video data processing flow includes: first downloading video data from a video source, then encoding the video data through an encoder to obtain an RTP message corresponding to the video data, and encapsulating the RTP message of the video data to obtain an IP message of the video data; sending the IP message of the video data to a client via the Internet, the client unpacks the received IP message to obtain the RTP message of the video data, caches the RTP message in a client cache area, and decodes the RTP message through a client decoder to obtain the source video data and display it on a display interface of the client.

[0064] It is worth noting that after the video data is encoded by the encoder, I frames and P frames of the video data can be obtained, among which I frame refers to the intra-frame coded frame of the video data, which is a fully compressed coded frame. The I frame can perform JPEG compression encoding and transportation of the image information of the entire frame. By decoding the I frame, a complete image can be constructed. The I frame is used to describe the detailed information of the image background and moving subject, and does not need to refer to other pictures to generate. The I frame has a certain impact on the quality of subsequent frames; P frame refers to a forward predictive coded frame, and the I frame is a reference frame of the P frame. The predicted value and motion vector of a certain point in the P frame can be found in the I frame.

[0065] There is often at least one P frame after an I frame, and they are sent periodically in this way. Multiple frames between two I frames together form a frame group. Both I frames and P frames are composed of multiple consecutive IP data packets, and the frame size of an I frame is much larger than that of a P frame.

[0066] In addition, I frames are divided into ordinary I frames and IDR frames. IDR frames carry SPS sequence parameter sets and PPS image parameter sets. The role of IDR frames is to enable the decoder to refresh data information immediately to avoid major encoding errors. When the client decoder decodes an IDR frame, it will immediately clear the reference queue in the client buffer, output or discard all decoded video data, and search for a new parameter set again, and start decoding the next new data sequence.

[0067] The video stream processing method provided in the embodiment of the present application is explained in detail below with reference to the accompanying drawings.

[0068] It is worth noting that the real-time transmission message is used to indicate the RTP message, and the video stream is mainly used to indicate the video data containing I frames and P frames. In this application, the video stream collision is mainly for the I frame data in the video stream, and the key frame is used to indicate the I frame. The real-time transmission message is replaced by the RTP message, and the key frame is replaced by the I frame. It will not be repeated below.

[0069] Figure 2 This is a flowchart of a video stream processing method provided by the present application. The method can be applied to a computer device, a server or a third-party terminal device, and the present application does not make any specific restrictions on this. Figure 2 , the embodiment of the present application provides a video stream processing method, including:

[0070] S201. Obtain current collision information of each video stream.

[0071] Optionally, the collision of video streams refers to the phenomenon that I frame data in two or more video streams appear at the same time, resulting in packet loss failure in the video stream. When the client receives I frame data of two or more video streams at the same time, the data transmission bandwidth of the client will increase significantly, and the transmission bandwidth required by the client is approximately equal to the sum of the transmission rates of the video streams involved in the collision. This application does not make specific limitations on this.

[0072] Optionally, the current collision information of each video stream is used to indicate the information generated by the I frame collision in each video stream when the video stream is transmitted to the client. The current collision information of each video stream includes: collision type, collision duration, collision start time, collision end time, identifier of the collision combination to which it belongs, identifier of the video streams involved in the same I frame collision, etc. This application does not make any specific limitations on this.

[0073] Optionally, the collision type is used to indicate the type of collision combination of I frame collisions in each video stream. The collision type of the video stream may be a five-frame collision, a four-frame collision, a three-frame collision, a two-frame collision, etc.; the number of collisions refers to the number of times that the I frame data in each video stream collides when it is transmitted to the client. The number of collisions of the video stream may be 1, 3, 4, 5, etc., and this application does not make any specific limitation on this.

[0074] Optionally, the collision duration refers to the length of time that the I frames in two or more video streams collide, the collision start time refers to the time point when the I frames in two or more video streams collide, the collision end time refers to the time point when the I frame data in two or more video streams end the collision, the collision combination refers to the combination of two or more video streams participating in the same I frame collision, the collision combination identifier is used to distinguish collision combinations with different collisions occurring at the same time, and the collision combination includes the identifiers of each video stream participating in different I frame collisions.

[0075] It is worth noting that the client is provided with an observation window, and the observation window detects collision information of each video stream in the current observation window according to the identifier of each video stream.

[0076] S202: Determine the target video stream to be adjusted according to the current collision information of each video stream and the frame feature variation of each video stream.

[0077] Optionally, the frame feature variation refers to the feature difference between consecutive frames in each video stream. The frame features include: pixel intensity, texture, edge, shape, corner point, etc., which is not specifically limited in this application.

[0078] Optionally, the target video stream to be adjusted refers to the video stream in which a new RTP message sequence is to be constructed in the video stream where an I frame collision occurs in the current observation window. The target video stream to be adjusted can be selected based on the current collision information of each video stream and the change in frame features of consecutive frames.

[0079] It is worth noting that, in the current observation window, the video streams with I frame collisions other than the target video stream to be adjusted do not undergo RTP message reconstruction. In addition, the target video streams to be adjusted for a single RTP message reconstruction may be one or more, and this application does not make specific restrictions on this.

[0080] S203: extracting an initial real-time transmission message sequence corresponding to the target video stream to be adjusted from the client buffer area according to the identifier of the target video stream to be adjusted.

[0081] Optionally, the identifier of the target video stream to be adjusted refers to the number of the video stream that needs to reconstruct the RTP message sequence, the client buffer area refers to the cache space for caching RTP messages on the client side, and the initial real-time transmission message sequence refers to the RTP message sequence containing IDR frames corresponding to the target video stream to be adjusted cached in the client buffer area at the current moment. The initial real-time transmission message sequence can also be identified as the RTP message sequence containing IDR frames corresponding to the target video stream to be adjusted that is about to be input into the client decoder.

[0082] Optionally, based on the identifier of the target video stream to be adjusted, an RTP message sequence matching the target video stream to be adjusted is extracted from multiple RTP messages cached in the client buffer at the current moment, and an RTP message sequence containing IDR frames is filtered from the RTP message sequence extracted initially to obtain an initial RTP message sequence corresponding to the target video stream to be adjusted.

[0083] S204: extract basic video features of the target video stream to be adjusted from the initial real-time transmission message sequence, construct a new real-time transmission message sequence according to the basic video features, and add the new real-time transmission message sequence to the client buffer area.

[0084] Optionally, the basic video features refer to the basic information required to construct a new RTP message sequence corresponding to the target video stream to be adjusted, the basic video features of the target video stream to be adjusted are extracted from the target video stream to be adjusted, and a new RTP message sequence corresponding to the target video stream to be adjusted is constructed according to the basic video features, and the newly created RTP message sequence is cached in the client cache area.

[0085] In an embodiment of the present application, the current collision information of each video stream is obtained, and the frame feature changes of consecutive frames of each video stream are extracted; based on the current collision information and frame feature changes of each video stream, a target video stream to be adjusted is selected from multiple video streams where a key frame collision occurs at the current moment; based on the identifier of the target video stream to be adjusted, an initial real-time transmission message sequence corresponding to the target video stream to be adjusted is extracted from the client cache at the current moment; and basic video information of the target video stream to be adjusted is extracted from the initial real-time transmission message sequence, and a new real-time transmission message sequence corresponding to the target video stream to be adjusted is constructed based on the basic video information of the target video stream to be adjusted, and the newly constructed real-time transmission message sequence is added to the client cache. Among them, when multiple video streams are concurrent at the same time and the key frames in the multiple video streams collide and cause network congestion, the real-time transmission message that is currently colliding can be discarded, and a new real-time transmission message sequence is constructed based on the basic video information of the video stream that discarded the real-time transmission message. During the packet loss period, the newly constructed real-time transmission message sequence is sent to the client decoder for decoding, which can solve the problem of network congestion caused by key frame collisions and ensure that there is no blank frame phenomenon on the user side during the packet loss period, thereby ensuring the continuity of the real-time video. In this way, the network congestion caused by the collision of key frames in the video stream can be effectively solved to improve the user experience.

[0086] Figure 3 This is a flow chart of an existing video stream packet encapsulation process. Figure 3 , take an H264 video stream as an example, the H264 video stream is I frame video data, when the H264 video stream is processed and encapsulated into a NALU message, the NALU message is mainly composed of a message header NALU Header and an RBSP message. Among them, the RBSP message can be understood as the payload of the NALU message.

[0087] Figure 4 This is a schematic diagram of the structure of a message header of an existing NALU message, see Figure 4 The message header of the NALU message is composed of the F bit, the NRI bit and the Type bit. The F bit refers to the prohibition bit mark, the NRI bit refers to the reference bit mark, and the Type bit refers to the type bit mark.

[0088] Among them, the value range of the NRI bit is 0 to 3, and the value range of the Type bit is 0 to 31. When the value of the NRI bit is 3 and the value of Type is 5, it means that the video frame carried by the NALU message is an IDR frame. The NALU message will be further packaged into an RTP message, and the NALU message will be transmitted as the payload content of the RTP message.

[0089] In a possible implementation, the operation of step S203 may specifically be:

[0090] Traverse each real-time transmission message in the client buffer area. For the current real-time transmission message traversed, if the video stream identifier corresponding to the current real-time transmission message is the identifier of the target video stream to be adjusted and the current real-time transmission message belongs to a real-time transmission message of a preset type, then add the current real-time transmission message to the initial real-time transmission message sequence.

[0091] Optionally, the RTP messages cached in the client buffer area are traversed to determine the homologous identifier of each RTP message and the load content of each RTP message, and the identifier of the video stream to which each RTP message belongs is determined based on the homologous identifier of each RTP message. The type of each RTP message can be determined based on the load content of each RTP message, that is, whether the RTP message is an ordinary I frame data packet or a flag IDR frame data packet.

[0092] Optionally, the preset type of real-time transmission message refers to an RTP message containing an IDR frame. Based on the identifier of the video stream to which each RTP message belongs and the identifier of the target video stream to be adjusted, it is determined whether each RTP message belongs to the target video stream to be adjusted. It is also necessary to determine whether the RTP message contains an IDR frame based on the value of the NRI bit and the value of the Type bit in the load content of each RTP message.

[0093] Optionally, if the homology identifier of the currently traversed RTP message matches the identifier of the target video stream to be adjusted, and the value of the NRI bit in the payload content of the currently traversed RTP message is 3, and the value of Type is 5, then it is determined that the RTP message belongs to the initial real-time transmission message of the target video stream to be adjusted, that is, the currently traversed RTP message belongs to the target video stream to be adjusted, and the currently traversed RTP message also includes an IDR frame. Only then can the currently traversed RTP message be used as a reference message of the target video stream to be adjusted, and the currently traversed RTP message is added to the initial real-time transmission message sequence corresponding to the target video stream to be adjusted.

[0094] It is worth noting that the RTP messages cached in the client buffer are the RTP messages that are about to be input into the client decoder for decoding. When the RTP messages are input into the client decoder for decoding, the client buffer will automatically clear the decoded and displayed RTP messages.

[0095] For a possible implementation, see Figure 5 The operation of step S204 may specifically be:

[0096] S501. Parse the initial real-time transmission message sequence to obtain the number of messages in the initial real-time transmission message sequence, time information of each message, and a sequence number of a reference message, where the reference message is the first message or the last message in the initial real-time transmission message sequence.

[0097] Optionally, the number of messages refers to the number of RTP messages containing IDR frames corresponding to the target data stream to be adjusted stored in the client buffer at the current moment, the time information of the message refers to the timestamp information T_ref of the RTP message, and the reference message sequence number refers to the sequence number N_ref of the first message in the initial RTP message sequence corresponding to the target video stream to be adjusted or the sequence number N_end of the last message.

[0098] It is worth noting that the initial RTP message sequence corresponding to the target video stream to be adjusted includes at least one initial RTP message, and this application does not make any specific limitation on this.

[0099] S502: Obtain the message payload of each message in the initial real-time transmission message sequence.

[0100] Optionally, from the initial RTP message sequence corresponding to the target video stream to be adjusted, the message load of each initial RTP message is extracted in sequence, and the message load refers to the NALU message included in the RTP message.

[0101] S503: The number of messages in the initial real-time transmission message sequence, the time information of each message, the sequence number of the reference message, and the message payload of each message are used as basic video features of the target video stream to be adjusted.

[0102] S504: construct a new real-time transmission message sequence according to the basic video features, wherein the number of messages in the new real-time transmission message sequence is the same as the number of messages in the initial real-time transmission message sequence.

[0103] Optionally, the array length of the newly created RTP message sequence is determined based on the number of initial RTP messages contained in the initial RTP message sequence, and the timestamp of each initial RTP message in the initial RTP message sequence is used as the timestamp of each RTP message in the newly created RTP message sequence, and the sequence number of each message in the newly created RTP message sequence is determined based on the sequence number of each message in the initial RTP message, so that the newly created RTP message sequence is continuous with the initial RTP message sequence.

[0104] For a possible implementation, see Figure 6 The operation of step S504 may specifically be:

[0105] S601: Generate a sequence number of each message in a new real-time transmission message sequence according to the number of messages in an initial real-time transmission message sequence and a sequence number of a reference message.

[0106] Optionally, continuous sequence numbers are generated according to the number of messages in the initial RTP message sequence and the sequence number of the reference message to serve as the sequence number of each message in the new RTP message sequence.

[0107] S602: Generate time information of each message in a new real-time transmission message sequence according to the time information of each message.

[0108] S603 generates a message header of each message in a new real-time transmission message sequence according to the message header of each message in the initial real-time transmission message sequence, the sequence number of each message in the new real-time transmission message sequence, and the time information of each message in the new real-time transmission message sequence.

[0109] Optionally, extract the message header of each initial RTP message in the initial RTP message sequence, and determine the message header of each RTP message in the newly created RTP message sequence based on the message header of each initial RTP message, the timestamp of each RTP message in the newly created RTP message sequence, and the sequence number of each message in the newly created RTP message sequence.

[0110] S604: Generate a message payload of each message in a new real-time transmission message sequence according to the message payload of each message.

[0111] Optionally, the message loads of the initial RTP messages in the initial RTP message sequence are sequentially used as the message loads of the messages in the newly created RTP message sequence.

[0112] In a possible implementation, the operation of step S601 may specifically be:

[0113] If the reference message is the first message, add the sequence number of the first message and the number of messages;

[0114] The sum of the added sum and i-1 is used as the sequence number of the i-th message in the new real-time transmission message sequence, where i is greater than or equal to 1, i is less than or equal to the number of messages, and i is an integer.

[0115] Optionally, the sequence number of each real-time transmission message in the newly constructed real-time transmission message sequence can be calculated according to the following formula (1), and the formula (1) is as follows:

[0116] C_t(i)=(N_ref+M)+(i-1)(1<=i<=M, i is an integer)(1)

[0117] Optionally, C_t(i) is used to indicate the sequence number of the i-th real-time transmission message in the new real-time transmission message sequence, i is used to indicate the i-th real-time transmission message in the new real-time transmission message sequence, N_ref is used to indicate the sequence number of the first message in the initial real-time transmission message sequence, and M is used to indicate the number of real-time transmission messages in the initial real-time transmission message sequence.

[0118] Figure 7 is a schematic diagram of the structure of an existing real-time transmission message, see Figure 7 The RTP message is composed of a message header RTPheader and RTP load content Payload. The load content of the RTP message is composed of a NALU message. The content of the NALU message is specifically determined by the video stream decoding result, and this application does not make any specific restrictions on this.

[0119] Figure 8 A schematic diagram of the structure of a message header of an existing real-time transmission message is shown in FIG. Figure 8 The RTP header of the RTP message is composed of version number, padding bit P, extension bit X, participating source bit, mark bit M, payload type, sequence number, timestamp and synchronization source identifier. The sequence number is used to sort the data packets in the video stream. After each data packet is sent, the sequence number of the next data packet is increased by 1; the timestamp refers to the sampling time of the data packet. When multiple RTP messages belong to the same video frame, these RTP messages have the same timestamp.

[0120] In a possible implementation, the specific operation of adding the new real-time transmission message sequence to the client buffer in the above step S204 may be:

[0121] It is determined whether there is a current real-time transmission message of the target video stream to be adjusted in the client buffer area. If not, a new real-time transmission message sequence is added to the client buffer area.

[0122] Optionally, whether to cache the newly constructed RTP message sequence to the client buffer at the current moment to wait for the client decoder to decode the newly constructed RTP message sequence, it is also necessary to determine whether there is a new RTP message sequence corresponding to the target video stream to be adjusted in the client buffer at the current moment. If it is determined that the client buffer has not received the new RTP message sequence corresponding to the target video stream to be adjusted at the current moment, the newly constructed RTP message sequence is added to the client buffer, and the newly constructed RTP message sequence is further input into the client decoder to prevent blank frames from appearing on the client display due to packet loss.

[0123] Optionally, the current real-time transmission message refers to a new RTP message corresponding to the target video stream to be adjusted that enters the client buffer at the current moment.

[0124] Optionally, if it is determined that a new RTP message sequence corresponding to the target video stream to be adjusted is received in the client buffer at the current moment, there is no need to add the newly constructed RTP message sequence to the client buffer, and the new RTP message sequence of the target video stream to be adjusted newly arrived in the client buffer is directly input into the client decoder to obtain the corresponding video data.

[0125] For a possible implementation, see Fig. 9 The operation of step S202 may specifically be:

[0126] S901: Determine a target video stream to be adjusted according to the collision type and collision number of each video stream.

[0127] Optionally, the collision type is used to indicate the type of collision combination of I frame collisions in each video stream. The collision type of the video stream may be a five-frame collision, a four-frame collision, a three-frame collision, a two-frame collision, etc.; the number of collisions refers to the number of times that the I frame data in each video stream collides when it is transmitted to the client. The number of collisions of the video stream may be 1, 3, 4, 5, etc., and this application does not make any specific limitation on this.

[0128] Optionally, based on the collision detection results of each video stream, the video stream in the collision combination with the largest number of collision frames corresponding to the collision combination in the collision detection results can be used as the target video stream to be adjusted; if there is only one collision combination in the collision detection results that contains the most video streams, the video stream in the collision combination is determined as the target video stream to be adjusted.

[0129] Optionally, based on the collision detection results of each video stream, the video stream in the collision combination with the largest number of occurrences in the collision detection results can be used as the target video stream to be adjusted. If there are two collision combinations of the same video streams in the collision combination, the number of occurrences of each collision combination is detected, and the video stream in the collision combination with the largest number of occurrences is determined as the target video stream to be adjusted.

[0130] Optionally, if there is no collision combination of the same video stream in the collision combination, the video stream in the collision combination that appears for the first time is determined as the target video stream to be adjusted by default.

[0131] S902, or, extract multiple frame features of each video stream, and determine the target video stream to be adjusted according to the change amount of the target frame features of each video stream within a preset time period.

[0132] Optionally, the preset time period refers to a time period preset by the user, and the preset time period may be 10 seconds, 15 seconds, 30 seconds, etc., and this application does not make any specific limitation on this.

[0133] Optionally, the target frame feature refers to the same frame feature selected from each video stream where an I frame collision occurs, and the variation refers to the variation range of the target frame feature of consecutive frames of each video stream within a preset time period.

[0134] S903, or, according to the number of collisions of each video stream, determine the target video stream to be adjusted.

[0135] Optionally, according to the collision times of each video stream in the collision detection result detected in the current observation window, the video stream with the most collisions may be regarded as the target video stream to be adjusted.

[0136] For a possible implementation, see Fig.10 The operation of step S201 may specifically be:

[0137] S1001. Number each detected video stream to obtain an identifier of each video stream.

[0138] Optionally, each video stream input to the client is detected at the current moment, and each detected video stream is numbered to obtain an identification of each video stream. The identification of the video stream is mainly used to distinguish the data of the video stream. The identification of the video stream can be represented by flow_ID, and this application does not make any specific limitations on this.

[0139] Exemplarily, if the number of video streams detected input to the client at the current moment is N, the j-th video stream can be represented as flow_ID_j, where j is an integer and j is greater than or equal to 1 and j is less than or equal to N. The video stream identification sequence of the client at the current moment is flow_ID = [1, 2, 3, ..., N].

[0140] S1002. Record the video information of each video stream according to the identifier of each video stream, the video information including: at least one key frame information, the key frame information including: the key frame start time, the key frame end time, the number of data packets in the key frame and at least one of the size of each data packet.

[0141] Optionally, the video information is used to indicate the basic information of the video stream; the key frame information is used to indicate the I frame information contained in the video stream, and the I frame is composed of at least one data packet; the key frame start time is used to indicate the arrival time of the first data packet in the current I frame entering the client; the I frame end time is used to indicate the arrival time of the last data packet in the current I frame entering the client; the number of data packets refers to the number of data packets contained in the current I frame, and the size of the data packet refers to the data length contained in each data packet in the current I frame. Among them, the I frame data contained in the video stream can be one or more, which is specifically determined by the encoding result of the video data downloaded by the video source, and this application does not make specific restrictions on this.

[0142] For example, at the current moment, it is recorded that there are M I frames in the v-th video stream flow_ID_v, the start time of the j-th I frame is recorded as: time_s_j (1<=j<=M), and the end time of the j-th I frame is recorded as: time_e_j (1<=j<=M). If the current I frame at the current moment contains Y data packets, the data packet size of the k-th data packet in the current I frame is recorded as PktSize_k (1<=k<=Y).

[0143] Optionally, the key frame information also includes: a key frame arrival delay, where the key frame arrival delay is used to indicate the difference between the arrival time of the first data packet of the first I frame of the current video stream received by the client and the start time of the video source to which the current video stream belongs.

[0144] Optionally, the time length of the client's observation window is set to T-win, and the start time of the observation window is set to be synchronized with the start time of the video source. The length of the observation window is not less than many times the maximum value of the I frame period in all video streams in the video source, so that multiple complete I frame periods can be observed simultaneously in the client's observation window.

[0145] Optionally, the I frame size refers to the sum of the sizes of all data packets constituting the current I frame, the I frame period refers to the average value of the difference between the arrival time of the first data packet of the current I frame and the arrival time of the first data packet of the adjacent I frame at the client buffer, and the I frame duration refers to the average value of the time difference between the arrival time of the first data packet of the current I frame at the client buffer and the arrival time of the last data packet of the current I frame at the client buffer.

[0146] S1003. Record current collision information of each video stream according to the video information of each video stream.

[0147] Optionally, according to the video information of each video stream received by the client, the current collision information of each video stream is recorded respectively.

[0148] The following describes the apparatus, device, and computer-readable storage medium used to execute the video stream processing method provided by the present application. The specific implementation process and technical effects are described above and will not be repeated below.

[0149] Fig.11 is a schematic diagram of the structure of a video stream processing device provided in an embodiment of the present application, see Fig.11 , the device comprises:

[0150] An acquisition module 1101 is used to acquire current collision information of each video stream;

[0151] A determination module 1102, configured to determine a target video stream to be adjusted according to current collision information of each video stream and a change amount of frame features of each video stream;

[0152] The extraction module 1103 is used to extract the initial real-time transmission message sequence corresponding to the target video stream to be adjusted from the client buffer area according to the identifier of the target video stream to be adjusted;

[0153] The construction module 1104 is used to extract the basic video features of the target video stream to be adjusted from the initial real-time transmission message sequence, construct a new real-time transmission message sequence according to the basic video features, and add the new real-time transmission message sequence to the client buffer area.

[0154] In an optional implementation manner, the extraction module 1103 is specifically used for:

[0155] Traverse each real-time transmission message in the client buffer area. For the current real-time transmission message traversed, if the video stream identifier corresponding to the current real-time transmission message is the identifier of the target video stream to be adjusted and the current real-time transmission message belongs to a real-time transmission message of a preset type, then add the current real-time transmission message to the initial real-time transmission message sequence.

[0156] In an optional implementation manner, the construction module 1104 is specifically used for:

[0157] Parsing the initial real-time transmission message sequence to obtain the number of messages in the initial real-time transmission message sequence, time information of each message, and a sequence number of a reference message, where the reference message is the first message or the last message in the initial real-time transmission message sequence;

[0158] Obtaining the message payload of each message in the initial real-time transmission message sequence;

[0159] The number of messages in the initial real-time transmission message sequence, the time information of each message, the sequence number of the reference message and the message payload of each message are used as the basic video features of the target video stream to be adjusted;

[0160] A new real-time transmission message sequence is constructed according to the basic video features, wherein the number of messages in the new real-time transmission message sequence is the same as the number of messages in the initial real-time transmission message sequence.

[0161] In an optional implementation manner, the construction module 1104 may be specifically used for:

[0162] Generate a sequence number of each message in a new real-time transmission message sequence according to the number of messages in the initial real-time transmission message sequence and the sequence number of the reference message;

[0163] Generate the time information of each message in a new real-time transmission message sequence according to the time information of each message;

[0164] Generate a message header of each message in a new real-time transmission message sequence according to a message header of each message in the initial real-time transmission message sequence, a sequence number of each message in the new real-time transmission message sequence, and time information of each message in the new real-time transmission message sequence;

[0165] The message payload of each message in a new real-time transmission message sequence is generated according to the message payload of each message.

[0166] In an optional implementation manner, the construction module 1104 may also be specifically used for:

[0167] If the reference message is the first message, add the sequence number of the first message and the number of messages;

[0168] The sum of the added sum and i-1 is used as the sequence number of the i-th message in the new real-time transmission message sequence, where i is greater than or equal to 1, i is less than or equal to the number of messages, and i is an integer.

[0169] In an optional implementation manner, the construction module 1104 may also be specifically used for:

[0170] It is determined whether there is a current real-time transmission message of the target video stream to be adjusted in the client buffer area. If not, a new real-time transmission message sequence is added to the client buffer area.

[0171] In an optional implementation manner, the above-mentioned determination module 1102 may be specifically used for:

[0172] Determine the target video stream to be adjusted according to the collision type and collision number of each video stream;

[0173] Or, extract multiple frame features of each video stream, and determine the target video stream to be adjusted according to the change amount of the target frame features of each video stream within a preset time period;

[0174] Or, the target video stream to be adjusted is determined according to the number of collisions of each video stream.

[0175] In an optional implementation manner, the acquisition module 1101 may be specifically used for:

[0176] Numbering each detected video stream to obtain an identification of each video stream;

[0177] According to the identifier of each video stream, respectively record the video information of each video stream, the video information includes: at least one key frame information, the key frame information includes: at least one of the key frame start time, the key frame end time, the number of data packets in the key frame and the size of each data packet;

[0178] According to the video information of each video stream, the current collision information of each video stream is recorded.

[0179] The above-mentioned device is used to execute the method provided by the aforementioned embodiment, and its implementation principle and technical effect are similar, which will not be repeated here.

[0180] The above modules may be one or more integrated circuits configured to implement the above methods, such as one or more application specific integrated circuits (ASICs), or one or more microprocessors, or one or more field programmable gate arrays (FPGAs). For another example, when a module is implemented in the form of a processing element scheduling program code, the processing element may be a general-purpose processor, such as a central processing unit (CPU) or other processor that can call program code. For another example, these modules may be integrated together and implemented in the form of a system-on-a-chip (SOC).

[0181] Fig.12 is a schematic diagram of the structure of a computer device provided in an embodiment of the present application. Fig.12 The computer device includes: a memory 1201 and a processor 1202. The memory 1201 stores a computer program that can be run on the processor 1202. When the processor 1202 executes the computer program, the steps in any of the above method embodiments are implemented.

[0182] An embodiment of the present application further provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments can be implemented.

[0183] Optionally, the present application also provides a program product, such as a computer-readable storage medium, including a program, which is used to execute any of the above-mentioned video stream processing method embodiments when executed by a processor.

[0184] In the several embodiments provided by the present invention, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0185] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0186] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0187] The above-mentioned integrated unit implemented in the form of a software functional unit can be stored in a computer-readable storage medium. The above-mentioned software functional unit is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor (English: processor) to perform some steps of the methods of various embodiments of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (English: Read-Only Memory, abbreviated: ROM), random access memory (English: Random Access Memory, abbreviated: RAM), disk or optical disk and other media that can store program codes.

[0188] The above are only specific implementations of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0189] The above description is only the preferred embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A video stream processing method, characterized in that: The method comprises: Get the current collision information of each video stream; Determine the target video stream to be adjusted according to the current collision information of each video stream and the frame feature change amount of each video stream; Extracting, from a client buffer area, an initial real-time transmission message sequence corresponding to the target video stream to be adjusted according to the identifier of the target video stream to be adjusted; Extracting basic video features of the target video stream to be adjusted from the initial real-time transmission message sequence, constructing a new real-time transmission message sequence according to the basic video features, and adding the new real-time transmission message sequence to the client buffer area.

2. The video stream processing method according to claim 1, characterized in that: The step of extracting, from a client buffer area, an initial real-time transmission message sequence corresponding to the target video stream to be adjusted according to the identifier of the target video stream to be adjusted, comprises: Traversing each real-time transmission message in the client buffer area, for the traversed current real-time transmission message, if the video stream identifier corresponding to the current real-time transmission message is the identifier of the target video stream to be adjusted and the current real-time transmission message belongs to a real-time transmission message of a preset type, then adding the current real-time transmission message to the real-time transmission message sequence.

3. The video stream processing method according to claim 1, characterized in that: The step of extracting basic video features of the target video stream to be adjusted from the initial real-time transmission message sequence and constructing a new real-time transmission message sequence according to the basic video features includes: Parsing the initial real-time transmission message sequence to obtain the number of messages in the initial real-time transmission message sequence, time information of each message, and a sequence number of a reference message, wherein the reference message is the first message or the last message in the initial real-time transmission message sequence; Obtaining the message payload of each message in the initial real-time transmission message sequence; The number of messages in the initial real-time transmission message sequence, the time information of each message, the sequence number of the reference message and the message payload of each message are used as the basic video features of the target video stream to be adjusted; The new real-time transmission message sequence is constructed according to the basic video features, wherein the number of messages in the new real-time transmission message sequence is the same as the number of messages in the initial real-time transmission message sequence.

4. The video stream processing method according to claim 3, characterized in that: The constructing the new real-time transmission message sequence according to the basic video features includes: Generate a sequence number of each message in the new real-time transmission message sequence according to the number of messages in the initial real-time transmission message sequence and the sequence number of the reference message; Generating the time information of each message in the new real-time transmission message sequence according to the time information of each message; Generate a message header of each message in the new real-time transmission message sequence according to the message header of each message in the initial real-time transmission message sequence, the sequence number of each message in the new real-time transmission message sequence, and the time information of each message in the new real-time transmission message sequence; The message payload of each message in the new real-time transmission message sequence is generated according to the message payload of each message.

5. The video stream processing method according to claim 4, characterized in that: The generating, according to the number of messages in the initial real-time transmission message sequence and the sequence number of the reference message, the sequence number of each message in the new real-time transmission message sequence comprises: If the reference message is the first message, then summing the sequence number of the first message and the number of messages; The sum of the sum and i-1 is used as the sequence number of the i-th message in the new real-time transmission message sequence, wherein i is greater than or equal to 1, and i is less than or equal to the number of messages, and i is an integer.

6. The video stream processing method according to claim 1, characterized in that: The step of adding a new real-time transmission message sequence to the client buffer area includes: It is determined whether there is a current real-time transmission message of the target video stream to be adjusted in the client buffer area; if not, a new real-time transmission message sequence is added to the client buffer area.

7. The video stream processing method according to claim 1, characterized in that: The step of determining the target video stream to be adjusted according to the current collision information of each video stream and the frame feature variation of each video stream comprises: Determine the target video stream to be adjusted according to the collision type and collision number of each video stream; Or, extract multiple frame features of each video stream, and determine the target video stream to be adjusted according to the change amount of the target frame features of each video stream within a preset time period; Or, the target video stream to be adjusted is determined according to the number of collisions of each video stream.

8. The video stream processing method according to claim 1, characterized in that: The obtaining of current collision information of each video stream includes: Numbering each detected video stream to obtain an identification of each video stream; Recording video information of each video stream according to an identifier of each video stream, the video information including: at least one key frame information, the key frame information including: at least one of a key frame start time, a key frame end time, a number of data packets in the key frame, and a size of each data packet; According to the video information of each video stream, the current collision information of each video stream is recorded.

9. A video stream processing device, characterized in that: The device comprises: An acquisition module is used to obtain the current collision information of each video stream; A determination module, used to determine the target video stream to be adjusted according to the current collision information of each video stream and the frame feature change amount of each video stream; An extraction module, used for extracting an initial real-time transmission message sequence corresponding to the target video stream to be adjusted from a client buffer area according to an identifier of the target video stream to be adjusted; A construction module is used to extract basic video features of the target video stream to be adjusted from the initial real-time transmission message sequence, construct a new real-time transmission message sequence according to the basic video features, and add the new real-time transmission message sequence to the client buffer area.

10. A computer device, characterized in that: include: A memory and a processor, wherein the memory stores a computer program that can be run on the processor, and when the processor executes the computer program, the steps of the method described in any one of claims 1 to 8 are implemented.

Images