Video lag identification method and device, storage medium and computer program product

Abstract

The embodiment of the invention provides a video lagging identification method and device, a storage medium and a computer program product, and relates to the technical field of communication, and the method comprises the steps: obtaining an RTP signaling data packet of a target RTP video stream; when the video frame serial numbers corresponding to the sequentially acquired RTP signaling data packets have discontinuous sequences, determining that the video frames where the RTP signaling data packets corresponding to the missing sequences among the discontinuous sequences are located are bad frames; and when the time interval data between the starting time of the bad frame and the starting time of the next key video frame which is a good frame is greater than a preset sensing delay threshold, it is determined that the target RTP video stream is lagged, and the video sequence number corresponding to the RTP signaling data packet of the good frame does not have a missing sequence. The video lagging identification method provided by the embodiment of the invention has nothing to do with the external environment, so that the video quality can be evaluated more accurately, and the user experience feeling of the video business service is improved.

Description

Technical Field

[0001] The embodiments of the present application relate to, but are not limited to, the field of communication technologies, and in particular to a method, device, storage medium, and computer program product for identifying video freezes. Background Art

[0002] With the development of 5G, video services are becoming increasingly widespread. To ensure the optimal provision of network access services and a high-quality mobile video experience within limited mobile network bandwidth, video quality assessment is necessary to accurately evaluate and analyze user experience issues related to video freezes. However, existing video freeze detection methods primarily rely on quality indicators based on image features. However, during video capture and transmission, objective interference from the external environment can affect video resource quality, making it difficult to accurately determine freezes and leading to misidentification. Furthermore, these technologies lack the ability to measure specialized metrics for network access services. Therefore, finding a more accurate way to assess video quality and improve the user experience of video services is a pressing technical challenge. Summary of the Invention

[0003] The following is a summary of the subject matter described in detail herein. This summary is not intended to limit the scope of the claims.

[0004] The embodiments of the present application provide a method, device, storage medium, and computer program product for identifying video freezes, which can more accurately evaluate video quality, thereby improving the user experience of video business services.

[0005] In a first aspect, a method for identifying video freeze according to an embodiment of the present application includes:

[0006] Obtain the RTP signaling data packet of the target RTP video stream; when a discontinuous sequence appears in the video frame sequence number corresponding to the RTP signaling data packet, determine that the video frame where the RTP signaling data packet corresponding to the missing sequence between the discontinuous sequences is located is a bad frame; when the time interval data between the start time of the bad frame and the start time of the next key video frame which is a good frame is greater than a preset perceptual delay threshold, determine that there is a freeze in the target RTP video stream, wherein the video sequence number corresponding to the RTP signaling data packet of the good frame does not appear in a missing sequence.

[0007] In a second aspect, an electronic device provided according to an embodiment of the present application includes: at least one processor and at least one memory for storing at least one program; when at least one of the programs is executed by at least one of the processors, the method described in any one of the first aspects is implemented.

[0008] In a third aspect, according to a computer-readable storage medium provided in an embodiment of the present application, the computer-readable storage medium stores computer-executable instructions, and the computer-executable instructions are used to execute the method as described in any one of the first aspects.

[0009] In a fourth aspect, a computer program product provided according to an embodiment of the present application includes a computer program or computer instructions, wherein the computer program or the computer instructions are stored in a computer-readable storage medium, and the processor of a computer device reads the computer program or the computer instructions from the computer-readable storage medium, and the processor executes the computer program or the computer instructions, so that the computer device executes: the method as described in any one of the first aspects.

[0010] In the embodiment of the present application, by judging the continuity of the video frame sequence number of the RTP signaling data packet in the target RTP video stream, it is possible to identify whether there is a bad video frame, and then determine whether there is a freeze in the target RTP video stream. Compared with the related art, the method for identifying video freeze in the embodiment of the present application is independent of the external environment. Therefore, the embodiment of the present application can perform a more accurate evaluation of the video quality, thereby improving the user experience of the video business service. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Figure 1 A schematic diagram of the position of an apparatus corresponding to the method for identifying video freeze in an embodiment of the present application in a signaling monitoring system;

[0012] Figure 2 A schematic diagram of a module of a device corresponding to the method for identifying video freezes according to an embodiment of the present application;

[0013] Figure 3 A schematic diagram of steps of an embodiment of the video freeze identification method according to the present application;

[0014] Figure 4 This is a flow chart of an embodiment of a method for identifying video freezes according to an embodiment of the present application;

[0015] Figure 5 This is a schematic diagram of a video frame of a target RTP video stream received in one embodiment of a method for identifying video freeze according to an embodiment of the present application;

[0016] Figure 6 This is a schematic diagram of the structure of the encoding format used in one embodiment of the method for identifying video freezes according to an embodiment of the present application;

[0017] Figure 7 This is a flow chart of another embodiment of the method for identifying video freezes according to the embodiment of the present application;

[0018] Figure 8 Schematic diagram of the hardware structure of the device corresponding to the video freeze identification method of the embodiment of the present application. DETAILED DESCRIPTION

[0019] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0020] It should be noted that although the device schematics illustrate functional module divisions and the flowcharts illustrate logical sequences, in certain circumstances, the steps shown or described may be performed in a sequence that differs from the module divisions in the device or the sequence in the flowcharts. The terms "first," "second," and so on, in the specification, claims, and drawings, are used to distinguish similar items and are not necessarily used to describe a specific sequence or precedence.

[0021] The flowcharts shown in the accompanying drawings are for illustrative purposes only and do not necessarily include all contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps may be decomposed, while others may be combined or partially combined. Therefore, the actual execution order may vary depending on the actual situation.

[0022] The following are the Chinese and English definitions of the terms used in the embodiments of this application:

[0023] Technical terms English abbreviations and full letters Chinese explanation Related standard information RTP Real-time Transport Protocol Real-time Transport Protocol RFC3550 XDR X Data Recording Data Recording NAL Network Abstract Layer Network Abstraction Layer RFC6184, RFC7798 IDR Instantaneous Decoding Refresh Instant decoding refresh RFC6184, RFC7798 Seq sequence Serial number RFC3550

[0024] The following is an explanation of the terms used in the embodiments of this application:

[0025] Video streams include coded streams and raw streams. Coded streams are stream data compressed using a compression algorithm. For example, stream data obtained using H264 as the compression / encoding algorithm is also called an H264 stream. Raw streams are uncompressed stream data and are decoded stream data.

[0026] Frames are the basic elements of a stream. For video encoding / compression, their core is to store a set of temporally continuous frame data in the smallest possible space. For video decoding, on the other hand, the core is to restore a set of encoded / compressed frame data to its original state as much as possible.

[0027] IDR frames are a special type of video encoding frame. They are a type of I-frame. I-frames are important frames in inter-frame compression coding. They are fully compressed coded frames. During decoding, the complete image can be reconstructed using only the I-frame data. I-frames do not require reference to other frames to generate.

[0028] It's understandable that from the invention of wired telephones to the advent of mobile phones, and then to the widespread adoption of 2G, 3G, and 4G mobile communication networks, calling, as a core function of mobile communications, has continuously evolved. 5G VoNR (Voice over New Radio) has been introduced based on 5G networks. This includes ultra-high-definition voice and video calling services, providing customers with a visual, multimedia, and highly perceptual ultra-high-definition video experience. However, with the development of 5G, the widespread application of video services has also placed more stringent requirements on optimizing operator networks and understanding and monitoring the quality of service experience in video service delivery. This is necessary to reasonably provide network access services and deliver a high-quality mobile video experience within limited mobile network bandwidth. However, relevant technologies lack mature quality assessment methods, making it difficult to accurately evaluate and analyze the perceived lag associated with service usage. Current approaches to detecting video lag primarily rely on quality indicators based on image features. However, these approaches, due to objective interference from the external environment during video capture and transmission, can affect video resource quality, making it difficult to accurately determine lag and potentially misidentify instances. Furthermore, they lack the ability to measure specialized metrics for network access services. Based on this, the embodiments of the present application provide a method, device, storage medium and computer program product for identifying video freezes, which can more accurately evaluate video quality, thereby improving the user experience of video business services.

[0029] Reference Figure 1 As shown, the embodiment of the present application provides a protocol identification device that can be built into a signaling monitoring system. Figure 1 The device is deployed within a signaling monitoring system. The protocol recognition device first identifies RTP signaling packets, then further identifies video frames. It then performs algorithmic processing on these frames, identifying indicators related to lag. Finally, the synthesis device performs video synthesis processing to produce the video stream XDR for display.

[0030] Reference Figure 2 As shown, the protocol identification device includes a protocol identification main module, a data storage module, an RTP decoding module, a synthesis module, a video frame decoding module and a jamming algorithm processing module.

[0031] The protocol identification main module is responsible for identifying whether the signaling data packet to be processed is an RTP signaling data packet.

[0032] The RTP decoding module receives the signaling data packet from the protocol identification main module, decodes it, and returns the decoding result. The decoding can be performed according to the format of the RTP HEADER.

[0033] The data storage module, on the one hand, receives the source IP and destination IP and PORT quadruple (corresponding to network address parameters) from the protocol identification main module and saves it into the database, and marks the application layer protocol as RTP; on the other hand, it receives the quadruple query message from the protocol identification main module, queries the database whether the application layer protocol carried by the quadruple is RTP, and returns the result.

[0034] The synthesis module is responsible for the final synthesis processing. Specifically, it first identifies the RTP signaling data packet as a video and sends it to the video frame decoding module. It distinguishes the H264 and H265 encoding formats based on the RTP payload identification result of the video frame, and decodes the Slice HEADER parameters (that is, the slice structure data of the encoding format) to identify the frame type.

[0035] The jam algorithm processing module receives the frame type returned from the video frame decoding module and uses the RTP sequence number to determine the quality of a frame. It then determines the validity of the frame (i.e., whether it is a good or bad frame) based on the user's latency threshold during the video call. In another embodiment, the number and duration of jams are calculated based on the frequency and duration of invalid frames (i.e., bad frames), and the result is returned.

[0036] For example, in conjunction with Figure 3 , a detailed description of the implementation process of the device in this application is given as follows:

[0037] In the protocol identification main module, after receiving the collected signaling of the unknown port, the protocol identification main module extracts the source IP and PORT four-tuple from the signaling data packet of the unknown first data stream as the network address parameter, and then executes step S1 to send the four-tuple to the data storage module. The data storage module queries the database for target network address parameters that match the network address parameters, and executes step S2 to return the query result of the database. The protocol identification main module determines that the returned result is a successful query result, and then identifies the current signaling data packet as an RTP signaling data packet. If the data storage module returns a failed query result, the protocol identification main module executes step S3 to send the current signaling data packet to the RTP decoding module for decoding. The protocol identification main module receives the decoding result from the RTP decoding module through step S4. If the decoding fails, the current signaling data packet is discarded. If the decoding is successful, the current signaling is identified as an RTP signaling data packet. At the same time, since RTP is a long link, RTP signaling can be transmitted multiple times after the link is established. For a link established using a non-well-known port, if a signaling on the link is identified as RTP, then the subsequent signaling must be RTP. In order to improve the efficiency of subsequent signaling identification, after a signaling is identified as RTP, the protocol identification main module executes step S5 to send the source IP and PORT four-tuple to the data storage module. After receiving the four-tuple, the data storage module records it in the memory database and marks the application layer protocol as RTP for subsequent query. For the identified RTP signaling data packet, the protocol identification main module executes step S6 to output the RTP signaling data packet to the synthesis module. Figure 3As shown, the synthesis module receives the identified RTP signaling data packet and identifies whether the RTP signaling data packet is video or audio based on the code stream characteristics (for example, if the payload length is greater than 500 bytes, it is determined to be video; if two consecutive RTP messages have the same timestamp, it is determined to be video; if two consecutive RTP messages have different timestamps, the interval is greater than or equal to 900 and the payload lengths of both RTP messages are greater than the length of the silence packet, it is determined to be video; if two consecutive RTP messages have an interval less than 900 or at least one of the two RTP message payload lengths is less than or equal to the length of the silence packet, it is determined to be audio. Audio packets are discarded and not processed); the synthesis module executes step S7 to send the RTP signaling data packet determined to be video to the video frame decoding module for decoding and identification. The video frame decoding module decodes and identifies the RTP Payload part of the identified video code stream. The synthesis module receives the video frame identification results (such as I frames and P frames) from the video frame decoding module through step S8, and then executes step S9 to send the identified video frames to the stuttering algorithm processing module for processing. All identified video frames (I frames, P frames) are judged as good or bad frames. The synthesis module obtains the jamming identification related index results from the jamming algorithm processing module in step S10, synthesizes them and outputs the XDR in step S11.

[0038] Therefore, the embodiments of the present application can be applied to freeze identification of video calls based on RTP streams.

[0039] Reference Figure 4 As shown, an embodiment of the present application provides a method for identifying video freezes, the method comprising:

[0040] Step S100: Acquire the RTP signaling data packet of the target RTP video stream;

[0041] Step S200: When a discontinuous sequence appears in the video frame sequence number corresponding to the RTP signaling data packet, the video frame corresponding to the missing sequence between the discontinuous sequences is determined to be a bad frame;

[0042] Step S300: When the time interval between the start time of the bad frame and the start time of the next key video frame which is a good frame is greater than the preset perception delay threshold, it is determined that there is a jam in the target RTP video stream, wherein the video sequence number corresponding to the RTP signaling data packet of the good frame does not have a missing sequence.

[0043] Therefore, by judging the continuity of the RTP signaling data packets in the target RTP video stream, it is possible to identify whether there are bad video frames, and then determine whether there is a freeze in the target RTP video stream. Compared with related technologies, the method for identifying video freezes in the embodiment of the present application is independent of the external environment. Therefore, the embodiment of the present application can perform a more accurate evaluation of the video quality, thereby improving the user experience of the video business service.

[0044] In one embodiment, the target RTP video stream is an RTP data stream of video type for which a freeze is to be determined. Normally, a target RTP video stream contains multiple video frames, each of which corresponds to at least one RTP signaling data packet. The RTP signaling data packet is a signaling data packet transmitted using the RTP protocol.

[0045] In one embodiment, the video frame sequence number corresponding to the RTP signaling data packet may be a Sequence number (Seq) in the header data of the RTP signaling data packet. Typically, the Sequence number of the first RTP signaling data packet in the RTP video stream may be random, and the Sequence number of each subsequent RTP signaling data packet may be incremented by 1.

[0046] It is understandable that when the RTP signaling data packets corresponding to the video frame sequence numbers of the discontinuous sequence are in the same video frame, then the video frame where the RTP data packets corresponding to the missing sequence between the discontinuous sequences are located is also in the same video frame, and this video frame is determined to be a bad frame, and the start time of the video frame is the start time of the bad frame; in another embodiment, when the RTP signaling data packets corresponding to the video frame sequence numbers of the discontinuous sequence are not in the same video frame, such as the previous RTP signaling data packet corresponds to video frame 1, the subsequent RTP signaling data packet corresponds to video frame 2, and the previous RTP signaling data packet is the tail packet of video frame 1, then the discontinuous sequence between the discontinuous sequences is a bad frame. The video frame where the RTP data packet corresponding to the missing sequence is located is the next frame of video frame 1, and the next frame of video frame 1 is determined to be a bad frame, and the start time of the next frame of video frame 1 is the start time of the bad frame; or in some embodiments, the previous RTP signaling data packet corresponds to video frame 1, the next RTP signaling data packet corresponds to video frame 2, and the previous RTP signaling data packet is not the last packet, and the next RTP signaling data packet is not the first packet, then the video frames where the RTP data packets corresponding to the missing sequence between the discontinuous sequences are located are video frames 1 to video frames 2, and it is determined that video frames 1 to video frames 2 are all bad frames, and the start time of video frame 1 is the start time of the bad frame. Therefore, when a discontinuous sequence appears in the video frame sequence number corresponding to the RTP signaling data packet, it can be determined that the video frame where the RTP signaling data packet corresponding to the missing sequence between the discontinuous sequences is located is a bad frame based on whether the RTP signaling data packet is the first packet or the last packet. When continuous bad frames appear, the start time of the bad frame can specifically refer to the start time of the first bad frame. In another embodiment, the first or last packet can be distinguished by other methods such as Seq setting, and this application does not impose specific limitations on this. The start time can be the timestamp recorded in the first packet of the corresponding bad frame, or it can be determined by recording the time of receiving the RTP signaling data packet; when the first packet of the bad frame cannot be determined, the end time of the previous good frame of the bad frame can be used (such as the timestamp recorded in the last packet of the previous good frame or the time of receipt).

[0047] For example, taking the RTP signaling data packet in step S200 as RTP signaling data packet 1, when the video frame sequence number of RTP signaling data packet 1 is discontinuous, it means that the video frame sequence number of RTP signaling data packet 1 and the RTP signaling data packet 2 in the target RTP video stream received immediately before are discontinuous. For example, when RTP signaling data packet 1 and RTP signaling data packet 2 are in the same video frame 1, the video frame where the RTP data packet corresponding to the missing sequence between the discontinuous sequences is located is the video frame 1, and the video frame 1 is determined to be a bad frame, and the start time of video frame 1 is the start time of the bad frame; for example, when RTP signaling data packet 1 and RTP signaling data packet 2 are not In the same video frame (such as the previous RTP signaling data packet 2 corresponds to video frame 2, and the subsequent RTP signaling data packet 1 corresponds to video frame 1), when the previous RTP signaling data packet 2 is the tail packet of video frame 2 and the subsequently received RTP signaling data packet 1 is any RTP signaling data packet of video frame 1, the bad frame is the next video frame of video frame 2, and the start time of the bad frame is the time corresponding to the tail packet of video frame 2; when the previous RTP signaling data packet 2 is a non-tail packet of video frame 2 and the subsequently received RTP signaling data packet 1 is any RTP signaling data packet of video frame 1, the bad frame is video frame 2, and the start time of the bad frame is the time corresponding to the first packet of video frame 2.

[0048] In one embodiment, determining that a video frame is a good frame means that the video sequence numbers corresponding to the RTP signaling packets included in the video frame do not contain any missing sequences, that is, the video sequence numbers are complete and continuous. For example, if video frame 2 contains RTP signaling packets 11-15, if RTP signaling packets 2-20 are obtained, then the video frame sequence numbers corresponding to the RTP signaling packets of video frame 2 are complete and continuous with no missing sequences, and video frame 2 is determined to be a good frame. Conversely, if RTP signaling packets 2-12 and 14-20, or 2-14, are obtained, then the video frame sequence numbers corresponding to the RTP signaling packets of video frame 2 contain missing sequences, and video frame 2 is determined to be a bad frame.

[0049] In one embodiment, the next key video frame that is a good frame indicates that the RTP sequence number of the RTP signaling data packet under the entire key video frame is continuous, and the frame type is an I frame. Exemplarily, taking the encoding of the RTP signaling data packet through the H264 and H265 encoding formats as an example, it can be determined whether it is a key video frame by decoding the Slice Header. Exemplarily, for the H264 encoding format, when nal_unit_type is 5, it is determined to be an IDR frame, that is, an I frame, and when nal_unit_type is 1, it is determined to be a non-IDR frame. At this time, it is further determined whether it is an I frame based on whether slice_type is 2 or 7, and whether it is a P frame based on whether slice_type is 0 or 5. Exemplarily, for the H265 encoding format, according to the type definition of the H265 encoding NAL unit, the Nal_unit_type is judged, and the Nal_unit_type value is 0-9 to identify it as a P frame, and the Nal_unit_type value is 16-21 to identify it as an I frame.

[0050] In one embodiment, video freeze judgment may not be performed on the first several frames of the target RTP video stream, such as not performing video freeze judgment on the first starting frame. In this regard, the embodiment of the present application does not make specific restrictions, and technical personnel in this field can selectively set it according to actual needs.

[0051] It is understandable that, for the purposes of this application, the identification of frame types is used to determine video freeze recovery, and it is sufficient to only identify the frame type as an I frame. For example, in actual applications, a video stream may contain I frames and multiple other types of video frames (such as B frames and P frames). It is possible to only determine whether the signaling data packet to be identified is an I frame; or it is possible to identify a specific frame type, which is not limited in this embodiment of the application.

[0052] It is understandable that the frame type analysis can be performed on each RTP signaling data packet of the target RTP video stream, or the frame type analysis can be performed only on the RTP signaling data packets between the appearance of the bad frame and the next good key video frame, wherein the judgment of the frame type can be made before or after the judgment of the continuity of the video frame sequence number. It is understandable that for the scenario where the judgment of the frame type is made before the judgment of the continuity of the video frame sequence number, during the invalid period between the appearance of the bad frame and the next good key video frame, only the frame type can be detected without making a judgment on the continuity of the video frame sequence number; illustratively, in one embodiment, Figure 5As shown, taking the first I frame as the starting frame of the target RTP video stream as an example, when the 7th video frame is identified as a P frame and a bad frame, the video frames between the first I frame and the 7th P frame are in the valid period, and there is no jamming. The continuity of the video frame serial number will be judged for each video frame in the valid period. At this time, the video frame from the 7th to the second I frame is in the invalid period. During the invalid period, the continuity judgment of the video frame serial number does not need to be performed, and only the frame type judgment is performed. By analogy, it is possible to identify whether each video frame is in the valid period or the invalid period, and to identify whether there is jamming. In another embodiment, the frame type will be parsed first, and then one video frame will be used as a unit to determine whether the video frame serial number of the RTP signaling data packet under each video frame is continuous. If it is continuous, it is a good frame, otherwise it is a bad frame. Further, as Figure 7 As shown, taking the device using H264 or H265 for encoding as an example, for each received RTP signaling data packet, the RTP signaling data packet with the audio and video type as video is decoded and identified by RTP Payload, and after identifying the encoding format as H264 or H265, the video code stream obtained by decoding each RTP signaling data packet based on RTP Payload is processed by SliceHeader to obtain the frame type of each RTP signaling data packet (such as I frame or P frame). For each RTP signaling data packet with an identified frame type, the video frame serial numbers of other RTP signaling data packets in the video frame are identified to determine whether all RTP signaling data packets in a video frame are all good frames or there are bad frames. When there is a bad frame, that is, there is an RTP signaling data packet that is discontinuous with the RTP signaling data packet received immediately thereafter, determine whether the time interval data from the start time of the bad frame to the next good key video frame is greater than the perceived delay threshold. When the time interval data is less than the perceived delay threshold, report that there is no freeze. When the time interval data is greater than the perceived delay threshold, report freeze-related indicators, such as reporting the presence of freeze, the number of freezes, and at least one of the freeze duration.

[0053] In one embodiment, after determining that the target RTP video stream has a freeze, data information can be output to indicate that the target RTP video stream has a freeze. In another embodiment, the data information that can be output includes freeze data, such as the freeze duration or the number of freezes. In this regard, the embodiment of the present application does not impose specific restrictions, and those skilled in the art can selectively set it according to actual needs.

[0054] Therefore, the embodiments of the present application further analyze good and bad frames based on RTP packet parsing, thereby calculating valid and invalid frames. Combined with the user's delay experience threshold during the video call, this method determines whether there is any lag. Compared with existing technologies, this method can achieve decoding and recognition based on RTP protocol data acquisition, without external interference with the video image, and report any lag during the user's video call. This can help service providers evaluate the quality of the video services they provide, optimize network operations, and improve the user experience of video services.

[0055] It is understandable that the video frame sequence number corresponding to the RTP signaling data packet of the target RTP video stream is the RTP sequence number in the header data of the RTP signaling data packet, and the RTP sequence number is the Seq field defined in the RTP protocol.

[0056] It is understandable that, before obtaining the RTP signaling data packet of the target RTP video stream, the method further includes:

[0057] Obtaining a network address parameter of a signaling data packet of an unknown first data flow;

[0058] Querying a preset database for target network address parameters that match the network address parameters;

[0059] When the target network address parameters are queried, it is determined that the first data stream is an RTP data stream; or,

[0060] When the target network address parameter is not found, RTP decoding is performed on the signaling data packet. When the decoding is successful, it is determined that the first data stream is an RTP data stream.

[0061] In one embodiment, the unknown first data stream is a data stream of a terminal device for which a freeze is to be detected, and the data stream sent by the same sender may also be transmitted using different protocol formats depending on the service type. Therefore, an RTP protocol determination is performed on the unknown first data stream to filter out the RTP signaling data packets of the target RTP video stream.

[0062] In one embodiment, the network address parameters for video transmission using RTP protocol encapsulation can be pre-set in the database. In other embodiments, the characteristic of RTP as a long link can be utilized to make real-time judgments based on the network address parameters of the first identified RTP signaling data packet.

[0063] In one embodiment, the database may be an in-memory database.

[0064] It should be understood that the network address parameter is a network quadruple, including a source address, a source port, a destination address, and a destination port.

[0065] It is understandable that, after determining that the first data stream is an RTP data stream, the method further includes:

[0066] The network address parameters of the successfully decoded signaling data packet are stored in the database.

[0067] Therefore, by storing the network address parameters of the last successfully parsed signaling data packet in the database, the recognition efficiency of subsequent signaling data packets can be improved.

[0068] It is understandable that, before obtaining the RTP signaling data packet of the target RTP video stream, the method further includes:

[0069] Get the header data of the RTP signaling data packet of the RTP data stream;

[0070] Determine the audio and video frame type corresponding to the RTP signaling data packet based on the packet header data;

[0071] An RTP signaling data packet whose audio and video frame type is video is determined as the RTP signaling data packet of the target RTP video stream.

[0072] In one embodiment, audio data is transmitted together with the target RTP video stream. Therefore, by determining the audio and video frame types, it can be identified whether the RTP signaling data packet is the RTP signaling data packet under the target RTP video stream.

[0073] It is understandable that the key video frame is determined by the video frame type of the RTP signaling data packet. The steps of identifying the video frame type include:

[0074] Extracting target frame format parameters at the same position as the network abstraction layer unit type in the preset first video format in the payload of the RTP signaling data packet;

[0075] Match the target frame format parameters with a preset video frame format list to obtain the target video frame format;

[0076] The RTP signaling data packet is parsed according to the target video frame format to obtain the video frame type.

[0077] In one embodiment, the I frame includes an IDR frame.

[0078] It should be understood that the network abstraction layer unit type corresponds to the NALU Type in the first video format.

[0079] For example, taking the H264 encoding format as an example, video frame parsing includes confirmation based on the nal_unit_type and slice_type fields. For example, when nal_unit_type is 5, it is determined to be an IDR frame, which must be an I frame. When nal_unit_type is 1, it is determined to be a non-IDR frame. At this time, if slice_type is 2 or 7, it is determined whether it is an I frame. If slice_type is 0 or 5, it is determined whether it is a P frame.

[0080] In one embodiment, the video frame format list is shown in Table 1 below:

[0081] Type_try Encoding format 0-2 H265 3 H264 4-9 H265 10 H264 11 Non-H264, non-H265 12-15 H264 16-21 H265 22 Non-H264, non-H265 23 H264 24-25 Non-H264, non-H265 26 H264 27 Non-H264, non-H265 28-31 H264 32-40 H265 41 Non-H264, non-H265 42 H264 43 Non-H264, non-H265 44-47 H264 48-50 H265 51 Non-H264, non-H265 52 H264 53-54 Non-H264, non-H265 55 H264 56-57 Non-H264, non-H265 58 H264 59 Non-H264, non-H265 60-63 H264

[0082] Table 1

[0083] It is understandable that the first video format is the H265 video format. Before parsing the RTP signaling data packet according to the target video frame format, the method further includes:

[0084] Extracting the target time domain layer label and the target abstract layer identifier at the same position as the time domain layer label and the network abstraction layer in the first video format in the payload of the RTP signaling data packet;

[0085] Parse the RTP signaling data packet according to the target video frame format to obtain the video frame type, including:

[0086] Determine whether the target video frame format is valid according to the target temporal layer label and the target abstraction layer identifier;

[0087] The RTP signaling data packet is parsed according to the valid target video frame format to obtain the video frame type.

[0088] It should be understood that the time domain layer label corresponds to the "Layer ID" field in H265, and the network abstraction layer corresponds to the "TID" field in H265. Since the first video format is used for pre-judgment, there is still a situation where the target video format does not match the actual encoding format. Therefore, through further judgment of the target time domain layer label and the target abstraction layer identifier, the accuracy of the target video frame format can be further determined.

[0089] For example, Figure 6 As shown, according to the H265 format, the target temporal layer label is extracted from the Layer ID field; the target abstraction layer identifier is extracted from the TID field. When the target temporal layer label is not all 0 or the target abstraction layer identifier is all 0, it indicates that the target video frame format is not H265. If the target video frame format obtained according to Table 1 is H264, it means that the target video frame format obtained in Table 1 is valid, so video frame parsing can be performed. Otherwise, video frame parsing is not performed.

[0090] It is understandable that the video frame format list is determined by the following steps:

[0091] Obtaining, according to the frame formats of the video frames supported by the RTP signaling data packet and the value of the network abstraction layer unit type in the first video format, a plurality of to-be-confirmed frame format parameter sets of the to-be-confirmed frame formats;

[0092] According to the historical usage status of the frame format parameter set to be confirmed, the H265 video format, and the H264 video format, the frame format corresponding to each frame format parameter in the frame format parameter set to be confirmed is determined, and a video frame format list is obtained.

[0093] It should be understood that there is a default available value for the network abstraction layer unit type of each frame format (ie, encoding format). Therefore, the video format list can be determined based on the value and historically used values.

[0094] For example, taking the first video frame format as H265, for H265, the unused values ​​of its network abstraction layer unit type (NALU Type) are (10-15, 22-31, 41-47, 51-63). If the value of NALU Type in H265 is recorded as Type_Try, the corresponding video format needs to be determined when Type_Try takes values ​​of 1-9, 16-21, 32-40, 48-50. For H264, a combination of NRI & NALU Type is required. At the same time, in actual applications, Type_Try values ​​of 4, 8, 9, 16, 19-20, 35-38, 40, and 48 must be H265. Therefore, after removing the values ​​supported by H265, the remaining video formats corresponding to Type_Try values ​​of 1-3, 5-7, 17-18, 33-34, 39, and 49-50 need to be determined. The specific analysis is as follows:

[0095] a.Type_Try value is 1-2 (corresponding to binary 000001 and 000010 respectively): Currently, H64 use cases of NALUType 2, 3, and 4 have not been encountered (that is, the binary systems corresponding to NRI&NALU TYPE are 0000010, 0000011, and 0000100 respectively), so it is temporarily assumed that Type_Try value 1-2 is H265.

[0096] b.Type_Try takes a value of 3, which corresponds to 000011 in binary): Currently, we have not encountered an H265 use case of NALU Type 3 (the binary system corresponding to NALU TYPE is 000011), so we assume that 3 is H264.

[0097] c.Type_Try takes values ​​of 5-7 (corresponding to binary values ​​of 000101, 000110, and 000111 respectively): Currently, the H64 use cases of NALU Type 10, 12, and 14 have not been encountered (that is, the binary systems corresponding to NRI&NALU TYPE are 0001010, 0001100, and 0001110 respectively), so 5-7 are temporarily considered to be H265.

[0098] d.Type_Try takes values ​​of 17-18 (corresponding to binary values ​​of 010001 and 010010, respectively): Currently, we have not encountered H64 use cases of NALU Type 2, 3, and 4 (i.e., the binary systems corresponding to NRI & NALU TYPE are 0100010, 0100011, and 0100100, respectively), and we have not encountered N264 use cases with NRI of 1, so we temporarily assume that 17-18 are H265.

[0099] The value of e.Type_Try is 39 (corresponding to the binary system 100111): Currently, the H64 use case of NALU Type 14 has not been encountered (that is, the binary system corresponding to NRI&NALU TYPE is 1001110), and the H165 use case of NALU Type 39 has not been encountered. Based on the principle of higher compatibility, 39 is temporarily considered to be H265.

[0100] f.Type_Try takes values ​​of 49-50 (corresponding to the binary systems of 110001 and 110010 respectively): Currently, the H64 use cases of NALU Type 2, 3, and 4 have not been encountered (that is, the binary systems corresponding to NRI&NALU TYPE are 1100010, 1100011, and 1100100 respectively). Therefore, based on the principle of higher compatibility, 49-50 are temporarily considered to be H265.

[0101] g.Type_Try takes the value 33-34 (corresponding to the binary system 100001 and 100010 respectively): Currently, the H64 use case of NALU Type 2, 3, and 4 has not been encountered (that is, the binary system corresponding to NRI&NALU TYPE is 1000010, 1000011, and 1000100 respectively), and the first packet of the stream is currently used for identification, so 33-34 is temporarily considered to be H265.

[0102] Therefore, through the above steps a to f combined with the historical usage status of H265 and H264, the video frame format list shown in Table 1 can be obtained.

[0103] It is understood that the packet header data includes the payload length and timestamp; based on the packet header data, the audio and video frame type corresponding to the RTP signaling data packet is determined, including one of the following:

[0104] When the effective load length of the packet header data is greater than the preset first byte length, determining the audio and video frame type is video;

[0105] When the timestamp of the RTP signaling data packet is the same as the timestamp of the immediately previous RTP signaling data packet, the audio and video frame type is determined to be video;

[0106] When the timestamp of the RTP signaling data packet is different from the timestamp of the immediately previous RTP signaling data packet, the interval between the two timestamps is greater than or equal to the preset first duration, and the payload length of the RTP signaling data packet and the immediately previous RTP signaling data packet are both greater than the preset silence packet length, the audio and video frame type is determined to be video.

[0107] Exemplarily, for an identified RTP signaling data packet, the RTP signaling data packet is identified as video or audio based on the code stream characteristics. Taking the first byte length as 500 bytes as an example, if the payload length is greater than 500 bytes, it is determined to be video; assuming two consecutive RTP signaling data packets A and B, if the timestamp (corresponding timestamp) of A is the same as the timestamp of B, it is determined to be video; for two consecutive RTP signaling data packets A and B, if the timestamp of A is different from the timestamp of B, the interval between the timestamp of A and the timestamp of B is greater than or equal to 900 and the payload lengths of A and B are both greater than the length of the silence packet, it is determined to be video; if the interval between the timestamp of A and the timestamp of B is less than 900 or at least one of the payload lengths of A and B is less than or equal to the length of the silence packet, it is determined to be audio. The audio packet is discarded and not processed. After being determined to be video, the RTP signaling data packet is subjected to a sequence number continuity check.

[0108] It is understood that the method further comprises at least one of the following:

[0109] The time interval between the start time of the bad frame and the next good key video frame is used as the freeze duration data, and the freeze duration data is output;

[0110] Get the historical number of jams, update the historical number of jams, get the updated number of jams, and output the updated number of jams.

[0111] It should be understood that in some embodiments, the number of freezes will be output; in some embodiments, the number of freezes and the duration of the freeze will be output; in other embodiments, the number of freezes and the duration of the freeze will be output.

[0112] Therefore, in one embodiment, the present invention can effectively detect video freezes experienced by users, including: collecting data at an operator service provider's network access point, obtaining RTP-related code streams, extracting information related to the video call process, and obtaining the following information during the video call: the total number of uplink and downlink video frames, the number of good uplink and downlink video frames, the number of valid uplink and downlink video frames, the number of uplink and downlink video I frames, the number of good uplink and downlink video I frames, the number of P frames, and the number of good uplink and downlink video P frames; and judging the extracted information to obtain the number of uplink and downlink freezes and the duration of the uplink and downlink freezes, thereby detecting the user's video freezes. The present invention can be used by operator service providers to evaluate the call video service quality provided to users at the network access point, providing a basis for improving service quality.

[0113] It can be understood that an electronic device provided according to the third aspect of an embodiment of the present application includes: a memory, a processor, and a computer program stored in the memory and runnable on the processor, and when the processor executes the computer program, it implements a method for identifying video freezes as described in any one of the first aspects above.

[0114] It is understandable that, referring to Figure 8 As shown, in a second aspect, an embodiment of the present application further provides an electronic device, including:

[0115] at least one processor 101;

[0116] At least one memory 102 is used to store at least one program, and when the at least one program is executed by at least one processor 101, any one of the video freeze identification methods of the first aspect is implemented.

[0117] The memory 102 is a non-transient network system that can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory 102 may include a high-speed random access memory and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory 102 may optionally include a memory 102 remotely located relative to the processor 101, and these remote memories 102 may be connected to the processor 101 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0118] The memory 102 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 102 can store an operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 102 and is called by the processor 101 to execute the methods of the embodiments of this application.

[0119] The processor 101 can be implemented using a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application.

[0120] In some embodiments, the electronic device further comprises:

[0121] Input / output interface, used to realize information input and output;

[0122] Communication interface, used to realize communication interaction between this device and other devices, which can be achieved through wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, Wi-Fi, Bluetooth, etc.);

[0123] A bus that transmits information between various components of the device (e.g., processor 101, memory 102, input / output interfaces, and communication interfaces);

[0124] The processor 101 , the memory 102 , the input / output interface and the communication interface can be communicatively connected to each other within the device via a bus.

[0125] In a third aspect, an embodiment of the present application further provides a computer-readable storage medium storing computer-executable instructions, which are used to execute the video freeze identification method of any one of the above-mentioned first aspects.

[0126] In a fourth aspect, an embodiment of the present application further provides a computer program product, including a computer program or computer instructions, which are stored in a computer-readable storage medium. A processor of a computer device reads the computer program or computer instructions from the computer-readable storage medium, and the processor executes the computer program or computer instructions, so that the computer device executes the method for identifying video freezes of any one of the above-mentioned first aspects.

[0127] The system architecture and application scenarios described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application, and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Those skilled in the art will appreciate that with the evolution of the system architecture and the emergence of new application scenarios, the technical solutions provided in the embodiments of the present application are equally applicable to similar technical problems.

[0128] Those skilled in the art will appreciate that all or some of the steps and systems in the method disclosed above can be implemented as software, firmware, hardware, and appropriate combinations thereof. Some physical components or all physical components can be implemented as software executed by a processor, such as a central processing unit, a digital signal processor, or a microprocessor, or implemented as hardware, or implemented as an integrated circuit, such as an application-specific integrated circuit. Such software can be distributed on a computer-readable medium, and the computer-readable medium can include computer storage media (or non-transitory media) and communication media (or temporary media). As known to those skilled in the art, the term computer storage media is included in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data) and is volatile and non-volatile, removable, and non-removable. Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory, or other memory technology, CD-ROM, digital versatile disks (DVD), or other optical disk storage, magnetic cassettes, magnetic tapes, disk storage, or other magnetic storage devices, or any other medium that can be used to store desired information and can be accessed by a computer. Furthermore, as is well known to those skilled in the art, communication media typically embodies computer-readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transport mechanism, and may include any information delivery media.

[0129] The above description of some embodiments of the present application with reference to the accompanying drawings does not limit the scope of the present invention. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and essence of the present invention shall be within the scope of the present application.

Claims

1. A method for identifying video freezes, the method comprising: Get the RTP signaling data packet of the target RTP video stream; When a discontinuous sequence appears in the video frame sequence number corresponding to the RTP signaling data packet, determining that the video frame where the RTP signaling data packet corresponding to the missing sequence between the discontinuous sequences is located is a bad frame; When the time interval between the start time of the bad frame and the start time of the next key video frame which is a good frame is greater than the preset perceptual delay threshold, it is determined that there is a freeze in the target RTP video stream, wherein the video sequence number corresponding to the RTP signaling data packet of the good frame does not have a missing sequence.

2. The method for identifying video freeze according to claim 1, wherein: Before obtaining the RTP signaling data packet of the target RTP video stream, the method further includes: Obtaining a network address parameter of a signaling data packet of an unknown first data flow; Querying a preset database for target network address parameters that match the network address parameters; When the target network address parameters are queried, it is determined that the first data stream is an RTP data stream, or, When the target network address parameter is not found, RTP decoding is performed on the signaling data packet. When the decoding is successful, it is determined that the first data stream is an RTP data stream.

3. The method for identifying video freeze according to claim 2, wherein: After determining that the first data stream is an RTP data stream, the method further includes: The network address parameters of the successfully decoded signaling data packet are stored in the database.

4. The method for identifying video freeze according to claim 1, wherein: Before obtaining the RTP signaling data packet of the target RTP video stream, the method further includes: Get the header data of the RTP signaling data packet of the RTP data stream; Determine the audio and video frame type corresponding to the RTP signaling data packet according to the packet header data; The RTP signaling data packet whose audio and video frame type is video is determined as the RTP signaling data packet of the target RTP video stream.

5. The method for identifying video freeze according to claim 1, wherein: The key video frame is determined by the video frame type of the RTP signaling data packet, and the step of identifying the video frame type includes: Extracting target frame format parameters at the same position as the network abstraction layer unit type in the preset first video format in the payload of the RTP signaling data packet; Matching the target frame format parameters with a preset video frame format list to obtain a target video frame format; The RTP signaling data packet is parsed for a video frame according to the target video frame format to obtain the video frame type.

6. The method for identifying video freeze according to claim 5, wherein: The first video format is an H265 video format. Before parsing the RTP signaling data packet according to the target video frame format, the method further includes: Extracting the target time domain layer label and the target abstract layer identifier from the payload of the RTP signaling data packet at the same position as the abstract layer identifier corresponding to the time domain layer label and the network abstraction layer in the first video format; The performing video frame parsing on the RTP signaling data packet according to the target video frame format to obtain the video frame type includes: Determining whether the target video frame format is valid according to the target temporal layer label and the target abstraction layer identifier; The RTP signaling data packet is parsed for a video frame according to the valid target video frame format to obtain the video frame type.

7. The method for identifying video freeze according to claim 5, wherein: The video frame format list is determined by the following steps: Obtaining, according to the frame format of the video frame supported by the RTP signaling data packet and the value of the network abstraction layer unit type in the first video format, a plurality of to-be-confirmed frame format parameter sets of the to-be-confirmed frame formats; According to the historical usage status of the frame format parameter set to be confirmed, the H265 video format and the H264 video format, the frame format corresponding to each frame format parameter in the frame format parameter set to be confirmed is determined to obtain the video frame format list.

8. The method for identifying video freeze according to claim 4, wherein: The packet header data includes a payload length and a timestamp; and determining, based on the packet header data, the audio and video frame type corresponding to the RTP signaling data packet, includes one of the following: When the payload length of the packet header data is greater than a preset first byte length, determining that the audio and video frame type is video; When the timestamp of the RTP signaling data packet is the same as the timestamp of the immediately previous RTP signaling data packet, determining that the audio and video frame type is video; When the timestamp of the RTP signaling data packet is different from the timestamp of the immediately previous RTP signaling data packet, the interval between the two timestamps is greater than or equal to the preset first duration, and the payload length of the RTP signaling data packet and the immediately previous RTP signaling data packet are both greater than the preset silence packet length, the audio and video frame type is determined to be video.

9. The method for identifying video freeze according to claim 1, wherein: The method further comprises at least one of the following: The time interval between the start time of the bad frame and the start time of the next good key video frame is used as the freeze duration data, and the freeze duration data is output; Obtain the historical number of jams, update the historical number of jams to obtain an updated number of jams, and output the updated number of jams.

10. An electronic device, characterized in that: include: at least one processor; at least one memory for storing at least one program; When at least one of the programs is executed by at least one of the processors, the method according to any one of claims 1 to 9 is implemented.

11. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer-executable instructions, and the computer-executable instructions are used to execute the method according to any one of claims 1 to 9.

12. A computer program product comprising a computer program or computer instructions, characterized in that The computer program or the computer instructions are stored in a computer-readable storage medium, and the processor of the computer device reads the computer program or the computer instructions from the computer-readable storage medium. The processor executes the computer program or the computer instructions, so that the computer device performs: the method according to any one of claims 1 to 9.

Images