Method and apparatus for identifying audio-video desynchronization during video call, and device

Abstract

The present application relates to the technical field of wireless communications. Provided are a method and apparatus for identifying audio-video desynchronization during a video call, and a device. The method comprises: acquiring audio and video RTP messages and audio and video RTCP messages in a video call; on the basis of NTP timestamps of the audio and video RTCP messages in a group of adjacent audio and video RTCP messages, determining a first time difference between the time when a sending end sends the audio RTCP message and the time when the sending end sends the video RTCP message; for at least one group of adjacent audio and video RTP messages subsequent to the group of adjacent audio and video RTCP messages, on the basis of the first time difference, determining a second time difference between the time when the sending end sends the audio RTP message and the time when the sending end sends the video RTP message; on the basis of the second time difference and a third time difference between the time when a collection point receives the audio RTP message and the time when the collection point receives the video RTP message, determining an audio-video desynchronization duration; and on the basis of the audio-video desynchronization duration of each group of adjacent audio and video RTP messages and an audio-video desynchronization threshold, determining an identification result of audio-video desynchronization in the video call. Therefore, the user experience can be effectively improved.

Description

Method, device and equipment for identifying audio and video asynchronization in video call process

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] The present application is based on the Chinese patent application No. 202411242364.0, filed on September 5, 2024, and claims priority to the Chinese patent application No. 202411242364.0, the entire contents of which are incorporated herein by reference. TECHNICAL FIELD

[0003] The present application relates to the technical field of wireless communication, and in particular to a method, device and equipment for identifying audio and video asynchronization in video call process. BACKGROUND

[0004] Audio and video asynchronization is one of the most typical abnormal scenarios in video call, and the problem of audio and video asynchronization in video call process will seriously affect user perception, especially in 5G new call, which has higher real-time requirement, the influence is particularly obvious. However, there is no effective and accurate method to identify audio and video asynchronization in video call process in the industry at present, which leads to the fact that user perception cannot be obtained, and the network side cannot be guaranteed, and the user experience is poor. SUMMARY

[0005] The present application provides a method, device and equipment for identifying audio and video asynchronization in video call process, to solve the problem that related technologies cannot effectively and accurately identify audio and video asynchronization in video call process.

[0006] In a first aspect, the present application provides a method for identifying audio and video asynchronization in video call process, comprising:

[0007] obtaining audio RTP packet, audio RTCP packet, video RTP packet and video RTCP packet in video call process;

[0008] determining a first time difference of sending audio RTCP packet and video RTCP packet by a sending end based on NTP timestamp of audio RTCP packet and NTP timestamp of video RTCP packet in adjacent RTCP packet group of audio and video;

[0009] determining a second time difference of sending audio RTP packet and video RTP packet by the sending end in adjacent RTP packet group of audio and video based on the first time difference for at least one adjacent RTP packet group of audio and video after the adjacent RTCP packet group of audio and video;

[0010] determine a second time difference between sending of the audio RTP packet and the video RTP packet by the sending end in the audio-video adjacent RTP packet group based on the first time difference;

[0011] determine an identification result of audio-video asynchronization in the video call process based on the audio-video asynchronization time length of each of the audio-video adjacent RTP packet groups and an audio-video asynchronization threshold.

[0012] In a second aspect, the present application further provides a device for identifying audio-video asynchronization in a video call process, comprising:

[0013] an obtaining module, configured to obtain an audio RTP packet, an audio RTCP packet, a video RTP packet and a video RTCP packet in a video call process;

[0014] a first determining module, configured to determine a first time difference between sending of the audio RTCP packet and the video RTCP packet by a sending end based on an NTP timestamp of the audio RTCP packet and an NTP timestamp of the video RTCP packet in an audio-video adjacent RTCP packet group;

[0015] a second determining module, configured to determine a second time difference between sending of the audio RTP packet and the video RTP packet by the sending end in an audio-video adjacent RTP packet group based on the first time difference, for at least one audio-video adjacent RTP packet group after the audio-video adjacent RTCP packet group;

[0016] a third determining module, configured to determine an audio-video asynchronization time length of the audio-video adjacent RTP packet group based on the second time difference and a third time difference between receiving of the audio RTP packet and the video RTP packet by a collecting point in the audio-video adjacent RTP packet group;

[0017] a fourth determining module, configured to determine an identification result of audio-video asynchronization in the video call process based on the audio-video asynchronization time length of each of the audio-video adjacent RTP packet groups and an audio-video asynchronization threshold.

[0018] In a third aspect, the present application further provides a collecting device, comprising a memory, a transceiver and a processor;

[0019] the memory is configured to store a computer program; the transceiver is configured to transceive data under control of the processor; and the processor is configured to read the computer program in the memory and perform the following operations:

[0020] obtain an audio RTP packet, an audio RTCP packet, a video RTP packet and a video RTCP packet in a video call process;

[0021] determine a first time difference between the sending of the audio RTCP packet and the video RTCP packet by the sending end based on the NTP timestamp of the audio RTCP packet and the NTP timestamp of the video RTCP packet in the audio-video adjacent RTCP packet group;

[0022] determine a second time difference between the sending of the audio RTP packet and the video RTP packet in the audio-video adjacent RTP packet group by the sending end based on the first time difference for at least one audio-video adjacent RTP packet group after the audio-video adjacent RTCP packet group;

[0023] determine an audio-video asynchronization duration of the audio-video adjacent RTP packet group based on the second time difference and a third time difference between the receiving of the audio RTP packet and the video RTP packet in the audio-video adjacent RTP packet group by the collecting point;

[0024] determine an identification result of the audio-video asynchronization in the video call process based on the audio-video asynchronization duration of each audio-video adjacent RTP packet group and an audio-video asynchronization threshold.

[0025] In a fourth aspect, the present application further provides an electronic device, comprising a processor and a memory storing a computer program, wherein the processor implements the method for identifying the audio-video asynchronization in the video call process according to the first aspect when executing the program.

[0026] In a fifth aspect, the present application further provides a processor-readable storage medium, which stores a computer program, wherein the computer program is executed by a processor to implement the method for identifying the audio-video asynchronization in the video call process according to the first aspect.

[0027] In a sixth aspect, the present application further provides a computer program product, comprising a computer program, wherein the computer program is executed by a processor to implement the method for identifying the audio-video asynchronization in the video call process according to the first aspect.

[0028] The method, device and equipment for identifying audio and video asynchronization in a video call process provided in the application, by acquiring audio RTP messages, audio RTCP messages, video RTP messages and video RTCP messages in the video call process, determining a first time difference of sending audio RTCP messages and video RTCP messages by a sending end based on the NTP time stamp of the audio RTCP message and the NTP time stamp of the video RTCP message in the adjacent RTCP message group of audio and video, then determining a second time difference of sending audio RTP messages and video RTP messages in the adjacent RTP message group of audio and video by the sending end based on the first time difference for at least one adjacent RTP message group of audio and video after the adjacent RTCP message group of audio and video, determining the audio and video asynchronization duration of the adjacent RTP message group of audio and video based on the second time difference and a third time difference of receiving the audio RTP messages and the video RTP messages in the adjacent RTP message group of audio and video by the collection point, and finally determining the identification result of the audio and video asynchronization in the video call process based on the audio and video asynchronization duration of each adjacent RTP message group of audio and video and the audio and video asynchronization threshold, thereby solving the problem that the related art cannot effectively and accurately identify the audio and video asynchronization in the video call process, so that the network side can specifically guarantee the video call experience of the user, which is beneficial to improving the user experience. BRIEF DESCRIPTION OF DRAWINGS

[0029] In order to more clearly illustrate the technical solutions in the application or the related art, the drawings needed to be used in the embodiments or the related art description will be briefly introduced. Obviously, the drawings in the following description are some embodiments of the application, and other drawings can be obtained by those skilled in the art without creative labor.

[0030] FIG. 1 is a flowchart of a method for identifying audio and video asynchronization in a video call process according to an embodiment of the application.

[0031] FIG. 2 is a schematic diagram of an adjacent RTCP message group of audio and video according to an embodiment of the application.

[0032] FIG. 3 is a schematic diagram of an adjacent RTP message group of audio and video according to an embodiment of the application.

[0033] FIG. 4 is a T value calculation diagram according to an embodiment of the application. av2

[0034] FIG. 5 is a T value calculation diagram according to an embodiment of the application. av1

[0035] FIG. 6 is an example diagram of message transmission in a video call process according to an embodiment of the application.

[0036] ​​FIG. 7 is an example of an algorithm for identifying audio and video asynchronization in a video call according to an embodiment of the present application.

[0037] FIG. 8 is a structural diagram of an apparatus for identifying audio and video asynchronization in a video call according to an embodiment of the present application.

[0038] FIG. 9 is a structural diagram of a collection device according to an embodiment of the present application.

[0039] FIG. 10 is a structural diagram of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0040] The term "and / or" in the embodiments of the present application describes the association relationship of the associated objects, which means that there can be three relationships, for example, A and / or B can mean that A exists alone, A and B exist together, and B exists alone. The character " / " generally represents an "or" relationship between the associated objects before and after it.

[0041] The term "multiple" in the embodiments of the present application means two or more, and other quantifiers are similar.

[0042] The terms "first", "second", and the like in the embodiments of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the terms used in this way can be interchanged under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described here, and the objects distinguished by "first", "second" are usually of the same type and do not limit the number of objects, for example, the first object can be one or more.

[0043] The technical solutions in the embodiments of the present application will be described clearly and completely below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, not all. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative labor fall within the scope of the present application.

[0044] In order to more clearly understand the technical solutions of the embodiments of the present application, first, some technical contents related to the embodiments of the present application are briefly described.

[0045] RTP: Real-time Transport Protocol (RTP), RTP packet is used to transmit media data such as audio and video over the Internet.

[0046] RTCP: Real-time Transport Control Protocol, is a control protocol for RTP message transmission. The main task of RTCP is to monitor the quality of RTP data transmission, and provide statistical information of terminal sending and receiving.

[0047] NTP Timestamp: Network Time Protocol, NTP timestamp, NTP timestamp is an absolute timestamp. The length of NTP timestamp is 64 bits, of which the high 32 bits are MSW (Most Significant Word) and the low 32 bits are LSW (Least Significant Word). The unit of MSW is second and the unit of LSW is picosecond.

[0048] RTP Timestamp: RTP timestamp is a relative timestamp, which can be used to represent the sampling interval of the message.

[0049] SIP: Session Initiation Protocol, in this application, whether the video call is answered can be determined by analyzing the SIP message, for example, the video call can be determined to be answered by "200 (OK)" in the SIP message.

[0050] FIG. 1 is a flowchart of a method for identifying audio and video asynchronization in a video call process according to an embodiment of the present application. As shown in FIG. 1, the method comprises operations 101-105.

[0051] Operation 101: Obtain audio RTP message, audio RTCP message, video RTP message and video RTCP message in the video call process.

[0052] In some embodiments, the execution subject of the method can be a collection device of a deep packet inspection (DPI) system. The collection device can obtain X-Detailed Record (XDR) data collected by a plurality of collection points deployed in a core network, and analyze and mine based on the XDR data to quickly and accurately locate the cause of service anomaly.

[0053] For the video call service, the collection device can parse the RTP message, RTCP message, SIP message and other message information in the video call process by deep analysis, output the identification result of audio and video asynchronization in the video call process according to the characteristics of audio and video synchronization, and the technical scheme of the application has important reference value for video call service perception evaluation, network operation and maintenance and user complaint handling in each period.

[0054] In some embodiments, for the 5G new call service, the RTP and RTCP data of the 5G new call media in the existing network can be collected to obtain the audio stream message (including audio RTP message and audio RTCP message) and video stream message (including video RTP message and video RTCP message) in the video call process, and the messages are parsed to obtain the basic information for identifying the audio and video asynchronization problem in the video call process. For example, the basic information includes: sequence number of the message, audio stream sampling frequency, video stream sampling frequency, synchronization source identifier (SSRC), RTP timestamp of the RTP message, NTP timestamp of the RTCP message, RTP timestamp of the RTCP message, etc.

[0055] Operation 102, based on the NTP timestamp of the audio RTCP message and the NTP timestamp of the video RTCP message in the audio and video adjacent RTCP message group, determining the first time difference of the sending end sending the audio RTCP message and the video RTCP message.

[0056] In some embodiments, in the video call process, the sending end will intermittently send RTCP messages in the process of sending audio RTP messages and video RTP messages, and the RTCP messages carry NTP timestamps and RTP timestamps and other information. Based on the NTP timestamp of the audio RTCP message and the NTP timestamp of the video RTCP message in the audio and video adjacent RTCP message group, the collection device can determine the time difference of the sending end sending the audio RTCP message and the video RTCP message, that is, the first time difference, which can be denoted as T diff Reference.

[0057] Among them, the audio and video adjacent RTCP message group refers to a group of adjacent audio RTCP messages and video RTCP messages received by the collection point in sequence. It should be noted that the order of receiving the audio RTCP message and the video RTCP message in the audio and video adjacent RTCP message group by the collection point is not limited, which can be audio RTCP message first and video RTCP message last, or video RTCP message first and audio RTCP message last.

[0058] Since the NTP timestamp of the RTCP packet is an absolute timestamp, only the NTP timestamps of the audio RTCP packet and the video RTCP packet in the adjacent audio-video RTCP packet group are converted into the same unit and then subtracted to obtain the first time difference T of the audio RTCP packet and the video RTCP packet sent by the sending end diff .

[0059] In some embodiments, the NTP timestamp can be converted into the second unit and then subtracted. The conversion logic of the NTP time is as follows: the MSW is in the second unit (the NTP timestamp starts from 1900-01-01 00:00), the LSW is in the picosecond unit, 1,000,000,000,000 / (2 32 ) = 232.83064365386962890625, so the LSW is multiplied by 232.83 and then divided by 10 9 to obtain the time value in the millisecond unit. According to the conversion logic, the NTP timestamps of the audio RTCP packet and the video RTCP packet in the adjacent audio-video RTCP packet group are converted into the second unit and then subtracted to obtain the first time difference T of the audio RTCP packet and the video RTCP packet sent by the sending end diff .

[0060] It should be noted that the embodiments of the present application are described by taking a certain transmission direction as an example, and the transmission direction is not limited, which can be an uplink direction or a downlink direction. The sending end in the uplink direction can be a terminal (or user equipment, UE), and the sending end in the downlink direction can be a network device.

[0061] In some embodiments, the adjacent audio-video RTCP packet group satisfies the following conditions:

[0062] An adjacent audio-video RTCP packet group includes one audio RTCP packet and one video RTCP packet, and there is no other RTCP packet and RTP packet between the one audio RTCP packet and the one video RTCP packet;

[0063] The RTCP packets before the first adjacent audio-video RTCP packet group can all be audio RTCP packets or video RTCP packets;

[0064] One RTCP packet cannot belong to two adjacent audio-video RTCP packet groups;

[0065] The RTCP packets between the adjacent two adjacent audio-video RTCP packet groups can all be audio RTCP packets or video RTCP packets.

[0066] By determining the audio-video adjacent RTCP packet group according to the above conditions, the effectiveness and accuracy of the technical solution of the present application in identifying the audio-video synchronization problem in the video call process can be better ensured.

[0067] Fig. 2 is a schematic diagram of an audio-video adjacent RTCP packet group provided by an embodiment of the present application. As shown in Fig. 2, after the video call response, the collection device can receive a plurality of audio RTCP packets and a plurality of video RTCP packets. Since the audio stream and the video stream belong to relatively independent data streams, there can be a certain time difference between the audio RTCP packet and the video RTCP packet, and there can be a plurality of continuous audio RTCP packets or video RTCP packets. The time axis in the figure refers to the time axis of the collection point (or collection device) receiving the packet.

[0068] In Fig. 2, rtcp_a_1, rtcp_a_2, rtcp_a_3, and rtcp_a_4 are each an audio RTCP packet, and rtcp_v_1, rtcp_v_2, rtcp_v_3, rtcp_v_4, and rtcp_v_5 are each a video RTCP packet. Among them, rtcp_a_2 and rtcp_v_1 form an audio-video adjacent RTCP packet group, rtcp_v_4 and rtcp_a_3 form an audio-video adjacent RTCP packet group, and rtcp_a_4 and rtcp_v_5 form an audio-video adjacent RTCP packet group.

[0069] As can be seen, each audio-video adjacent RTCP packet group in Fig. 2 includes an audio RTCP packet and a video RTCP packet, and there is no other RTCP packet and RTP packet between the audio RTCP packet and the video RTCP packet in each audio-video adjacent RTCP packet group. In addition, none of the RTCP packets belongs to two audio-video adjacent RTCP packet groups, that is, any two audio-video adjacent RTCP packet groups do not overlap.

[0070] For the first audio-video adjacent RTCP packet group, there can be RTCP packets before the first audio-video adjacent RTCP packet group or there can be no RTCP packets before the first audio-video adjacent RTCP packet group. If there are RTCP packets, they can only be all audio RTCP packets or all video RTCP packets. For example, in Fig. 2, the first audio-video adjacent RTCP packet group is the packet group formed by rtcp_a_2 and rtcp_v_1, and the RTCP packet before the packet group is an audio RTCP packet.

[0071] For two adjacent audio-video adjacent RTCP message groups, there can be RTCP messages between the two adjacent audio-video adjacent RTCP message groups, or there can be no RTCP messages between the two adjacent audio-video adjacent RTCP message groups. If there are RTCP messages, all the RTCP messages can be audio RTCP messages or video RTCP messages. As shown in FIG. 2, all the messages between the message group composed of “rtcp_a_2 and rtcp_v_1” and the message group composed of “rtcp_v_4 and rtcp_a_3” are video RTCP messages, and there are no messages between the message group composed of “rtcp_v_4 and rtcp_a_3” and the message group composed of “rtcp_a_4 and rtcp_v_5”.

[0072] It should be noted that the audio-video adjacent RTCP message group can be the first audio-video adjacent RTCP message group that appears at the first collection point after the video call response, or the audio-video adjacent RTCP message group can be refreshed every time a new audio-video adjacent RTCP message group is received. The RTCP messages in each audio-video adjacent RTCP message group in FIG. 2 are the first adjacent RTCP messages, which means that FIG. 2 corresponds to the case of receiving the RTCP messages at the first collection point. It can be understood that, due to the possibility of out-of-order transmission, the order of the RTCP messages received at other collection points can be different from that shown in FIG. 2.

[0073] Operation 103: For at least one audio-video adjacent RTP message group after the audio-video adjacent RTCP message group, a second time difference between the audio RTP message and the video RTP message sent by the sending end in the audio-video adjacent RTP message group is determined based on the first time difference.

[0074] In some embodiments, the audio-video adjacent RTP message group has a similar meaning to the audio-video adjacent RTCP message group described above, and refers to a group of adjacent audio RTP messages and video RTP messages received by the collection point in sequence. The order of the audio RTP message and the video RTP message received by the collection point in the audio-video adjacent RTP message group is not limited, and can be that the audio RTP message is first and the video RTP message is second, or the video RTP message is first and the audio RTP message is second.

[0075] After the collection point receives the audio-video adjacent RTCP message group, the detection of the RTP message is triggered, and an audio-video out-of-sync duration can be calculated for each audio-video adjacent RTP message group received subsequently. Since the RTP timestamp in the RTP message is a relative timestamp, the first time difference obtained in operation 102 is used to determine a theoretical time difference between the audio RTP message and the video RTP message sent by the sending end in the audio-video adjacent RTP message group, that is, a second time difference. Hereinafter, the second time difference can be denoted as T av1 .

[0076] The method for calculating the second time difference can be various, which is not limited here, and various embodiments for calculating the second time difference will be provided below.

[0077] In some embodiments, the audio-video adjacent RTP packet groups satisfy the following conditions:

[0078] An audio-video adjacent RTP packet group includes one audio RTP packet and one video RTP packet, and there is no other RTP packet and RTCP packet between an audio RTP packet and a video RTP packet;

[0079] The RTP packets before the first audio-video adjacent RTP packet group can all be audio RTP packets or video RTP packets;

[0080] An RTP packet cannot belong to two audio-video adjacent RTP packet groups;

[0081] The RTP packets between the adjacent two audio-video adjacent RTP packet groups can all be audio RTP packets or video RTP packets.

[0082] By determining the audio-video adjacent RTP packet groups according to the above conditions, the effectiveness and accuracy of the technical solution of the present application in identifying the audio-video unsynchronization problem in the video call process can be better ensured.

[0083] FIG. 3 is a schematic diagram of an audio-video adjacent RTP packet group provided by an embodiment of the present application. As shown in FIG. 3, after the video call is answered, the collection device can receive a plurality of audio RTP packets and a plurality of video RTP packets. Since the audio stream and the video stream belong to relatively independent data streams, there can be a certain time difference between the audio RTP packets and the video RTP packets, and there can be a plurality of continuous audio RTP packets or video RTP packets. The time axis in the figure refers to the time axis of the collection point (or collection device) receiving the packets.

[0084] In FIG. 3, audio seq1, audio seq2, audio seq3, and audio seq4 are each an audio RTP packet, and video seq1, video seq2, video seq3, video seq4, and video seq5 are each a video RTP packet. Among them, audio seq2 and video seq1 form an audio-video adjacent RTP packet group, video seq4 and audio seq3 form an audio-video adjacent RTP packet group, and audio seq4 and video seq5 form an audio-video adjacent RTP packet group.

[0085] As can be seen, each audio-video-adjacent RTP packet group in Figure 3 includes one audio RTP packet and one video RTP packet, and there is no other RTP packet and RTCP packet between the audio RTP packet and the video RTP packet in each audio-video-adjacent RTP packet group. In addition, none of the RTP packets belongs to two audio-video-adjacent RTP packet groups, that is, any two audio-video-adjacent RTP packet groups have no overlapping packets.

[0086] For the first audio-video-adjacent RTP packet group, there can be RTP packets before the first audio-video-adjacent RTP packet group, and if there are RTP packets, they can only be all audio RTP packets or all video RTP packets. For example, in Figure 3, the first audio-video-adjacent RTP packet group is the packet group composed of audio seq2 and video seq1, and the RTP packets before the packet group are audio RTP packets.

[0087] For the two adjacent audio-video-adjacent RTP packet groups, there can be RTP packets between the two adjacent audio-video-adjacent RTP packet groups, and if there are RTP packets, they can only be all audio RTP packets or all video RTP packets. For example, in Figure 3, between the packet group composed of "audio seq2 and video seq1" and the packet group composed of "video seq4 and audio seq3", there are all video RTP packets, and between the packet group composed of "video seq4 and audio seq3" and the packet group composed of "audio seq4 and video seq5", there are no packets.

[0088] Operation 104: determining the audio-video-asynchronous duration of the audio-video-adjacent RTP packet group based on the second time difference and a third time difference of the audio RTP packet and the video RTP packet in the audio-video-adjacent RTP packet group received at the collection point.

[0089] In some embodiments, the third time difference can be calculated according to the respective receiving time (or collection time) of the audio RTP packet and the video RTP packet in the audio-video-adjacent RTP packet group. For example, assuming that the collection times of the audio RTP packet and the video RTP packet in the audio-video-adjacent RTP packet group are rtp.time_a and rtp.time_v respectively, the result of rtp.time_v minus rtp.time_a can be taken as the third time difference, which can be denoted as Tdiff in the following. av2 As shown in Figure 4. In Figure 4, rtp_a_1, rtp_a_n are each an audio RTP packet, rtp_v_1, rtp_v_n are each a video RTP packet, rtcp_a_1 is an audio RTCP packet, and rtcp_v_1 is a video RTCP packet.

[0090] In some embodiments, based on the second time difference and a third time difference of a time difference of receiving an audio RTP packet and a video RTP packet in the audio-video adjacent RTP packet group at the collection point, a length of audio-video asynchronization of the audio-video adjacent RTP packet group is determined, including:

[0091] A difference between the second time difference and the third time difference is determined as the length of audio-video asynchronization of the audio-video adjacent RTP packet group.

[0092] For example, assuming that T avsync represents the length of audio-video asynchronization of the audio-video adjacent RTP packet group, T avsync may be equal to T av2 -T av1 .

[0093] Operation 105, based on the length of audio-video asynchronization of each audio-video adjacent RTP packet group and an audio-video asynchronization threshold, determining a recognition result of audio-video asynchronization in the video call process.

[0094] In some embodiments, the audio-video asynchronization threshold is not specifically limited in the present application. In some embodiments, the audio-video asynchronization threshold can be a length threshold at which a user can perceive that there is audio-video asynchronization phenomenon from the picture of the video and the sound of the audio. For example, due to the buffer area of the terminal in the existing network, generally, 100 ms (milliseconds) of audio leading or 200 ms of audio lagging compared with video does not affect the user experience. Therefore, a reasonable audio-video asynchronization threshold can be set according to this.

[0095] When judging the audio-video asynchronization problem in the video call process, the length of audio-video asynchronization of each audio-video adjacent RTP packet group calculated can be processed in a statistical manner, such as taking a maximum value or other statistical values, which is not limited herein. Then, the value is compared with the audio-video asynchronization threshold to determine whether there is an audio-video asynchronization problem in the video call process.

[0096] The method for identifying audio and video asynchronization in a video call process provided by the embodiments of the present application comprises the following steps: acquiring audio RTP packets, audio RTCP packets, video RTP packets and video RTCP packets in the video call process; determining a first time difference between the sending of the audio RTCP packets and the video RTCP packets by the sending end based on the NTP time stamp of the audio RTCP packet and the NTP time stamp of the video RTCP packet in the adjacent audio and video RTCP packet group; then, for at least one adjacent audio and video RTP packet group after the adjacent audio and video RTCP packet group, determining a second time difference between the sending of the audio RTP packet and the video RTP packet in the adjacent audio and video RTP packet group by the sending end based on the first time difference; then, based on the second time difference and a third time difference between the receiving of the audio RTP packet and the video RTP packet in the adjacent audio and video RTP packet group by the collection point, determining the audio and video asynchronization duration of the adjacent audio and video RTP packet group; and finally, based on the audio and video asynchronization duration of each adjacent audio and video RTP packet group and an audio and video asynchronization threshold, determining the identification result of the audio and video asynchronization in the video call process, thereby solving the problem that the related art cannot effectively and accurately identify the audio and video asynchronization in the video call process, and enabling the network side to specifically guarantee the video call experience of the user, which is beneficial to improving the user experience.

[0097] In some embodiments, based on the first time difference, the second time difference between the sending of the audio RTP packet and the video RTP packet in the adjacent audio and video RTP packet group by the sending end is determined, comprising:

[0098] Based on the first time difference, the RTP time stamp of the audio RTCP packet, the RTP time stamp of the video RTCP packet, the RTP time stamp of the audio RTP packet, the RTP time stamp of the video RTP packet, the audio stream sampling frequency and the video stream sampling frequency, the second time difference between the sending of the audio RTP packet and the video RTP packet by the sending end is determined.

[0099] It should be noted that the audio RTCP packet and the video RTCP packet here refer to the audio RTCP packet and the video RTCP packet in the adjacent audio and video RTCP packet group in the foregoing, and the audio RTP packet and the video RTP packet are the audio RTP packet and the video RTP packet in the adjacent audio and video RTP packet group for which the audio and video asynchronization duration needs to be calculated.

[0100] In the embodiment, the second time difference of sending the audio RTP packet and the video RTP packet by the sending end can be derived based on the first time difference calculated in the foregoing, the RTP timestamp of the audio RTCP packet, the RTP timestamp of the video RTCP packet, the RTP timestamp of the audio RTP packet, the RTP timestamp of the video RTP packet, the audio stream sampling frequency and the video stream sampling frequency, thereby solving the problem that the second time difference of sending the audio RTP packet and the video RTP packet by the sending end cannot be calculated according to the RTP timestamp in the RTP packet.

[0101] In some embodiments, determining the second time difference of sending the audio RTP packet and the video RTP packet by the sending end based on the first time difference, the RTP timestamp of the audio RTCP packet, the RTP timestamp of the video RTCP packet, the RTP timestamp of the audio RTP packet, the RTP timestamp of the video RTP packet, the audio stream sampling frequency and the video stream sampling frequency comprises:

[0102] determining a fourth time difference of sending the audio RTP packet and the audio RTCP packet by the sending end based on the RTP timestamp of the audio RTCP packet, the RTP timestamp of the audio RTP packet and the audio stream sampling frequency, and determining a fifth time difference of sending the video RTP packet and the video RTCP packet by the sending end based on the RTP timestamp of the video RTCP packet, the RTP timestamp of the video RTP packet and the video stream sampling frequency;

[0103] determining the second time difference of sending the audio RTP packet and the video RTP packet by the sending end based on the first time difference, the fourth time difference and the fifth time difference.

[0104] In some embodiments, there are multiple ways to calculate the second time difference, in some implementations, the fourth time difference of sending the audio RTP packet and the audio RTCP packet by the sending end and the fifth time difference of sending the video RTP packet and the video RTCP packet by the sending end can be calculated respectively, and then the second time difference can be calculated based on the first time difference, the fourth time difference and the fifth time difference, thereby improving the algorithm efficiency.

[0105] In some embodiments, the calculation formula of the fourth time difference is: T1=(rtp.timestamp_a-rtcp.timestamp_a) / samplerate_a

[0106] In the formula, T1 represents the fourth time difference, rtp.timestamp_a represents the RTP timestamp of the audio RTP packet, rtcp.timestamp_a represents the RTP timestamp of the audio RTCP packet, and samplerate_a represents the audio stream sampling frequency.

[0107] It should be noted that the formula of the fourth time difference can be other equivalent or modified formulae other than the above formula, and the present application does not limit this.

[0108] In some embodiments, the formula of the fifth time difference is: T2= (rtp.timestamp_v-rtcp.timestamp_v) / samplerate_v

[0109] In the formula, T2 represents the fifth time difference, rtp.timestamp_v represents the RTP timestamp of the video RTP packet, rtcp.timestamp_v represents the RTP timestamp of the video RTCP packet, and samplerate_v represents the video stream sampling frequency.

[0110] It should be noted that the formula of the fifth time difference can be other equivalent or modified formulae other than the above formula, and the present application does not limit this.

[0111] In some embodiments, the formula of the second time difference is: T av1 = T2+T diff -T1

[0112] In the formula, T av1 represents the second time difference, T2 represents the fifth time difference, T diff represents the first time difference, and T1 represents the fourth time difference.

[0113] FIG. 5 is a schematic diagram of T av1 value calculation provided by the embodiments of the present application. As shown in FIG. 5, after the video call is answered, the first audio RTP packet sent by the terminal (taking the uplink direction as an example) is recorded as rtp_a_1, and the first video RTP packet is recorded as rtp_v_1; the first audio RTCP packet sent by the terminal is recorded as rtcp_a_1, and the first video RTCP packet is recorded as rtcp_v_1; the audio RTP packet and the video RTP packet sent by the terminal after receiving the audio RTCP packet and the video RTCP packet are recorded as rtp_a_n and rtp_v_n respectively. The second time difference T av1 = T2+T diff -T1.

[0114] It should be noted that the formula of the second time difference can be other equivalent or modified formulae other than the above formula, and the present application does not limit this.

[0115] In some embodiments, the identification result of the audio-video asynchronization in the video call process is determined based on the audio-video asynchronization time length of each audio-video adjacent RTP message group and the audio-video asynchronization threshold, and the identification result comprises:

[0116] determining the maximum value in the audio-video asynchronization time length of each audio-video adjacent RTP message group;

[0117] if the maximum value is greater than the audio-video asynchronization threshold, it is determined that there is an audio-video asynchronization problem in the video call process, and the maximum value is determined as the asynchronization time length of the audio-video asynchronization problem.

[0118] For example, assuming that there are n audio-video asynchronization time lengths of audio-video adjacent RTP message groups, respectively denoted as T avsync 1, T avsync 2, T avsync 3, …, T avsync n, the maximum value in the audio-video asynchronization time length can be taken, and then the maximum value is compared with the audio-video asynchronization threshold. If the maximum value is greater than the audio-video asynchronization threshold, it can be determined that there is an audio-video asynchronization problem in the video call process, and the maximum value is determined as the asynchronization time length of the audio-video asynchronization problem.

[0119] By comparing the maximum value in the audio-video asynchronization time length of each audio-video adjacent RTP message group with the audio-video asynchronization threshold, it can be determined whether there is an audio-video asynchronization problem in the video call process, which can maximize the identification of the audio-video asynchronization problem in the video call process, thereby improving the user experience.

[0120] In some embodiments, the method further comprises: if the maximum value is less than or equal to the audio-video asynchronization threshold, it is determined that there is no audio-video asynchronization problem in the video call process. In the case that the maximum value in the audio-video asynchronization time length of the audio-video adjacent RTP message group is less than or equal to the audio-video asynchronization threshold, it is directly determined that there is no audio-video asynchronization problem in the video call process, which can improve the identification efficiency and reduce the complexity of the identification model.

[0121] The method provided by the above embodiments of the present application is illustrated by a specific example in combination with FIG. 6 and FIG. 7. In this example, the theoretical time difference T av1 value of the audio-video RTP message sent by the terminal and the actual time difference T av2 value of the audio-video RTP message received by the collection device, i.e., the audio-video asynchronization time length T avsync is obtained, and finally, in combination with the audio-video asynchronization threshold determined according to the tolerance of ordinary people to sound and picture, it is determined whether there is an audio-video asynchronization problem in the video call service and the asynchronization time length of the audio-video asynchronization problem.

[0122] It should be noted that although FIG. 6 only describes the scenario of uplink packets, the technical solution of the present application is also applicable to the scenario of downlink packets. Referring to FIG. 7, the algorithm flow for identifying the audio-video asynchronization in the video call mainly includes:

[0123] Operation 1, after the video call is answered, the adjacent audio RTCP packet and video RTCP packet are received by the collection device. Since the audio stream and the video stream are relatively independent data streams, there will be a certain time difference between the audio RTCP packet and the video RTCP packet, and there will be a phenomenon of continuous multiple audio RTCP packets or video RTCP packets, so the adjacent RTCP packets of audio and video are extracted for calculation. Based on the NTP time stamp in the adjacent RTCP packets of audio and video, T diff is calculated.

[0124] Operation 2, after receiving the packet of operation 1, the detection of the RTP packet is triggered at the same time, and the adjacent audio RTP packet and video RTP packet are received by the collection device. Similar to the RTCP packet scenario, there will be a certain time difference between the audio RTP packet and the video RTP packet, and there will be a phenomenon of continuous multiple audio RTP packets or video RTP packets, so the adjacent RTP packets of audio and video are extracted for calculation, and T av1 and T av2 are obtained.

[0125] Operation 3, each adjacent RTP packet of audio and video can calculate an audio-video asynchronization duration, which is respectively denoted as T avsync 1, T avsync 2, …, T avsync n.

[0126] Operation 4, the maximum audio-video asynchronization duration T avsync is compared with the audio-video asynchronization threshold, if the maximum T avsync is greater than the threshold, it is judged that the video call has the audio-video asynchronization problem, and the audio-video asynchronization duration is output, that is, the maximum T avsync value; otherwise, it is judged that the video call does not have the audio-video asynchronization problem, and the previously calculated audio-video asynchronization duration can be cleared.

[0127] In addition, the average value of each audio-video asynchronization duration can also be used to measure the relative fluctuation of the whole audio stream and the video stream.

[0128] The technical scheme provided in the application can output the identification result of audio and video asynchronization in the video call process according to the characteristics of audio and video synchronization, without relying on additional test instrument terminals or test samples, and can complete the evaluation and analysis of the audio and video asynchronization problem based on the software algorithm upgrade of the existing DPI system, can meet the analysis of the audio and video asynchronization of all video call users in the existing network, greatly saves the investment cost, and the identification result is more accurate and reliable.

[0129] The device for identifying audio and video asynchronization in the video call process provided in the embodiments of the application is described below, and the device for identifying audio and video asynchronization in the video call process described below can be mutually referred to the method for identifying audio and video asynchronization in the video call process described above.

[0130] FIG. 8 is a structural schematic diagram of the device for identifying audio and video asynchronization in the video call process provided in the embodiments of the application, as shown in FIG. 8, the device comprises:

[0131] The acquisition module 810 is configured to acquire the audio RTP message, the audio RTCP message, the video RTP message and the video RTCP message in the video call process.

[0132] The first determination module 820 is configured to determine the first time difference of the sending end for sending the audio RTCP message and the video RTCP message based on the NTP time stamp of the audio RTCP message and the NTP time stamp of the video RTCP message in the adjacent audio and video RTCP message group.

[0133] The second determination module 830 is configured to determine the second time difference of the sending end for sending the audio RTP message and the video RTP message in the adjacent audio and video RTP message group based on the first time difference for at least one adjacent audio and video RTP message group after the adjacent audio and video RTP message group.

[0134] The third determination module 840 is configured to determine the audio and video asynchronization duration of the adjacent audio and video RTP message group based on the second time difference and the third time difference of the audio RTP message and the video RTP message received by the collection point in the adjacent audio and video RTP message group.

[0135] The fourth determination module 850 is configured to determine the identification result of the audio and video asynchronization in the video call process based on the audio and video asynchronization duration of each adjacent audio and video RTP message group and the audio and video asynchronization threshold.

[0136] In some embodiments, the second time difference of the sending end for sending the audio RTP message and the video RTP message in the adjacent audio and video RTP message group is determined based on the first time difference, comprising:

[0137] determine a second time difference between sending of the audio RTP packet and the video RTP packet by the sending end based on the first time difference, the RTP timestamp of the audio RTCP packet, the RTP timestamp of the video RTCP packet, the RTP timestamp of the audio RTP packet, the RTP timestamp of the video RTP packet, the audio stream sampling frequency and the video stream sampling frequency.

[0138] In some embodiments, determining the second time difference between sending of the audio RTP packet and the video RTP packet by the sending end based on the first time difference, the RTP timestamp of the audio RTCP packet, the RTP timestamp of the video RTCP packet, the RTP timestamp of the audio RTP packet, the RTP timestamp of the video RTP packet, the audio stream sampling frequency and the video stream sampling frequency comprises:

[0139] determine a fourth time difference between sending of the audio RTP packet and the audio RTCP packet by the sending end based on the RTP timestamp of the audio RTCP packet, the RTP timestamp of the audio RTP packet and the audio stream sampling frequency, and determine a fifth time difference between sending of the video RTP packet and the video RTCP packet by the sending end based on the RTP timestamp of the video RTCP packet, the RTP timestamp of the video RTP packet and the video stream sampling frequency;

[0140] determine the second time difference between sending of the audio RTP packet and the video RTP packet by the sending end based on the first time difference, the fourth time difference and the fifth time difference.

[0141] In some embodiments, the formula for calculating the fourth time difference is: T1 = (rtp.timestamp_a-rtcp.timestamp_a) / samplerate_a

[0142] wherein T1 represents the fourth time difference, rtp.timestamp_a represents the RTP timestamp of the audio RTP packet, rtcp.timestamp_a represents the RTP timestamp of the audio RTCP packet, and samplerate_a represents the audio stream sampling frequency.

[0143] In some embodiments, the formula for calculating the fifth time difference is: T2 = (rtp.timestamp_v-rtcp.timestamp_v) / samplerate_v

[0144] wherein T2 represents the fifth time difference, rtp.timestamp_v represents the RTP timestamp of the video RTP packet, rtcp.timestamp_v represents the RTP timestamp of the video RTCP packet, and samplerate_v represents the video stream sampling frequency.

[0145] In some embodiments, the second time difference is calculated according to the following formula: T av1 = T2+ T diff -T1

[0146] In the formula, T av1 represents the second time difference, T2 represents the fifth time difference, T diff represents the first time difference, and T1 represents the fourth time difference.

[0147] In some embodiments, based on the second time difference and a third time difference of audio RTP packets and video RTP packets in the audio-video adjacent RTP packet group received by the collection point, a length of audio-video asynchronization of the audio-video adjacent RTP packet group is determined, including:

[0148] A difference between the second time difference and the third time difference is determined as the length of audio-video asynchronization of the audio-video adjacent RTP packet group.

[0149] In some embodiments, based on the length of audio-video asynchronization of each audio-video adjacent RTP packet group and an audio-video asynchronization threshold, an identification result of audio-video asynchronization in the video call process is determined, including:

[0150] A maximum value in the length of audio-video asynchronization of each audio-video adjacent RTP packet group is determined.

[0151] If the maximum value is greater than the audio-video asynchronization threshold, it is determined that there is an audio-video asynchronization problem in the video call process, and the maximum value is determined as the asynchronization length of the audio-video asynchronization problem.

[0152] In some embodiments, the fourth determination module 850 is further configured to:

[0153] If the maximum value is less than or equal to the audio-video asynchronization threshold, it is determined that there is no audio-video asynchronization problem in the video call process.

[0154] In some embodiments, the audio-video adjacent RTCP packet group satisfies the following conditions:

[0155] An audio-video adjacent RTCP packet group includes one audio RTCP packet and one video RTCP packet, and there is no other RTCP packet and RTP packet between the one audio RTCP packet and the one video RTCP packet;

[0156] RTCP packets before the first audio-video adjacent RTCP packet group can all be audio RTCP packets or video RTCP packets;

[0157] One RTCP packet cannot belong to two audio-video adjacent RTCP packet groups;

[0158] The RTP packets between two adjacent audio-video adjacent RTP packet groups can only be all audio RTP packets or all video RTP packets.

[0159] In some embodiments, the audio-video adjacent RTP packet group satisfies the following conditions:

[0160] An audio-video adjacent RTP packet group includes one audio RTP packet and one video RTP packet, and there is no other RTP packet and RTCP packet between the audio RTP packet and the video RTP packet;

[0161] The RTP packets before the first audio-video adjacent RTP packet group can only be all audio RTP packets or all video RTP packets;

[0162] One RTP packet cannot belong to two audio-video adjacent RTP packet groups;

[0163] The RTP packets between two adjacent audio-video adjacent RTP packet groups can only be all audio RTP packets or all video RTP packets.

[0164] It should be noted that the above-mentioned device for identifying audio-video asynchronization in a video call process provided by the embodiments of the present application can realize all the method operations realized by the method embodiments and can achieve the same technical effects, and the same parts and beneficial effects in the method embodiments will not be described in detail.

[0165] FIG. 9 is a structural schematic diagram of a collection device provided by the embodiments of the present application. As shown in FIG. 9, the collection device includes a memory 910, a transceiver 920 and a processor 930; wherein the memory 910 is used to store a computer program; the transceiver 920 is used to transceive data under the control of the processor 930; and the processor 930 is used to read the computer program in the memory 910 and perform the following operations:

[0166] obtaining an audio RTP packet, an audio RTCP packet, a video RTP packet and a video RTCP packet in a video call process;

[0167] determining a first time difference of sending an audio RTCP packet and a video RTCP packet by a sending end based on the NTP timestamp of the audio RTCP packet and the NTP timestamp of the video RTCP packet in an audio-video adjacent RTCP packet group;

[0168] for at least one audio-video adjacent RTP packet group after the audio-video adjacent RTCP packet group, determining a second time difference of sending an audio RTP packet and a video RTP packet in the audio-video adjacent RTP packet group by the sending end based on the first time difference;

[0169] determine, based on the second time difference and a third time difference at which the audio RTP packet and the video RTP packet in the audio-video adjacent RTP packet group are received at the collection point, a length of audio-video asynchronization of the audio-video adjacent RTP packet group;

[0170] determine, based on the length of audio-video asynchronization of each audio-video adjacent RTP packet group and an audio-video asynchronization threshold, a recognition result of audio-video asynchronization in the video call process.

[0171] In some embodiments, based on the first time difference, the second time difference at which the audio RTP packet and the video RTP packet in the audio-video adjacent RTP packet group are sent by the sending end includes:

[0172] determine, based on the first time difference, the RTP timestamp of the audio RTCP packet, the RTP timestamp of the video RTCP packet, the RTP timestamp of the audio RTP packet, the RTP timestamp of the video RTP packet, the audio stream sampling frequency and the video stream sampling frequency, the second time difference at which the audio RTP packet and the video RTP packet are sent by the sending end.

[0173] In some embodiments, based on the first time difference, the RTP timestamp of the audio RTCP packet, the RTP timestamp of the video RTCP packet, the RTP timestamp of the audio RTP packet, the RTP timestamp of the video RTP packet, the audio stream sampling frequency and the video stream sampling frequency, the second time difference at which the audio RTP packet and the video RTP packet are sent by the sending end includes:

[0174] determine, based on the RTP timestamp of the audio RTCP packet, the RTP timestamp of the audio RTP packet and the audio stream sampling frequency, a fourth time difference at which the audio RTP packet and the audio RTCP packet are sent by the sending end; and, based on the RTP timestamp of the video RTCP packet, the RTP timestamp of the video RTP packet and the video stream sampling frequency, determine a fifth time difference at which the video RTP packet and the video RTCP packet are sent by the sending end;

[0175] determine, based on the first time difference, the fourth time difference and the fifth time difference, the second time difference at which the audio RTP packet and the video RTP packet are sent by the sending end.

[0176] In some embodiments, the calculation formula of the fourth time difference is: T1=(rtp.timestamp_a-rtcp.timestamp_a) / samplerate_a

[0177] In the formula, T1 represents the fourth time difference, rtp.timestamp_a represents the RTP timestamp of the audio RTP packet, rtcp.timestamp_a represents the RTP timestamp of the audio RTCP packet, and samplerate_a represents the sampling frequency of the audio stream.

[0178] In some embodiments, the fifth time difference is calculated according to the formula: T2 = (rtp.timestamp_v-rtcp.timestamp_v) / samplerate_v

[0179] In the formula, T2 represents the fifth time difference, rtp.timestamp_v represents the RTP timestamp of the video RTP packet, rtcp.timestamp_v represents the RTP timestamp of the video RTCP packet, and samplerate_v represents the sampling frequency of the video stream.

[0180] In some embodiments, the second time difference is calculated according to the formula: T av1 = T2+T diff -T1

[0181] In the formula, T av1 represents the second time difference, T2 represents the fifth time difference, T diff represents the first time difference, and T1 represents the fourth time difference.

[0182] In some embodiments, based on the second time difference and a third time difference of the audio RTP packet and the video RTP packet received by the collection point in the audio-video adjacent RTP packet group, a length of audio-video asynchronization of the audio-video adjacent RTP packet group is determined, including:

[0183] A difference between the second time difference and the third time difference is determined as the length of audio-video asynchronization of the audio-video adjacent RTP packet group.

[0184] In some embodiments, based on the length of audio-video asynchronization of each audio-video adjacent RTP packet group and an audio-video asynchronization threshold, an identification result of audio-video asynchronization in the video call process is determined, including:

[0185] A maximum value in the lengths of audio-video asynchronization of each audio-video adjacent RTP packet group is determined.

[0186] If the maximum value is greater than the audio-video asynchronization threshold, it is determined that there is an audio-video asynchronization problem in the video call process, and the maximum value is determined as the asynchronization length of the audio-video asynchronization problem.

[0187] In some embodiments, the operation further includes:

[0188] If the maximum value is less than or equal to the audio-video synchronization threshold, it is determined that there is no audio-video synchronization problem in the video call process.

[0189] In some embodiments, an audio-video adjacent RTCP packet group satisfies the following conditions:

[0190] An audio-video adjacent RTCP packet group includes one audio RTCP packet and one video RTCP packet, and there is no other RTCP packet and RTP packet between the one audio RTCP packet and the one video RTCP packet;

[0191] The RTCP packets before the first audio-video adjacent RTCP packet group can only be all audio RTCP packets or video RTCP packets;

[0192] One RTCP packet cannot belong to two audio-video adjacent RTCP packet groups;

[0193] The RTCP packets between the adjacent two audio-video adjacent RTCP packet groups can only be all audio RTCP packets or video RTCP packets.

[0194] In some embodiments, an audio-video adjacent RTP packet group satisfies the following conditions:

[0195] An audio-video adjacent RTP packet group includes one audio RTP packet and one video RTP packet, and there is no other RTP packet and RTCP packet between the one audio RTP packet and the one video RTP packet;

[0196] The RTP packets before the first audio-video adjacent RTP packet group can only be all audio RTP packets or video RTP packets;

[0197] One RTP packet cannot belong to two audio-video adjacent RTP packet groups;

[0198] The RTP packets between the adjacent two audio-video adjacent RTP packet groups can only be all audio RTP packets or video RTP packets.

[0199] In Figure 9, the bus architecture can include any number of interconnected buses and bridges, specifically the various circuitry of the one or more processors represented by the processor 930 and the memory represented by the memory 910 linked together. The bus architecture can also link various other circuitry such as peripheral devices, voltage regulators, and power management circuitry, which are well known in the art, and therefore, are not further described herein. The bus interface provides an interface. The transceiver 920 can be a plurality of elements, i.e., including a transmitter and a receiver, providing a unit for communicating with various other devices on a transmission medium. The processor 930 is responsible for managing the bus architecture and general processing, and the memory 910 can store data used by the processor 930 in performing operations. The processor and the memory can also be arranged physically separately.

[0200] It should be noted that the above acquisition device provided by the embodiments of the present application can realize all the method operations realized by the above method embodiments, and achieve the same technical effects. Therefore, the same parts and beneficial effects of the method embodiments in the embodiments will not be described in detail.

[0201] Figure 10 is a structural schematic diagram of an electronic device provided by the embodiments of the present application. As shown in Figure 10, the electronic device can include a processor 1010, a communication interface 1020, a memory 1030, and a communication bus 1040, wherein the processor 1010, the communication interface 1020, and the memory 1030 complete mutual communication through the communication bus 1040. The processor 1010 can invoke a computer program in the memory 1030 to perform the operations of the method for identifying audio-video asynchronization in a video call process provided by the above embodiments, for example, including:

[0202] obtaining an audio RTP packet, an audio RTCP packet, a video RTP packet, and a video RTCP packet in a video call process;

[0203] determining a first time difference of sending an audio RTCP packet and a video RTCP packet by a sending end based on an NTP timestamp of the audio RTCP packet and an NTP timestamp of the video RTCP packet in an audio-video adjacent RTCP packet group;

[0204] for at least one audio-video adjacent RTP packet group after the audio-video adjacent RTCP packet group, determining a second time difference of sending an audio RTP packet and a video RTP packet in the audio-video adjacent RTP packet group by the sending end based on the first time difference;

[0205] determine the audio-video asynchronization duration of the audio-video adjacent RTP packet group based on the second time difference and a third time difference of the audio RTP packet and the video RTP packet received by the collection point in the audio-video adjacent RTP packet group;

[0206] determine the identification result of the audio-video asynchronization in the video call process based on the audio-video asynchronization duration of each audio-video adjacent RTP packet group and an audio-video asynchronization threshold.

[0207] In addition, the logical instructions in the memory 1030 described above can be implemented in the form of a software function unit and sold or used as an independent product, which can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the present application or the part that essentially contributes to the related art or part of the technical solutions can be embodied in the form of a software product, which is stored in a storage medium and includes instructions for making a computer device (which can be a personal computer, a server, or a network device, etc.) execute all or part of the operations of the methods described in the embodiments of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various media that can store program codes.

[0208] On the other hand, the embodiments of the present application also provide a computer program product, which includes a computer program that can be stored on a non-transitory computer readable storage medium, and when the computer program is executed by a processor, the computer can execute the operations of the method for identifying the audio-video asynchronization in the video call process provided by the above-mentioned embodiments, for example, including:

[0209] obtain the audio RTP packet, the audio RTCP packet, the video RTP packet and the video RTCP packet in the video call process;

[0210] determine a first time difference of the sending of the audio RTCP packet and the video RTCP packet by the sending end based on the NTP timestamp of the audio RTCP packet and the NTP timestamp of the video RTCP packet in the audio-video adjacent RTCP packet group;

[0211] For at least one audio-video adjacent RTP packet group after the audio-video adjacent RTCP packet group, determine a second time difference of the sending of the audio RTP packet and the video RTP packet by the sending end based on the first time difference;

[0212] determine a time difference between the sending of the audio RTCP packet and the video RTCP packet by the sending terminal based on the NTP timestamp of the audio RTCP packet and the NTP timestamp of the video RTCP packet in the audio-video adjacent RTCP packet group;

[0213] determine a time difference between the sending of the audio RTCP packet and the video RTCP packet by the sending terminal based on the NTP timestamp of the audio RTCP packet and the NTP timestamp of the video RTCP packet in the audio-video adjacent RTCP packet group;

[0214] It should be noted that the computer program product provided in the embodiments of the present application can realize all the method operations achieved by the above method embodiments and achieve the same technical effects. Therefore, the same parts and beneficial effects in the embodiments of the present application as the method embodiments will not be described in detail.

[0215] In another aspect, the embodiments of the present application also provide a processor-readable storage medium, which stores a computer program for causing a processor to perform the operations of the method for identifying audio-video asynchronization in a video call process provided by the above embodiments, for example, including:

[0216] obtain an audio RTP packet, an audio RTCP packet, a video RTP packet and a video RTCP packet in a video call process;

[0217] determine a time difference between the sending of the audio RTCP packet and the video RTCP packet by the sending terminal based on the NTP timestamp of the audio RTCP packet and the NTP timestamp of the video RTCP packet in the audio-video adjacent RTCP packet group;

[0218] determine a time difference between the sending of the audio RTCP packet and the video RTCP packet by the sending terminal based on the NTP timestamp of the audio RTCP packet and the NTP timestamp of the video RTCP packet in the audio-video adjacent RTCP packet group;

[0219] determine a time difference between the sending of the audio RTCP packet and the video RTCP packet by the sending terminal based on the NTP timestamp of the audio RTCP packet and the NTP timestamp of the video RTCP packet in the audio-video adjacent RTCP packet group;

[0220] determine a time difference between the sending of the audio RTCP packet and the video RTCP packet by the sending terminal based on the NTP timestamp of the audio RTCP packet and the NTP timestamp of the video RTCP packet in the audio-video adjacent RTCP packet group;

[0221] It should be noted that the processor-readable storage medium provided in the embodiments of the present application can realize all the method operations achieved by the above method embodiments and achieve the same technical effects. Therefore, the same parts and beneficial effects in the embodiments of the present application as the method embodiments will not be described in detail.

[0222] The processor-readable storage medium can be any available medium or data storage that can be accessed by a processor including both volatile and nonvolatile media, removable and non-removable media, and computer readable and machine readable storage media. By way of example, and not limitation, computer readable media can comprise RAM, ROM, EEPROM, flash card, disc tape, etc. or any combination thereof.

[0223] The apparatus embodiments described above are only illustrative, wherein the units described as separate components can or can not be physically separate, and the components displayed as units can or can not be physical units, i.e., can be located in one place, or can be distributed to multiple network units. Part or all of the modules can be selected to achieve the purposes of the embodiments according to actual needs. Those skilled in the art can understand and implement without creative labor.

[0224] Through the description of the above embodiments, those skilled in the art can clearly understand that the embodiments can be realized by means of software plus necessary universal hardware platforms, and of course, can also be realized by hardware. Based on such understanding, the above technical solutions can be embodied in the form of software products, and the computer software products can be stored in a computer readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and include a plurality of instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) execute the methods described in each embodiment or some parts of the embodiments.

[0225] The technical solutions provided by the embodiments of the present application can be applied to various systems. For example, the applicable systems can be a long term evolution (LTE) system, an LTE frequency division duplex (FDD) system, an LTE time division duplex (TDD) system, a long term evolution advanced (LTE-A) system, a universal mobile system (UMTS), a worldwide interoperability for microwave access (WiMAX) system, a 5G new radio (NR) system and its evolved communication system, a 6G (sixth generation mobile communication technology) system, and the like. The various systems can include terminal devices and network devices. The system can also include a core network part, such as an evolved packet system (EPC), a 5G core network (5GC), and the like.

[0226] The terminal referred to in the embodiments of the present application can be a device providing voice and / or data connectivity to a user, a handheld device having wireless connection capability, or other processing devices connected to a wireless modem, etc. In different systems, the name of the terminal can also be different, for example, in a 5G system, the terminal can be referred to as a user equipment (UE). The wireless terminal device can communicate with one or more core networks (CN) through a radio access network (RAN). The wireless terminal device can be a mobile terminal device, such as a mobile phone (also known as a "cellular" phone) and a computer with a mobile terminal device, for example, a portable, pocket, handheld, computer built-in or vehicle-mounted mobile device that exchanges voice and / or data with a radio access network. For example, personal communication service (PCS) phones, cordless phones, session initiated protocol (SIP) phones, wireless local loop (WLL) stations, personal digital assistants (PDAs), etc. The wireless terminal device can also be referred to as a system, a subscriber unit, a subscriber station, a mobile station, a mobile, a remote station, an access point, a remote terminal, an access terminal, a user terminal, a user agent, a user device, which are not limited in the embodiments of the present application.

[0227] Those skilled in the art should understand that the embodiments of the present application can be provided as a method, a system, or a computer program product. Therefore, the present application can take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application can take the form of a computer program product implemented on one or more computer usable storage media (including but not limited to magnetic disk storage and optical storage, etc.) containing computer usable program code.

[0228] The computer executable instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operations to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide operations for implementing the functions specified in the flowchart block or blocks or in the flowchart one or more blocks.

[0229] These processor-executable instructions can also be stored in a processor-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the processor-readable memory produce an article of manufacture including instructions which implement the function specified in the flowchart block or blocks or in the flowchart one or more blocks.

[0230] These processor-executable instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operations to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide operations for implementing the functions specified in the flowchart block or blocks or in the flowchart one or more blocks.

[0231] Obviously, numerous modifications and variations of the present application are possible in light of the above teachings. It is therefore to be understood that within the scope of the claims and their equivalents, the application can be practiced otherwise than as specifically described.

Claims

1. A method for identifying audio-video asynchronization in a video call process, comprising: obtaining audio RTP packets, audio RTCP packets, video RTP packets and video RTCP packets in the video call process; determining a first time difference between sending of the audio RTCP packet and the video RTCP packet by a sending terminal based on NTP time stamps of the audio RTCP packet and the video RTCP packet in an audio-video adjacent RTCP packet group; for at least one audio-video adjacent RTP packet group after the audio-video adjacent RTCP packet group, determining a second time difference between sending of the audio RTP packet and the video RTP packet by the sending terminal in the audio-video adjacent RTP packet group based on the first time difference; determining an audio-video asynchronization duration of the audio-video adjacent RTP packet group based on the second time difference and a third time difference between receiving of the audio RTP packet and the video RTP packet by a collection point in the audio-video adjacent RTP packet group; determining an identification result of the audio-video asynchronization in the video call process based on the audio-video asynchronization duration of each audio-video adjacent RTP packet group and an audio-video asynchronization threshold.

2. The method of identifying audio-video desynchronization in a video call according to claim 1, wherein, The determining of the second time difference between sending of the audio RTP packet and the video RTP packet by the sending terminal in the audio-video adjacent RTP packet group based on the first time difference comprises: determining the second time difference between sending of the audio RTP packet and the video RTP packet by the sending terminal based on the first time difference, RTP time stamps of the audio RTCP packet and the video RTCP packet, RTP time stamps of the audio RTP packet and the video RTP packet, an audio stream sampling frequency and a video stream sampling frequency.

3. The method of identifying audio-video desynchronization in a video call according to claim 2, wherein, The determining of the second time difference between sending of the audio RTP packet and the video RTP packet by the sending terminal based on the first time difference, RTP time stamps of the audio RTCP packet and the video RTCP packet, RTP time stamps of the audio RTP packet and the video RTP packet, an audio stream sampling frequency and a video stream sampling frequency comprises: determining a fourth time difference between sending of the audio RTP packet and the audio RTCP packet by the sending terminal based on the RTP time stamp of the audio RTCP packet, the RTP time stamp of the audio RTP packet and the audio stream sampling frequency, and determining a fifth time difference between sending of the video RTP packet and the video RTCP packet by the sending terminal based on the RTP time stamp of the video RTCP packet, the RTP time stamp of the video RTP packet and the video stream sampling frequency; determining the second time difference between sending of the audio RTP packet and the video RTP packet by the sending terminal based on the first time difference, the fourth time difference and the fifth time difference.

4. The method of identifying audio-video desynchronization in a video call according to claim 3, wherein, The fourth time difference is calculated by the following formula: T1=(rtp.timestamp_a-rtcp.timestamp_a) / samplerate_a In the formula, T1 represents the fourth time difference, rtp.timestamp_a represents the RTP timestamp of the audio RTP packet, rtcp.timestamp_a represents the RTP timestamp of the audio RTCP packet, and samplerate_a represents the sampling frequency of the audio stream.

5. The method of identifying audio-video desynchronization in a video call according to claim 3, wherein, The fifth time difference is calculated according to the formula: T2 = (rtp.timestamp_v-rtcp.timestamp_v) / samplerate_v In the formula, T2 represents the fifth time difference, rtp.timestamp_v represents the RTP timestamp of the video RTP packet, rtcp.timestamp_v represents the RTP timestamp of the video RTCP packet, and samplerate_v represents the sampling frequency of the video stream.

6. The method of identifying audio-video desynchronization in a video call according to claim 3, wherein, The calculation formula of the second time difference is: T av1 = T2+ T diff - T1 wherein T av1 denotes the second time difference, T2denotes the fifth time difference, T diff denotes the first time difference, T1denotes the fourth time difference.

7. The method of identifying audio-video desynchronization in a video call according to claim 1, wherein, The method further comprises: The difference between the second time difference and the third time difference is determined as the audio-video unsynchronized duration of the adjacent audio-video RTP packet group.

8. The method of identifying audio-video desynchronization in a video call according to claim 1, wherein, The method further comprises: The audio-video unsynchronized durations of the adjacent audio-video RTP packet groups are counted to obtain a statistical value. The statistical value is compared with the audio-video unsynchronized threshold to determine whether there is an audio-video unsynchronized problem in the video call process.

9. The method of identifying audio-video desynchronization in a video call according to any one of claims 1 to 8, wherein, The method further comprises: The maximum value of the audio-video unsynchronized durations of the adjacent audio-video RTP packet groups is determined. If the maximum value is greater than the audio-video unsynchronized threshold, it is determined that there is an audio-video unsynchronized problem in the video call process, and the maximum value is determined as the unsynchronized duration of the audio-video unsynchronized problem.

10. The method of identifying audio-video desynchronization in a video call according to claim 9, wherein, The method further comprises: If the maximum value is less than or equal to the audio-video unsynchronized threshold, it is determined that there is no audio-video unsynchronized problem in the video call process.

11. The method of identifying audio-video desynchronization in a video call according to claim 1, wherein, The adjacent audio-video RTCP packet group satisfies the following conditions: One of the adjacent audio-video RTCP packet groups includes one audio RTCP packet and one video RTCP packet, and there is no other RTCP packet and RTP packet between the one audio RTCP packet and the one video RTCP packet. The RTCP packets before the first adjacent audio-video RTCP packet group can all be audio RTCP packets or video RTCP packets. One RTCP packet cannot belong to two adjacent audio-video RTCP packet groups. The RTCP packets between two adjacent audio-video RTCP packet groups can all be audio RTCP packets or video RTCP packets.

12. The method of identifying audio-video desynchronization in a video call according to claim 1, wherein, The adjacent audio-video RTP packet group satisfies the following conditions: One of the audio-video adjacent RTP message groups comprises one audio RTP message and one video RTP message, and there are no other RTP messages and RTCP messages between the one audio RTP message and the one video RTP message; The RTP messages before the first audio-video adjacent RTP message group can all be audio RTP messages or video RTP messages; One RTP message cannot belong to two audio-video adjacent RTP message groups; The RTP messages between two adjacent audio-video adjacent RTP message groups can all be audio RTP messages or video RTP messages.

13. The method of identifying audio-video desynchronization in a video call according to claim 8, wherein, The method further comprises: If the maximum value is less than or equal to the audio-video asynchronization threshold, it is determined that there is no audio-video asynchronization problem in the video call process, and the audio-video asynchronization durations of the calculated audio-video adjacent RTP message groups are cleared.

14. The method of identifying audio-video desynchronization in a video call according to claim 1, wherein, The audio-video adjacent RTCP message group comprises a group of adjacent audio RTCP messages and video RTCP messages received by the collection point in sequence; or The audio-video adjacent RTP message group comprises a group of adjacent audio RTP messages and video RTP messages received by the collection point in sequence.

15. An apparatus for identifying audio-video asynchronization in a video call process, comprising: an acquisition module configured to acquire audio RTP messages, audio RTCP messages, video RTP messages and video RTCP messages in a video call process; a first determination module configured to determine a first time difference between sending of the audio RTCP messages and the video RTCP messages by a sending end based on NTP time stamps of the audio RTCP messages and NTP time stamps of the video RTCP messages in an audio-video adjacent RTCP message group; a second determination module configured to determine a second time difference between sending of audio RTP messages and video RTP messages in at least one audio-video adjacent RTP message group after the audio-video adjacent RTCP message group by the sending end based on the first time difference; a third determination module configured to determine an audio-video asynchronization duration of the audio-video adjacent RTP message group based on the second time difference and a third time difference between receiving of the audio RTP messages and the video RTP messages in the audio-video adjacent RTP message group by a collection point; a fourth determination module configured to determine an identification result of audio-video asynchronization in the video call process based on the audio-video asynchronization durations of the audio-video adjacent RTP message groups and an audio-video asynchronization threshold.

16. A collection device comprising a memory, a transceiver and a processor; the memory is configured to store a computer program; the transceiver is configured to transceive data under control of the processor; the processor is configured to read the computer program in the memory and perform the following operations: acquire audio RTP messages, audio RTCP messages, video RTP messages and video RTCP messages in a video call process; determine a first time difference between sending of the audio RTCP packet and the video RTCP packet by the sending end based on the NTP timestamp of the audio RTCP packet and the NTP timestamp of the video RTCP packet in the audio-video adjacent RTCP packet group; for at least one audio-video adjacent RTP packet group after the audio-video adjacent RTCP packet group, determine a second time difference between sending of the audio RTP packet and the video RTP packet in the audio-video adjacent RTP packet group by the sending end based on the first time difference; determine an audio-video asynchronization duration of the audio-video adjacent RTP packet group based on the second time difference and a third time difference between receiving of the audio RTP packet and the video RTP packet in the audio-video adjacent RTP packet group by the collection point; determine an identification result of audio-video asynchronization in the video call process based on the audio-video asynchronization duration of each of the audio-video adjacent RTP packet groups and an audio-video asynchronization threshold.

17. The collection device of claim 16, wherein, The collection device comprises a collection device of a deep packet inspection system.

18. An electronic device comprising a processor and a memory having a computer program stored therein, wherein, The processor executes the computer program to implement the method for identifying audio-video asynchronization in a video call process according to any one of claims 1 to 14.

19. A processor-readable storage medium having stored thereon a computer program, wherein, The computer program is executed by the processor to implement the method for identifying audio-video asynchronization in a video call process according to any one of claims 1 to 14.

20. A computer program product comprising a computer program, wherein, The computer program is executed by the processor to implement the method for identifying audio-video asynchronization in a video call process according to any one of claims 1 to 14.

Images