Method and device for acquiring time delay of first frame

Abstract

The invention provides a method and device for obtaining first frame time delay, and relates to the technical field of short video processing. The method comprises the following steps: judging whether a video is preloaded or not when being played in terminal equipment; if the video is preloaded when being played in the terminal equipment, determining that the time delay of the first frame of the video is 0; and if the video is not preloaded when being played in the terminal equipment, determining the first frame time delay of the video according to the video playing start time and the playing request time of the video. According to the method provided by the embodiment of the invention, the first frame time delay of the video can be conveniently and accurately determined.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of short video processing, in particular to a method and device for acquiring a first frame delay. BACKGROUND

[0002] A video service provider can improve the quality of experience (QoE) of a video service based on user experience perception. The first frame delay is an important indicator in user experience perception. Therefore, how to acquire the first frame delay of a video so as to improve the QoE of the video service becomes a technical problem to be solved urgently. SUMMARY

[0003] The present application provides a method and device for acquiring a first frame delay, which can conveniently and accurately determine the first frame delay of a video.

[0004] In a first aspect, a method for acquiring a first frame delay is provided, comprising: judging whether a video has been preloaded when the video is played in a terminal device; if the video has been preloaded when the video is played in the terminal device, determining the first frame delay of the video as 0; and if the video has not been preloaded when the video is played in the terminal device, determining the first frame delay of the video according to a video starting time and a play request time of the video.

[0005] In the embodiments of the present application, whether the video has been preloaded when the video is played in the terminal device is judged, and the first frame delay of the video can be conveniently determined according to the preloading condition. Meanwhile, since the influence of preloading is considered, the determined first frame delay is more accurate. If the video has not been preloaded when the video is played in the terminal device, the first frame delay of the video can be determined according to the video starting time and the play request time of the video, without a large number of video samples or a complex statistical model, so that the first frame delay of the video can be conveniently determined.

[0006] In some possible implementation manners, determining whether the video has been preloaded when being played includes: obtaining M play request times of the terminal device in a first time period, M being a positive integer; obtaining N preloaded segment request times of the terminal device in the first time period, N being a positive integer; for each play request time in the M play request times, if a difference between a number of preloaded segment request times before each play request time and a number of play request times before the each play request time in the M play request times and the N preloaded segment request times is greater than 0, it is determined that a video corresponding to the each play request time has been preloaded when being played; if the difference between the number of preloaded segment request times before each play request time and the number of play request times before the each play request time in the M play request times and the N preloaded segment request times is zero or negative, it is determined that the video corresponding to the each play request time has not been preloaded when being played.

[0007] In this implementation manner, whether the video is a preloaded video is determined based on a sequence of the play request time and the preloaded request time. If a number of preloaded segment requests before a play request is greater than a number of play requests by one or more, it is indicated that a time-frequency to be played by the play request has been preloaded, and therefore the video is a preloaded video. If the number of preloaded segment requests before the play request is the same as the number of play requests, it is indicated that the video has not been preloaded before the play request, and therefore the video is a non-preloaded video. This method for determining whether the video is a preloaded video is simpler and has lower implementation complexity.

[0008] In some possible implementation manners, the obtaining of the M play request times of the terminal device in the first time period includes: obtaining a data stream in which a video play request log of the terminal device is located in the first time period; obtaining R segment records in the data stream, each segment record in the R segment records containing a segment length, R being a positive integer; determining M packet length sequences based on R segment lengths corresponding to the R segment records, each packet length sequence in the M packet length sequences containing four segment lengths in a sequence of the R segment lengths in time sequence, and four segments corresponding to the four segment lengths in time sequence being an uplink segment, a downlink segment, an uplink segment and a downlink segment in turn, and each segment length in the four segment lengths being less than or equal to a first preset length; and determining the M play request times based on the M packet length sequences, wherein the M play request times correspond to the M packet length sequences in one-to-one correspondence, and each play request time in the M play request times is a transmission time of a segment corresponding to a first segment length in a corresponding packet length sequence.

[0009] The implementation is an implementation of obtaining the playing request time of the video. In the implementation, the transmission characteristics of the uplink and downlink data packets caused by the video playing request in the data stream where the video playing request log is located are fully utilized to distinguish or identify the data packets corresponding to each video playing request, so that the time of each video playing request can be conveniently determined.

[0010] The segment length in the implementation can be replaced by the merged payload length.

[0011] In some possible implementations, the obtaining of the M playing request times of the terminal device in the first time period includes: obtaining M segment records in a video cover download stream, each segment record in the M segment records containing a segment length, and the segment length in each segment record in the M segment records being greater than a second preset length; and determining M playing request times according to the M segment records, the M playing request times corresponding to the M segment records in a one-to-one manner, and each playing request time in the M playing request times being a transmission time corresponding to the corresponding segment record.

[0012] The implementation is another implementation of obtaining the playing request time of the video. In the implementation, each video playing request time is identified or distinguished based on the transmission characteristics of the data packets in the video cover download stream, so that the time of each video playing request can be conveniently determined.

[0013] The segment length in the implementation can be replaced by the merged payload length.

[0014] In some possible implementations, the obtaining of the N preloading segment request times of the terminal device in the first time period includes: obtaining a video stream of the terminal device in the first time period; obtaining K segment records in the video stream, each segment record in the K segment records containing a segment length, and K being a positive integer; and determining the N preloading request times according to K segment lengths corresponding to the K segment records, wherein the N preloading request times correspond to N segment lengths of the K segment lengths in a one-to-one manner, each segment length in the N segment lengths is greater than a third preset length and less than or equal to a fourth preset length, the each segment length indicates an uplink segment, and each preloading request time in the N preloading request times is a transmission time of the segment indicated by the corresponding segment length.

[0015] The implementation is an implementation of obtaining the preloading segment request time. The implementation fully utilizes the transmission characteristics, such as the data packet length, of the data packets of the preloading request, so that the preloading segment request can be distinguished or identified from the preloading segment request and the video loading request, and then each preloading segment request time can be conveniently determined.

[0016] The segment length in this implementation manner can be replaced by a merged load length.

[0017] In some possible implementation manners, the method further includes: obtaining a video stream of the terminal device in the first time period; obtaining K segment records in the video stream, each segment record in the K segment records containing a segment length, K being a positive integer; determining P play segment request times according to K segment lengths corresponding to the K segment records, wherein the P play segment request times correspond to P segment lengths of the K segment lengths one by one, each segment length in the P segment lengths being greater than a fourth preset length, the segment length indicating an uplink segment, and each play segment request time in the P play segment request times being a transmission time of the segment indicated by the corresponding segment length, P being a positive integer less than or equal to K; and determining F video start times according to the P play segment request times, the F video start times corresponding to F play segment request times in the P play segment request times one by one, a total length of a play segment corresponding to each play segment request time in the F play segment request times being greater than or equal to a preset length, and each video start time in the F video start times being a time with a starting time of the corresponding play segment request time and a length accumulation value greater than or equal to a preset length threshold for the first time, F being a positive integer less than or equal to P.

[0018] The implementation manner is an implementation manner for obtaining a video start time, and fully utilizes transmission characteristics of a data packet of a video loading request, such as a data packet length, so that the video loading request can be distinguished or recognized from preloaded segment requests, and each video loading request time can be conveniently determined.

[0019] The segment length in this implementation manner can be replaced by a merged load length.

[0020] In some possible implementation manners, the terminal device transmits data by using a transport layer security (TSL) protocol. That is to say, in a scenario of transmitting a video by using the TSL protocol, the technical solution provided in this application can quickly and conveniently determine a first frame delay of the video.

[0021] In some possible implementation manners, the terminal device transmits data by using a transport layer protocol (QUIC) based on a user datagram protocol (UDP). That is to say, in a scenario of transmitting a video by using the QUIC protocol, the technical solution provided in this application can quickly and conveniently determine a first frame delay of the video.

[0022] In some possible implementation manners, the determining the first frame delay of the video according to a video start time of the video and a play request time includes: determining a difference between the video start time and the play request time as the first frame delay.

[0023] In a second aspect, a device for obtaining a first frame delay is provided. The device can include a module corresponding to each of the methods / operations / steps / actions described in the first aspect. The module can be a hardware circuit, a software, or a combination of hardware circuit and software.

[0024] In some possible implementation manners, the device includes a processor coupled to the memory, and the memory is configured to store a computer program (also referred to as code or instructions) that, when executed by the processor, causes the device to perform the method in the first aspect or any possible implementation manner of the first aspect.

[0025] In some possible implementation manners, the device further includes a memory coupled to the processor.

[0026] In some possible implementation manners, the processor is one or more, and / or the memory is one or more.

[0027] In some possible implementation manners, the memory can be integrated with the processor, or the memory is disposed separately from the processor.

[0028] In a design, the device can be a network device, a device, a module, a circuit or a chip configured to be disposed in a network device, or a device capable of being matched with a network device.

[0029] As an example, the network device can be a gateway, a router, a firewall, an intrusion prevention system (IPS), an online behavior manager, an operator network device (also referred to as an operator server, etc.), a cloud server (also referred to as a cloud-side server, etc.), or a terminal device (also referred to as a terminal, etc.).

[0030] In a third aspect, a computer readable storage medium is provided. The computer readable storage medium has stored thereon a computer program (also referred to as code or instructions). When the computer program is run on a computer, the computer is caused to perform the method in any one of the aspects or any possible implementation manner of any one of the aspects.

[0031] In a fourth aspect, a computer program product is provided, including a computer program (which can also be referred to as code or instructions), which, when executed on a computer, causes the computer to perform the method in any one of the above aspects or any possible implementation manner of any one of the above aspects. BRIEF DESCRIPTION OF DRAWINGS

[0032] Figure 1 is a schematic diagram of an application scenario suitable for the present application;

[0033] Figure 2 is a schematic diagram of video-related time of an embodiment of the present application;

[0034] Figure 3 is a schematic flow of a method for acquiring a first frame delay of an embodiment of the present application;

[0035] Figure 4 is a schematic flow of a method for acquiring a first frame delay of an embodiment of the present application;

[0036] Figure 5 is a schematic flow of a method for acquiring a first frame delay of an embodiment of the present application;

[0037] Figure 6 is a schematic flow of a method for acquiring a first frame delay of an embodiment of the present application;

[0038] Figure 7 is a schematic flow of a method for acquiring a first frame delay of an embodiment of the present application;

[0039] Figure 8 is a schematic structural diagram of an apparatus for acquiring a first frame delay provided by an embodiment of the present application;

[0040] Figure 9 is a schematic structural diagram of an apparatus for acquiring a first frame delay provided by an embodiment of the present application. DETAILED DESCRIPTION

[0041] The technical solutions in the embodiments of the present application will be described below with reference to the accompanying drawings in the embodiments of the present application.

[0042] In the description of the present application, unless otherwise specified, the character " / " represents that the objects associated before and after are in an "or" relationship, for example, A / B can represent A or B; "and / or" in the present application is only a description of the associated relationship of the associated objects, which means that there can be three relationships, for example, A and / or B, which can represent: A exists alone, A and B exist simultaneously, and B exists alone, wherein A and B can be singular or plural. And, in the description of the present application, unless otherwise specified, "at least one" means one or more, and "multiple" means two or more than two. "At least one of the following" or the like means any combination of the items, including any combination of single item or multiple items. For example, at least one of a, b, or c can represent: a, b, c, a-b, a-c, b-c, or a-b-c, wherein a, b, and c can be single or multiple. In addition, in order to clearly describe the technical solutions of the embodiments of the present application, in the embodiments of the present application, "first", "second", and the like are used to distinguish the functions and effects of the same items or similar items. Those skilled in the art can understand that "first", "second", and the like do not limit the quantity and execution order, and "first", "second", and the like do not necessarily mean different. It should be understood that "in the case of", "if", "when", "if", and the like similar descriptions in the present application can be replaced.

[0043] Some concepts in the present application are introduced below.

[0044] Short video refers to video content played on various new media platforms, suitable for watching in mobile state and short leisure state, and high-frequency push, with a length of several seconds to several minutes.

[0045] QUIC packet payload length (QUIC Payload Size) refers to the length of the data encrypted after removing the header, connection ID and other information of the quick UDP internet connection (QUIC) packet. UDP is the user datagram protocol (user datagram protocol).

[0046] TLS packet payload length (TLS Payload Size) refers to the length of the data encrypted after removing the header, connection ID and other information of the TLS packet.

[0047] Packet payload length refers to the length of the data encrypted after removing the header, connection ID and other information of the packet, which can be referred to as packet length, and simply referred to as packet length.

[0048] Preload, refers to downloading the data of the subsequent video in advance when playing the current video, so as to realize fast start when switching to the next video, thereby optimizing the user's playing experience.

[0049] First frame delay, also known as first frame time or first frame delay, refers to the time spent from the user's operation of playing related actions (clicking play, sliding card, etc.) to the rendering of the first frame.

[0050] Video slice, refers to a series of slice files obtained by dividing a video according to a certain rule.

[0051] Play slice, refers to the downloaded video slice that is the video slice of the video currently being played.

[0052] Preload slice, refers to the downloaded video slice of the other video to be played.

[0053] Traffic plaintext information, refers to the plaintext information extracted from the traffic of the plaintext transmission protocol.

[0054] Packet length sequence, refers to a sequence formed by a series of continuous or discontinuous packet lengths.

[0055] Record length, refers to the number of bytes of the record protocol layer segment data.

[0056] Start failure, refers to the failure to play the first frame of the video.

[0057] Resuming, refers to resuming downloading from the interrupted position when a network failure is encountered until the downloaded data is complete.

[0058] Flow, the general way of dividing flows is to attribute packets with the same five-tuple (source IP, source port, destination IP, destination port, and transmission layer protocol) to the same flow. For QUIC flow, packets with the same QUIC connection ID can also be attributed to the same flow.

[0059] Figure 1 is a schematic diagram of an application scenario of an embodiment of the present application. As shown in Figure 1 , the application scenario can include a terminal device 110, a router 120, a firewall 130, a router 140, an operator network device 150, an Internet 160, and a cloud server 170.

[0060] The terminal device 110 is an electronic device that can play video traffic. As an example, the terminal device 110 is deployed with an application or service capable of playing video. For example, the terminal device 110 can be a mobile phone, a tablet computer, a smart TV, a computer, or the like.

[0061] The terminal device 110 can send a request for downloading video traffic to the Internet 160 and receive video traffic from the Internet 160 through the router 120, the firewall 130, the router 140 and the operator network device 150 in turn.

[0062] The router 120 is closer to the terminal device than the router 140.

[0063] The operator network device 150 can mirror the network traffic flowing through itself.

[0064] The cloud server 170 can obtain the traffic flowing through the router 120, the firewall 130, the router 140 or the operator network device 150.

[0065] It should be noted that, Figure 1 The schematic diagram is only for the convenience of understanding, and in actual application, the application scenario of the technical scheme of the present application can include more or fewer communication devices.

[0066] For example, the application scenario can not include the firewall 130, the router 140, the operator network device 150 and the cloud server 170.

[0067] For another example, the application scenario can not include the router 120, the operator network device 150 and the cloud server 170.

[0068] For another example, the application scenario can not include the router 120, the firewall 130 and the cloud server 170.

[0069] Figure 2 FIG. 1 is a schematic diagram of the operation process of playing and loading a video on a terminal device according to an embodiment of the present application. As shown in FIG. 1, Figure 2 in a first time period, the terminal device plays a video A between t1 and t4; at t2, the terminal device starts preloading a video B; at t3, the terminal device starts preloading a video C; at t5, the terminal device receives a request for playing the video B, and then starts playing the video B; at t6, the terminal device receives a request for playing the video C, and then starts playing the video C; at t7, the terminal device receives a request for playing a video D, and then sends a request for loading the video D; at t8, the terminal device starts playing the video D.

[0070] It can be understood that at the beginning of preloading a segment, the terminal device needs to send a request for preloading the segment to the network side; after the terminal device receives a request for playing a video, the terminal device first judges whether the video has been preloaded, if the video has been preloaded, the terminal device reads the video from the cache for playing; if the video has not been preloaded, the terminal device sends a request for loading the video to the network side.

[0071] Figure 2In some embodiments, the first frame delay of the B video and the first frame delay of the C video are both 0. The first frame delay of the D video is the difference between the time when the D video is requested to be played and the time when the D video is started to be played, i.e., t8-t7. The time when the D video is started to be played is referred to as the start time, i.e., t8.

[0072] Figure 3 FIG. 1 is a schematic flowchart of a method for obtaining a first frame delay according to an embodiment of the present application. Figure 3 The method shown can include steps S310, S320 and S330. An apparatus for implementing the method can be referred to as an apparatus for obtaining a first frame delay. The apparatus can be a router, a terminal device, a firewall, an operator network device, a cloud server, etc. Figure 1

[0073] S310, determining whether the video has been preloaded when the video is played on the terminal device. If the video has been preloaded when the video is played on the terminal device, S320 is performed; if the video has not been preloaded when the video is played on the terminal device, S330 is performed.

[0074] In some scenarios, the video is a short video.

[0075] S320, determining that the first frame delay of the video is 0.

[0076] S330, determining the first frame delay of the video according to the video start time of the video and the play request time of the video.

[0077] In some implementations, the play request time can be understood as the time when the user requests to play the video, or the time when it is determined that the video needs to be played, or the time when the user performs an action related to the play request.

[0078] For example, when the user slides a window on the terminal device to indicate a request to play the next video, the time when the user slides the window can be determined as the play request time of the video.

[0079] For another example, the time when the user clicks a button on the terminal device for requesting to play the next video can be determined as the play request time of the video.

[0080] The method for determining whether the video has been preloaded when the video is played is described below.

[0081] ​In some implementations, determining whether a video has been preloaded during playback includes: obtaining M playback request times of the terminal device within a first time period, where M is a positive integer; obtaining N preloading request times of the terminal device within the first time period, where N is a positive integer; for each of the M playback request times, if the difference between the number of preloading request times before each playback request time and the number of playback request times before the playback request time among the M playback request times and the N preloading request times is greater than 0, then determining that the video corresponding to each playback request time has been preloaded during playback; if the difference between the number of preloading request times before each playback request time and the number of playback request times before the playback request time among the M playback request times and the N preloading request times is zero or a negative number, then determining that the video corresponding to each playback request time has not been preloaded during playback.

[0082] An example of the first time period is a period during which the apparatus for obtaining the first frame delay collects video traffic from the terminal device. For example, the period may be one hour, one day, one week, or one month. The period may be preset. In some implementations, the period may be different for different time periods. For example, the video traffic collection period may be shorter during periods with high video traffic usage density, while the video traffic collection period may be longer during periods with low video traffic usage density.

[0083] The preload request time can be understood as the time it takes for the terminal device to request the network to download or load the video. During this time, the terminal device may still be playing other videos, or the currently playing video may not be finished.

[0084] by Figure 2 Taking the operation process shown as an example, the first time period includes, in chronological order: a preloading request (for video B), a preloading request (for video C), a video playback request (for video B), a video playback request (for video C), and a video playback request (for video D). Because there are 2 preloading requests before the first video playback request, and the number of video playback requests is 0, it means that at least one of the previous preloading requests preloaded the video corresponding to the first video playback request, and therefore the corresponding video has been preloaded. Because there are 2 preloading requests before the second video playback request, and the number of video playback requests is 1, it means that at least one of the previous preloading requests preloaded the video corresponding to the second video playback request, and therefore the corresponding video has been preloaded. Because there are 2 preloading fragment requests before the third video playback request, and the number of video playback requests is 2, it means that the corresponding video has not been preloaded.

[0085] In some implementations, after obtaining all the preloading request times and all the playing request times in the first time period, the request times are sorted in chronological order; a variable x = 0 is set, and the aforementioned list is traversed in chronological order, x = x + 1 when a preloading request time occurs, and x = x - 1 when a playing request time occurs; when each playing request time is traversed, if x is 0, it is recorded that the video corresponding to the playing request time (or the video when the playing request is received) has no preloading, otherwise it is recorded that the video corresponding to the playing request time has preloading, and the first frame delay of the video is 0.

[0086] In some implementations, in order to reduce the complexity of the scheme, when x = x + 1 is greater than 5, x can be reset to 5; when x = x - 1 is less than 0, x can be reset to 0.

[0087] The method for obtaining the playing request time in the embodiments of the present application is introduced below.

[0088] In some implementations of obtaining the playing request time, the stream in which the log reported at the same time as the playing request or after a fixed time can be obtained, and the playing request time can be obtained through the plaintext information in the traffic or the packet length sequence.

[0089] As an example, the stream in which the log reported at the same time as the playing request or after a fixed time is analyzed, and the length of each record is obtained.

[0090] It can be understood that the record length in the present application indicates the length of the record in the uplink and downlink transmission process of the terminal. For example, the tls.record.length in the TLS protocol or the packet payload length in the QUIC protocol. The segmented information is used in the description of some embodiments of the present application, and in some scenarios, the segmented length in the segmented information can be replaced by the packet payload length or the merged payload length.

[0091] All packet lengths are arranged in chronological order from front to back to obtain a packet length sequence; a specific packet length sequence is obtained by matching the packet length sequence, and the transmission time of the first packet in the specific packet length sequence can be recorded as the playing request time. The packet length sequence matching can refer to matching Q packet lengths that meet the preset conditions from all packet lengths, the Q packet lengths are arranged in chronological order to obtain a packet length sequence, Q is a positive integer, and the preset conditions include that the transmission directions of the Q consecutive packet lengths are specified directions, and each packet length in the multiple packet lengths is within a specified range.

[0092] As an example, Q is 4, and the transmission directions of the 4 packet lengths are uplink, downlink, uplink and downlink in turn. It can be understood that the value of Q and the requirement of the transmission direction can be determined based on the transmission characteristics of the video stream.

[0093] In some scenarios, the implementation includes: obtaining a data stream in which a video playback request log of the terminal device is located within the first time period; obtaining R segment records in the data stream, R being a positive integer, each of the R segment records containing a segment length; determining M packet length sequences based on the R segment records, each of the M packet length sequences containing four segment lengths in a sequence obtained by arranging, in time sequence, the R segment lengths corresponding to the R segment records one-to-one, and four segments corresponding to the four segment lengths one-to-one arranged in time sequence in a sequence obtained by arranging, in time sequence, the four segments corresponding to the four segment lengths one-to-one, the four segments being an uplink segment, a downlink segment, an uplink segment, and a downlink segment in turn, and each of the four segment lengths being less than or equal to a first preset length; and determining M playback request times based on the M packet length sequences, wherein the M playback request times correspond to the M packet length sequences one-to-one, and each of the M playback request times is a transmission time of a segment indicated by a first segment length in the corresponding packet length sequence.

[0094] When the segment corresponding to the first segment length contains multiple packets, each of the M playback request times is a transmission time of a first packet in the segment corresponding to the first segment length in the corresponding packet length sequence.

[0095] In another implementation of obtaining the playback request times, it is considered that the terminal downloads the resource of the short video immediately after receiving the playback request, and such a resource is not preloaded, so the download time of these resources can be identified by matching the plaintext information in the traffic or the packet length sequence of the traffic, so as to obtain the playback request times. Examples of such resources that are not preloaded include a short video cover picture, a user avatar, and the like.

[0096] For example, a video cover download stream of the terminal device within the first time period is obtained; M segment records in the video cover download stream are obtained, each of the M segment records containing a segment length, and the segment length in each of the segment records being greater than a second preset length; and M playback request times are determined according to the M segment records, the M playback request times corresponding to the M segment records one-to-one, and each of the M playback request times being a transmission time corresponding to the corresponding segment record.

[0097] As an example, the second preset length is 200 bytes.

[0098] When each of the M playback request times corresponds to a segment indicated by the corresponding segment record and the segment contains multiple packets, each of the M playback request times is a transmission time of a first packet in the corresponding segment.

[0099] In some implementations of obtaining the playing request time, the downloaded resource can be identified by matching the clear text information in the traffic or the packet length sequence of the traffic. If there is no preloaded segment within the threshold time range of the resource download, the request time of the resource download can be considered as the playing request time, and it can be directly determined that the downloaded resource is not a preloaded video.

[0100] The method of obtaining the preloading request time is introduced as follows.

[0101] In some implementations, the audio and video stream is collected, the stream is parsed, the uplink segment record and the downlink segment record are obtained, and the segment length is contained in the segment record. If the segment length is greater than a third preset length and less than or equal to a fourth preset length, it can be considered that the transmission time of the segment indicated by the segment length is the preloading request time.

[0102] If the segment contains multiple packets, the transmission time of the segment is the transmission time of the first packet in the segment.

[0103] An example of the third preset length is 800 bytes, and an example of the fourth preset length is 1000 bytes.

[0104] If the segment length is greater than a fifth preset length, it can be considered that the transmission time of the segment indicated by the segment length is the playing loading request time.

[0105] In some implementations, the fifth preset length can be the same as the fourth preset length.

[0106] It can be understood that the values of the third preset length, the fourth preset length and the fifth preset length can be determined based on the message length of the playing loading request and the preloading playing request of the audio and video.

[0107] In some implementations, the period between the transmission time of each loading request and the transmission time of the previous packet until the next loading request or the end of the stream is referred to as a video segment download period. If the aforementioned each loading request is a playing loading request, the video segment download period is a playing video segment download period; if the aforementioned each loading request is a preloading request, the video segment download period is a preloading video segment download period.

[0108] In some implementations, the occurrence of a TCP FIN flag or a RST flag is considered as the end of the stream. If both the TCP FIN flag and the RST flag occur, the earliest occurring flag is taken as the end of stream flag.

[0109] In some implementations, the length of each preloaded video segment is accumulated in the period of downloading the preloaded video segment, and the length of the preloaded video segment is obtained. If the length of the preloaded video segment is less than the seventh preset length, it is determined whether there is a breakpoint continuation preloaded video segment after the preloaded video segment. If there is, the length of the subsequent preloaded video segment is accumulated to the length of the preloaded video segment, and the segment record of the subsequent preloaded video segment is deleted.

[0110] In some implementations, if the current preloaded video segment is the last video segment of the stream, the length of the video segment is less than the seventh preset length, and the first video segment in the first segment after the stream is a preloaded video segment, it can be considered that whether there is a breakpoint continuation preloaded video segment after the current preloaded video segment.

[0111] An example of the seventh preset length is 4000000 bytes. The value of the seventh preset length can be determined based on the characteristics of the breakpoint continuation preloaded video segment of the audio and video.

[0112] In some scenarios, the implementation includes the following: obtaining a video stream of a terminal device in a first time period; obtaining K segment records in the video stream, each segment record in the K segment records containing a segment length, and K being a positive integer; determining N preloaded request times according to K segment lengths corresponding to the K segment records, wherein the N preloaded request times correspond to N segment lengths of the K segment lengths one by one, each segment length in the N segment lengths is greater than a third preset length and less than or equal to a fourth preset length, the segment indicated by the each segment length is an uplink segment, and each preloaded request time in the N preloaded request times is a transmission time of the segment indicated by the corresponding segment length.

[0113] If the segment contains multiple packets, the transmission time of the segment is the transmission time of the first packet in the segment.

[0114] In some implementations, as shown in FIG. 3, before S330, the method can further include S325, that is, determining whether the video is successfully started. If the video is successfully started, S330 is performed. Figure 4

[0115] The method of determining whether the video is successfully started and obtaining the starting time of the video is introduced below.

[0116] ​The lengths of all the downlink segments in the playing video segment download period are accumulated to obtain the length of the playing video segment. If the length is less than the sixth preset length, it can be considered that the playing of the playing video segment fails; otherwise, it can be considered that the playing of the playing video segment succeeds, wherein the playing start time is the transmission time of the segment whose accumulated length is greater than or equal to the sixth preset length for the first time.

[0117] An example of the sixth preset length is 75000 bytes. The value of the sixth preset length can be determined based on the segment length characteristics of the playing start success or failure of the audio and video.

[0118] The method of associating the playing start time with the playing request time is introduced below. The association of the playing start time with the playing request time can be understood as determining which playing start time in the plurality of playing start times and which playing request time in the plurality of playing request times belong to the same video.

[0119] In some implementations, the playing request times and the playing segment request times are sorted in the order from front to back to obtain a list. For each playing request time t, all the playing segment request times in the range of t-0.5 to t+0.5 are taken to obtain the playing start information of the playing segments corresponding to the playing segment request times, the playing start information including whether the playing start succeeds (or fails) and the playing start time in the case of the playing start success. If the playing start fails, it is recorded that the video playing start fails.

[0120] It can be understood that the 0.5 therein is only an example, which can be determined based on the playing characteristics of the video in different scenarios.

[0121] The method of obtaining the first frame delay of the terminal transmitting the short video traffic data based on the transport layer security (TLS) protocol and the playing request operation of the sliding video is introduced below. Figure 5 Or Figure 6 The method flow of obtaining the first frame delay of the embodiment of the application is introduced.

[0122] As shown in Figure 5 , the method of obtaining the first frame delay of the embodiment of the application includes steps S510, S520, S530, S540, S550 and S560.

[0123] S510, collect the short video traffic transmitted based on the TLS protocol.

[0124] In some implementations, TLS traffic data during the playback of short videos on a terminal is collected. For example, traffic data can be captured through packet capture tools such as Wireshark or TCPDump, or it can be obtained through mirroring, or other equivalent methods.

[0125] In some implementations, the server name indication (sni) can be used to filter out the audio / video stream and the stream where logs are reported when swiping the video.

[0126] S520, extract video segment information. The video segment information includes start-play information and segment request time.

[0127] For example, packets with the same five-tuple (source IP, source port, destination IP, destination port, transport layer protocol) are grouped into the same stream, and the stream is parsed to obtain the tls.record.length of the uplink and downlink after the tls handshake, and the obtained tls.record.length is sorted by time.

[0128] If the uplink tls.record.length > 1000 bytes, the time corresponding to this uplink tls.record.length is recorded as the play segment request time; if 800 < uplink tls.record.length <= 1000 bytes, the time corresponding to this uplink tls.record.length is recorded as the preload segment request time. The play segment request time and the preload segment request time are collectively referred to as the segment request time.

[0129] Among them, the time from the start of each segment request time until the time of the previous packet before the next segment request time or the end of the stream can be recorded as the segment download time. If the segment request time is the play segment request time, the corresponding segment download time is the play segment download time; if the segment request time is the preload segment request time, the corresponding segment download time is the preload segment download time. The set of downlink packets within the play segment download time range is denoted as the play segment, and the set of downlink packets within the preload segment download time range is denoted as the preload segment.

[0130] The downlink tls.record.length in the time range of each play slice download is accumulated, and the time when the accumulated value is first greater than or equal to 75,000 bytes is recorded as the play start time of the video. If the accumulated tls.record.length does not reach 75,000 bytes, the video play start failure is recorded. The play start success and the play start time, or the play start failure can be recorded as the play start information of the play slice.

[0131] The downlink tls.record.length in the time range of each preload slice download is accumulated, and the length of the preload slice is recorded. If the length of the preload is less than 4,000,000 bytes, the breakpoint resume judgment is performed.

[0132] If it belongs to the breakpoint resume, the length of the resumed preload slice is added to obtain the final preload length of the preload slice, and the record of the resumed preload slice is deleted.

[0133] One judgment method of the breakpoint resume: If the preload slice is the last slice of the tls stream, the length of the slice is less than 4,000,000 bytes, and the newly created tls stream downloads the preload slice immediately after the end of the tls stream, the preload slice downloaded by the newly created tls stream is the resumed preload slice.

[0134] S530, extracts the sliding video time according to the stream where the sliding video time reporting log is located.

[0135] The stream where the sliding video time reporting log is located is parsed, the tls.record.length of the uplink and the downlink after the tls handshake is obtained, and these tls.record.length are sorted in the order of time from front to back to obtain a length sequence.

[0136] The length sequence is subjected to a packet length sequence matching process to obtain one or more specific packet length sequences. Each specific packet length sequence can include four consecutive packet lengths, the transmission directions of the four packets corresponding to the packet lengths are uplink, downlink, uplink and downlink in turn, and each packet length is in the range. The time corresponding to the first packet length in the specific packet length sequence is recorded as the sliding video time.

[0137] S540, judges whether it is a preloaded video. If it is a preloaded video, S550 is performed, otherwise S560 is performed.

[0138] The preload slice request time obtained in step S520 and the sliding video time obtained in step S530 are summarized and sorted in the order of time from front to back to obtain a sequence.

[0139] Assume a variable x=0, and traverse the aforementioned sequence by time; if the traversed time is the preload segment request time, then x=x+1; if the traversed time is the sliding video time, then x=x-1; if the traversed time is the sliding video time, and x is 0 before x=x-1 is calculated, then it is determined that the video to be played during this sliding video is not preloaded, otherwise it is determined that the video to be played during this sliding video is preloaded.

[0140] In some implementations, x>=0 and x<=5 may be maintained. For example, when x=x+1 is greater than 5, x is reset to 5; when x=x−1 is less than 0, x is reset to 0.

[0141] S550: Determine that the first frame delay of the video is 0.

[0142] S560: Calculate the first frame delay.

[0143] Summarize the non-preloaded sliding video time in step S540 and the request time of the play segment in S520, sort the sliding video time and the play video segment request time in order of time, and obtain a list.

[0144] For each sliding video time t, take all play segment request times between t – 0.5 and t + 0.5 and obtain the corresponding start information for these play video segment request times. If the start information contains a start failure, the video corresponding to this sliding video is determined to have failed to start. Otherwise, the first frame delay of the video corresponding to this sliding video is determined to be equal to the start time minus the sliding video time.

[0145] like Figure 6 As shown, the method for obtaining the first frame delay in one embodiment of the present application includes steps S610, S620, S630, S640, S650 and S660. Figure 5 The difference between the illustrated embodiments lies in the different ways of obtaining the sliding video time.

[0146] S610 collects short video traffic transmitted based on the TLS protocol.

[0147] This step can refer to step S510, with the following differences: the filtered streams include audio and video streams and short video cover download streams.

[0148] S620: Extract video segment information.

[0149] This step may refer to step S520.

[0150] S630: Extract the sliding video time according to the short video cover download stream.

[0151] Parse the short video cover download stream, get the tls.record.length after tls handshake, if tls.record.length>200 bytes, record the time of the first package of tls.record as the sliding video time.

[0152] S640, determine whether it is a preloaded video. If it is a preloaded video, execute S650, otherwise execute S660.

[0153] This step can refer to step S540.

[0154] S650, determine the first frame delay of the video as 0.

[0155] S660, calculate the first frame delay.

[0156] This step can refer to step S560.

[0157] The following takes the terminal transmitting short video traffic data through the QUIC protocol and playing the request operation as a sliding video as an example, and introduces the method flow of acquiring the first frame delay of the embodiments of the application in combination with Figure 7 .

[0158] As shown in Figure 7 , the method for acquiring the first frame delay of an embodiment of the application comprises steps S710, S720, S730, S740, S750 and S760.

[0159] S710, collect short video traffic based on the QUIC protocol.

[0160] This step can refer to step S510, and the difference comprises that the collected short video traffic protocol is not TLS, but QUIC.

[0161] S720, extract video segment information.

[0162] In some implementations, messages with the same five-tuple (source IP, source port, destination IP, destination port, transport layer protocol) are attributed to the same flow.

[0163] In other implementations, messages with the same QUIC connection ID are attributed to the same QUIC flow.

[0164] Parse the short video stream, get the QUIC package payload length, record the transmission direction, and the transmission direction is uplink or downlink.

[0165] Sort the packages by time, wherein the continuous package payload length of uplink is merged as “merged payload length”, and the time of “merged payload length” is recorded as the time of the first package in the merged packages.

[0166] The time record of the "merged load length" > 1000 bytes is the play segment request time, and the time record of the "merged load length" < 1000 bytes is the preload segment request time.

[0167] The time from the start of each segment request to the time of the previous packet or the end of the previous packet of the stream is the segment download time.

[0168] For each play segment download time range, the downlink QUIC packet load length is accumulated, and the time when the accumulated value is greater than or equal to 75,000 bytes for the first time is recorded as the video start time; if the last accumulated value cannot reach 75,000 bytes, the corresponding video start failure is recorded.

[0169] For each preload segment, the downlink QUIC packet load length in the download time range is accumulated, and the length of the video segment is recorded. If the length is less than 4,000,000 bytes, a breakpoint continuation judgment is performed.

[0170] If it is a breakpoint continuation, the length of the continued preload segment is added, and the record of the continued preload segment is deleted.

[0171] One judgment method of breakpoint continuation: the video segment is the last segment of the QUIC stream, the segment length is less than 4,000,000 bytes, a new QUIC stream is immediately created after the end of the QUIC stream, and the first video segment downloaded by the new QUIC stream is a preload segment.

[0172] S730, according to the sliding video time report log flow, extract the sliding video time.

[0173] Messages with the same five-tuple (source IP, source port, destination IP, destination port, and transport layer protocol) are attributed to the same stream, or messages with the same QUIC connection ID are attributed to the same QUIC stream.

[0174] Analyze the sliding video time report log flow, obtain the uplink and downlink QUIC packet load length, combine the continuous packet load length in the same direction to obtain the "merged load length", record the time of the first packet in the merged packets as the time of the merged load length, and sort the merged load length by time to obtain the merged load length sequence.

[0175] The packet length sequence is subjected to a packet length sequence matching process to obtain one or more specific combined load length sequences, each of which can contain four continuous combined load lengths, the transmission directions of the packets corresponding to the four combined load lengths being uplink, downlink, uplink and downlink in turn, and each of the combined load lengths being in the range, and the time corresponding to the first combined load length in the specific combined load length sequence is recorded as the sliding video time.

[0176] S740, determining whether the video is a preloaded video. If yes, S750 is performed, otherwise S760 is performed.

[0177] This step can refer to step S540.

[0178] S750, determining that the first frame delay of the video is 0.

[0179] S760, calculating the first frame delay.

[0180] This step can refer to step S560.

[0181] In another implementation of the embodiment, S730 can be replaced by extracting the sliding video time from the short video cover download stream. The implementation of extracting the sliding video time from the short video cover download stream can refer to step S630.

[0182] The method embodiments of the present application are described in detail above in combination with Figures 1 to 7 , and the device embodiments of the present application are described in detail below in combination with Figure 8 and Figure 9 . It should be understood that the description of the method embodiments and the description of the device embodiments correspond to each other, and therefore, the parts not described in detail can be referred to the foregoing method embodiments.

[0183] Figure 8 is a schematic structural diagram of an apparatus for obtaining a first frame delay provided by an embodiment of the present application. As shown in Figure 8 , the apparatus 800 includes a judging unit 801 and a determining unit 802.

[0184] The apparatus 800 can be used to implement the method of the embodiments shown in Figure 3 , Figure 4 , Figure 5 , Figure 6 or Figure 7 . The judging unit 801 is used to implement the actions related to judging, and the determining unit 802 is used to implement the actions related to determining, obtaining, collecting, etc.

[0185] For example, the judging unit 801 is used to implement S310, and the determining unit 802 is used to implement S320 and S330.

[0186] Figure 9 A structural diagram of an apparatus is provided for another embodiment of the present application. As shown in Figure 9 The apparatus 900 includes processing circuitry 901 and 902. The processing circuitry 901 and the communication circuitry 902 are coupled with each other.

[0187] It can be understood that the processing circuitry can be one or more processors, or can be all or part of a processor.

[0188] It can be understood that the communication circuitry 902 can be an input / output interface.

[0189] Optionally, the apparatus 900 can further include a memory 903 for storing instructions executed by the processing circuitry 901 or storing input data required by the processing circuitry 901 to run instructions or storing data generated after the processing circuitry 901 runs instructions.

[0190] It can be understood that the memory 903 can be located outside the processing circuitry 901, or located inside the processing circuitry 901.

[0191] As an example, the processing circuitry 901 is configured to implement the functions of the determination unit 802 and the determination unit 802.

[0192] As an example, the apparatus 900 can be a network device, or a chip applied to a network device. The network device can be a terminal, a router, an operator network device, a firewall or a cloud server, etc. Figure 1

[0193] When the apparatus 900 is a chip, the communication circuitry can be an input / output circuit, a bus, a pin or other types of communication interfaces, wherein the input circuit in the input / output circuit can be used for receiving, and the output interface can be used for sending.

[0194] Some embodiments of the present application also provide a computer readable storage medium, which includes computer instructions. When the computer instructions are run on a processor, the method in any of the above embodiments can be implemented.

[0195] Some embodiments of the present application also provide a computer program product, which includes computer instructions. When the computer instructions are run on a processor, the method in any of the above embodiments can be implemented.

[0196] ​It can be understood that the processor in the embodiments of the present application can be all or part of the circuit for processing functions of the following devices: a central processing unit (CPU), and can also be other general-purpose processors, digital signal processors (DSPs), field programmable gate arrays (FPGAs) or other programmable logic devices, transistor logic devices, hardware components or any combination thereof. The general-purpose processor can be a microprocessor or any conventional processor.

[0197] The method steps in the embodiments of the present application can be realized by hardware or by the processor executing software instructions. The software instructions can be composed of corresponding software modules, and the software modules can be stored in a random access memory, a flash memory, a read-only memory, a programmable read-only memory, an erasable programmable read-only memory, an electrically erasable programmable read-only memory, a register, a hard disk, a mobile hard disk, a CD-ROM or any other form of storage medium well known in the art. An exemplary storage medium is coupled to the processor, so that the processor can read information from and write information to the storage medium. Of course, the storage medium can also be an integral part of the processor. The processor and the storage medium can be located in a chip, such as an application-specific integrated circuit (ASIC), or a chip system, such as a system on a chip (SOC). In addition, the chip or chip system can be located in a network device or a terminal device. Of course, the processor and the storage medium can also exist as discrete components in the network device or the terminal device.

[0198] In the above embodiments, all or part of the embodiments can be implemented by software, hardware, firmware or any combination thereof. When implemented by software, all or part of the embodiments can be implemented in the form of a computer program product. The computer program product includes one or more computer programs or instructions. When the computer programs or instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of the present application are performed. The computer can be a general purpose computer, a special purpose computer, a computer network, a network device, a user equipment or other programmable apparatus. The computer programs or instructions can be stored in a computer readable storage medium or transmitted from one computer readable storage medium to another computer readable storage medium, for example, the computer programs or instructions can be transmitted from one website site, computer, server or data center to another website site, computer, server or data center through wired or wireless manner. The computer readable storage medium can be any available medium accessible by a computer or a data storage device such as a server, data center and the like integrated with one or more available media. The available media can be a magnetic medium, such as a floppy disk, a hard disk, a magnetic tape; an optical medium, such as a digital video disc; and a semiconductor medium, such as a solid state disk.

[0199] In various embodiments of the present application, the terms and / or descriptions of different embodiments are consistent and can be referred to each other if there is no special description and logical conflict, and the technical features in different embodiments can be combined to form new embodiments according to their inherent logical relationship.

[0200] It can be understood that various numerical numbers involved in the embodiments of the present application are only distinguished for convenience of description, and are not used to limit the scope of the embodiments of the present application. The size of the serial number of the above processes does not mean the execution order, and the execution order of the processes should be determined according to its function and inherent logic.

Claims

1. A method for obtaining first frame delay, characterized in that: The method comprises: determining whether the video has been preloaded when played in the terminal device; if the video has been preloaded when played in the terminal device, determining that the first frame time delay of the video is 0; if the video has not been preloaded when played in the terminal device, determining the first frame time delay of the video according to the video start time and the play request time.

2. The method of claim 1, wherein, The method comprises: obtaining M play request times of the terminal device in a first time period, M being a positive integer; obtaining N preloading segment request times of the terminal device in the first time period, N being a positive integer; for each play request time in the M play request times, if the difference between the number of preloading request times before the each play request time and the number of play request times before the each play request time in the M play request times and N preloading request times is greater than 0, determining that the video corresponding to the each play request time has been preloaded when played; if the difference between the number of preloading request times before the each play request time and the number of play request times before the each play request time in the M play request times and N preloading request times is zero or negative, determining that the video corresponding to the each play request time has not been preloaded when played.

3. The method of claim 2, wherein, The method comprises: obtaining a data stream in which a video play request log of the terminal device in the first time period is located; obtaining R segment records in the data stream, each segment record in the R segment records containing a segment length, R being a positive integer; determining M packet length sequences based on R segment lengths corresponding to the R segment records, each packet length sequence in the M packet length sequences containing four segment lengths in a sequence obtained by arranging the R segment lengths in time sequence, and the four segments corresponding to the four segment lengths in time sequence being an uplink segment, a downlink segment, an uplink segment and a downlink segment in turn, and each segment length in the four segment lengths being less than or equal to a first preset length; determining the M play request times based on the M packet length sequences, wherein the M play request times correspond to the M packet length sequences one by one, and each play request time in the M play request times is the transmission time of a segment corresponding to a first segment length in a corresponding packet length sequence.

4. The method of claim 2, wherein, The method comprises: obtaining M segment records in a video cover download stream, each segment record in the M segment records containing a segment length, and each segment length in the M segment records being greater than a second preset length; determining M play request times according to the M segment records, the M play request times corresponding to the M segment records one by one, and each play request time in the M play request times being a transmission time corresponding to a corresponding segment record.

5. The method according to any one of claims 2 to 4, characterized in that, The method comprises: acquire a video stream of the terminal device in the first time period; acquire K segment records in the video stream, each of the K segment records containing a segment length, K being a positive integer; determine N preloading request times according to K segment lengths corresponding to the K segment records, wherein the N preloading request times correspond to N segment lengths of the K segment lengths, each of the N segment lengths is greater than a third preset length and less than or equal to a fourth preset length, each of the segment lengths indicates an uplink segment, and each of the N preloading request times is a transmission time of the segment indicated by the corresponding segment length.

6. The method according to any one of claims 1 to 5, characterized in that, The method further comprises: acquiring a video stream of the terminal device in the first time period; acquiring K segment records in the video stream, each of the K segment records containing a segment length, K being a positive integer; determining P playback segment request times according to K segment lengths corresponding to the K segment records, wherein the P playback segment request times correspond to P segment lengths of the K segment lengths, each of the P segment lengths is greater than a fourth preset length, each of the segment lengths indicates an uplink segment, each of the P playback segment request times is a transmission time of the segment indicated by the corresponding segment length, and P is a positive integer less than or equal to K; determining F video start times according to the P playback segment request times, wherein the F video start times correspond to F playback segment request times of the P playback segment request times, a total length of the playback segment corresponding to each of the F playback segment request times is greater than or equal to a preset length, each of the F video start times is a time at which a length accumulation value is greater than or equal to a preset length threshold for the first time with the corresponding playback segment request time as a starting time, and F is a positive integer less than or equal to P.

7. The method according to any one of claims 1 to 6, characterized in that, The method further comprises: determining a first frame delay of the video according to the video start time and the playback request time of the video.

8. An apparatus for acquiring a first frame latency, the apparatus comprising: The difference between the video start time and the playback request time is determined as the first frame delay. The method further comprises:

9. A computer-readable storage medium, characterized in that, a processor and a memory, the processor being coupled to the memory, and the memory being configured to store a computer program, the computer program being executed by the processor to cause the apparatus to perform the method according to any one of claims 1 to 7.

10. A computer program product, characterised in that, The computer readable storage medium stores a computer program, and when the computer program is run on a computer, the computer is caused to perform the method according to any one of claims 1 to 7. The computer readable storage medium stores a computer program, and when the computer program is run on a computer, the computer is caused to perform the method according to any one of claims 1 to 7. The computer readable storage medium stores a computer program, and when the computer program is run on a computer, the computer is caused to perform the method according to any one of claims 1 to 7.

Images