Video stream delay observation method and device, electronic equipment and storage medium

Abstract

The invention discloses a video stream delay observation method and device, electronic equipment and a storage medium. The method comprises the following steps: acquiring supplementary enhancement information of video sequence data; pushing the video sequence data to each node server, analyzing the supplemental enhancement information through each node server, and counting node delay information corresponding to each node server; and generating a video stream transmission link according to the supplemental enhancement information and the node delay information corresponding to each node server, and performing video stream delay observation by using the video stream transmission link. According to the invention, a reasonable and efficient video stream delay observation scheme can be provided, accurate video stream delay observation and video stream node tracking are realized, the accuracy and efficiency of video stream delay observation are improved, and the method can be widely applied to the technical field of digital media.

Description

Technical Field

[0001] The present application relates to the field of digital media technology, and in particular to a method, device, electronic device and storage medium for observing video stream delay. Background Art

[0002] In the field of live video, end-to-end video transmission delay is usually observed in a subjective way, that is, the current clock screen is pushed at the sender, and then the live stream screen of the server is pulled. By comparing the screens, the end-to-end video transmission delay can be obtained. However, this method cannot be obtained objectively, so there is no way to form an observable indicator. Usually, the video stream will pass through more than one intermediate node, and the delay between each node cannot be calculated.

[0003] Other methods use video frame timestamp information to observe end-to-end delay, but this video frame timestamp may be modified on the server or player side, resulting in inaccurate timestamps that cannot be accurately measured.

[0004] Therefore, the relevant technology cannot achieve accurate video stream delay observation, resulting in low accuracy and efficiency of video stream delay observation, and there is no reasonable and efficient video stream delay observation solution. Summary of the invention

[0005] The main purpose of the embodiments of the present application is to propose a video stream delay observation method, device, electronic device and storage medium, which can realize video stream delay observation and improve the accuracy and efficiency of video stream delay observation.

[0006] On the one hand, an embodiment of the present application provides a video stream delay observation method, the method comprising the following steps: Obtaining supplementary enhancement information for video sequence data; Push the video sequence data to each node server, obtain the supplementary enhancement parsing information after each node server parses the supplementary enhancement information, and count the node delay information corresponding to each node server according to the supplementary enhancement parsing information corresponding to each node server; A video stream transmission link is generated according to the supplementary enhancement information and the node delay information corresponding to each of the node servers, and the video stream delay observation is performed using the video stream transmission link.

[0007] In some embodiments, the obtaining of supplementary enhancement information of the video sequence data specifically includes: Acquire the video sequence data generated by a video encoding end, wherein the video encoding end is a node server end that generates the video sequence data using a video encoding technology, and the video sequence data includes a plurality of video frames; Obtaining a preset supplementary enhancement information format; the supplementary enhancement information format at least includes a current node identifier, a current node network protocol address, a current video frame sequence number, and an absolute timestamp of a node parsing or generating a current video frame; Obtain the node identifier of the video encoding end and the current absolute timestamp of each video frame generated by the video encoding end, and generate the supplementary enhancement information corresponding to each video frame according to the supplementary enhancement information format, the node identifier of the video encoding end and the current absolute timestamp of each video frame generated by the video encoding end.

[0008] In some embodiments, the streaming of the video sequence data to each node server, obtaining the supplementary enhancement parsing information after each node server parses the supplementary enhancement information, and counting the node delay information corresponding to each node server according to the supplementary enhancement parsing information corresponding to each node server, specifically includes: Acquire the encoding generation time of each video frame in the video sequence data, and add the encoding generation time of each video frame to the supplementary enhancement information corresponding to each video frame; For each of the node servers, the current recording time of each of the video frames parsed by the node server is obtained, and the video frame parsing interval is determined according to the current recording time of the adjacent video frames, and the encoding generation time interval of the adjacent video frames is obtained according to the supplementary enhanced parsing information corresponding to each of the node servers, and the node delay interval corresponding to the node server is determined according to the video frame parsing interval and the encoding generation time interval; The node identifier corresponding to each of the node servers is obtained, and the node delay information corresponding to each of the node servers is generated according to the node identifier corresponding to each of the node servers and the node delay interval.

[0009] In some embodiments, the method further comprises: Generate video frame supplementary enhancement information corresponding to each of the video frames in each of the node servers according to the supplementary enhancement information format and the node delay information corresponding to each of the node servers; For each of the video frames, the video frame supplementary enhancement information corresponding to the video frame in each of the node servers is added to the supplementary enhancement information corresponding to the video frame.

[0010] In some embodiments, obtaining the encoding generation time interval of adjacent video frames according to the supplementary enhancement parsing information corresponding to each of the node servers specifically includes: Obtaining a first absolute timestamp and a second absolute timestamp according to the supplementary enhanced parsing information obtained after the node server parses the adjacent video frames, wherein the first absolute timestamp is a current absolute timestamp when a preceding video frame among the adjacent video frames is generated in the video encoding end, and the second absolute timestamp is a current absolute timestamp when a succeeding video frame among the adjacent video frames is generated in the video encoding end; The encoding generation time interval is determined according to the first absolute timestamp and the second absolute timestamp.

[0011] In some embodiments, generating a video stream transmission link according to the supplementary enhancement information and the node delay information corresponding to each of the node servers, and using the video stream transmission link to perform video stream delay observation specifically includes: Determining a video stream transmission path corresponding to the video sequence data according to the supplemental enhancement information corresponding to each of the video frames; The video stream transmission link is generated according to the video stream transmission path and the node delay information corresponding to each of the node servers.

[0012] In some embodiments, generating the video frame supplementary enhancement information corresponding to each of the video frames in each of the node servers according to the supplementary enhancement information format and the node delay information corresponding to each of the node servers specifically includes: The node server to which the video sequence data is currently streaming is obtained as the current node server, and the absolute timestamp corresponding to each video frame at the previous node server is obtained according to the supplementary enhanced parsing information obtained after each video frame is parsed at the current node server, wherein the previous node server is the node server that pushes the video frame to the current node server; According to the node delay interval corresponding to the current node server and the absolute timestamp corresponding to the video frame at the previous node server, the current node absolute timestamp is determined, wherein the current node absolute timestamp is the absolute timestamp of the video frame in the current node server, and the current node absolute timestamp is expressed by the following formula: ; in, is the absolute timestamp of the current node, It is the absolute timestamp of the video frame on the previous node server. The node delay interval corresponding to the current node server; The video frame supplementary enhancement information of the video frame at the current node server is generated according to the current node absolute timestamp and the node delay information corresponding to the current node server.

[0013] On the other hand, an embodiment of the present application provides a video stream delay observation device, the device comprising: The first module is used to obtain supplementary enhancement information of video sequence data; The second module is used to push the video sequence data to each node server, obtain the supplementary enhancement parsing information after each node server parses the supplementary enhancement information, and count the node delay information corresponding to each node server according to the supplementary enhancement parsing information corresponding to each node server; The third module is used to generate a video stream transmission link according to the supplementary enhancement information and the node delay information corresponding to each node server, and use the video stream transmission link to observe the video stream delay.

[0014] On the other hand, an embodiment of the present application proposes an electronic device, which includes a memory and a processor, wherein the memory stores a computer program, and the processor implements the video stream delay observation method described above when executing the computer program.

[0015] On the other hand, an embodiment of the present application proposes a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the video stream delay observation method described above is implemented.

[0016] The embodiments of the present application include at least the following beneficial effects: the present application provides a method, device, electronic device and storage medium for observing video stream delay, which pushes the video sequence data to each node server by acquiring the supplementary enhancement information of the video sequence data, analyzes the supplementary enhancement information through each node server, counts the node delay information corresponding to each node server, generates a video stream transmission link based on the supplementary enhancement information and the node delay information corresponding to each node server, and uses the video stream transmission link to observe the video stream delay. The present application can provide a reasonable and efficient video stream delay observation scheme, realize accurate video stream delay observation and video stream node tracking, and improve the accuracy and efficiency of video stream delay observation. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 is a flow chart of a video stream delay observation method provided by an embodiment of the present application; Figure 2 is a flowchart of step S101 of an embodiment of the present application; Figure 3 It is a structural schematic diagram of the video stream transmission path in an embodiment of the present application; Figure 4 is a flowchart of step S102 of an embodiment of the present application; Figure 5is another flow chart of a video stream delay observation method provided by an embodiment of the present application; Figure 6 is a flowchart of step S501 of an embodiment of the present application; Figure 7 is a flowchart of step S103 of an embodiment of the present application; Figure 8 is another structural schematic diagram of the video transmission path in an embodiment of the present application; Fig. 9 It is a structural schematic diagram of a video stream delay observation device provided in an embodiment of the present application; Fig.10 It is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0018] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application is further described in detail below in conjunction with the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the embodiments of the present application. They are only examples of devices and methods consistent with some aspects of the embodiments of the present application as detailed in the attached claims.

[0019] It is understood that the terms "first", "second", etc. used in this application can be used to describe various concepts in this article, but unless otherwise specified, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another concept. For example, without departing from the scope of the embodiment of the present application, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the words "if" and "if" as used herein can be interpreted as "at the time of" or "when" or "in response to determination".

[0020] The terms "at least one", "multiple", "each", "any", etc. used in this application, at least one includes one, two or more, multiple includes two or more, each refers to each of the corresponding multiple, and any refers to any one of the multiple.

[0021] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0022] It should be noted that in each specific implementation of the present application, when it comes to the need to perform relevant processing based on data related to user identity or characteristics such as user information, user behavior data, user historical data, and user location information, the user's permission or consent will be obtained first, and the collection, use, and processing of these data will comply with relevant laws, regulations, and standards. In addition, when the embodiment of the present application needs to obtain the user's sensitive personal information, the user's separate permission or consent will be obtained through a pop-up window or by jumping to a confirmation page. After clearly obtaining the user's separate permission or consent, the necessary user-related data for the normal operation of the embodiment of the present application will be obtained.

[0023] Before describing the embodiments of the present application in detail, some nouns and terms involved in the embodiments of the present application are first described. The nouns and terms involved in the embodiments of the present application are subject to the following explanations.

[0024] 1) H.264 encoding: It is a widely used digital video compression standard, jointly developed by the ITU-T Video Coding Experts Group (VCEG) and the ISO / IEC Moving Picture Experts Group (MPEG). H.264 provides higher compression efficiency than previous video coding standards.

[0025] 2) H.265 encoding: It is a new generation of video compression standard jointly developed by ITU-T Video Coding Experts Group (VCEG) and ISO / IEC Moving Picture Experts Group (MPEG). It is a follow-up improvement of the previous H.264 / AVC standard, aiming to provide higher compression efficiency and better image quality.

[0026] 3) NAL: NAL is a concept in the H.264 and H.265 video coding standards, which defines the encapsulation format of video data. The NAL unit contains the data of the video frame and some control information, such as the start and end of the frame. NAL is designed to split the video frame into smaller units for network transmission and error recovery.

[0027] 4) NALU: NALU is a unit of NAL and is the container that actually carries the encoded video data. Each NALU has its own type and length information, as well as possible timestamps and importance levels. The type of NALU defines the type of data it contains, such as IDR frames, P frames, B frames, or important control information.

[0028] 5) SEI: SEI (Supplemental Enhancement Information) is a term in video coding standards, especially in H.264 / AVC and H.265 / HEVC, which refers to supplementary enhancement information. SEI is a metadata in the video stream that is used to provide additional information that can enhance the decoding process of the video or provide other information about the video content.

[0029] 6) NTP: (Network Time Protocol, NTP) is a protocol used to synchronize computer system time, ensuring consistent time standards between computers and network devices on the Internet.

[0030] In the field of live video, end-to-end video transmission delay is usually observed in a subjective way, that is, the current clock screen is pushed at the sender, and then the live stream screen of the server is pulled. By comparing the screens, the end-to-end video transmission delay can be obtained. However, this method cannot be obtained objectively, so there is no way to form an observable indicator. Usually, the video stream will pass through more than one intermediate node, and the delay between each node cannot be calculated.

[0031] Other methods use video frame timestamp information to observe end-to-end delay, but this video frame timestamp may be modified on the server or player side, resulting in inaccurate timestamps that cannot be accurately measured.

[0032] Therefore, the relevant technology cannot achieve accurate video stream delay observation, resulting in low accuracy and efficiency of video stream delay observation, and there is no reasonable and efficient video stream delay observation solution.

[0033] During live video broadcasts, the video delay from the host to the audience is usually several seconds to more than ten seconds, while the video conferencing delay is usually within 200 milliseconds. Different scenarios have different requirements for image delay. In video streaming, video delay is a relatively important indicator, which can be used to guide more technical optimization directions.

[0034] Based on this, the embodiments of the present application propose a video stream delay observation method, device, electronic device and storage medium that do not rely on timing services. First, SEI data of video sequence data must be generated at the video encoding end. Each node server has the ability to parse and update SEI data. When updating SEI data, information insertion is required according to specific rules and logic. At the first node server, i.e., the video encoding end, relevant information such as timestamps are added to the SEI data of the video sequence data. The timestamp is a relative timestamp or an absolute timestamp. Since subsequent nodes perform relative operations based on the timestamp, this does not rely on the machine clock and related timing services. Based on the parsed SEI data, the video stream link delay and video stream link tracking can be calculated to achieve high-precision video stream delay observation.

[0035] Reference Figure 1 , Figure 1 This is an optional flowchart of a video stream delay observation method provided by an embodiment of the present application. The method may include but is not limited to steps S101 to S103: Step S101, obtaining supplementary enhancement information of video sequence data; Step S102, streaming the video sequence data to each node server, obtaining supplementary enhancement parsing information after each node server parses the supplementary enhancement information, and counting node delay information corresponding to each node server according to the supplementary enhancement parsing information corresponding to each node server; Step S103: Generate a video stream transmission link according to the supplementary enhancement information and the node delay information corresponding to each node server, and use the video stream transmission link to perform video stream delay observation.

[0036] In some embodiments, the supplementary enhancement information is SEI data, and the video sequence data includes multiple video frames. When encoding the video data, each video frame has its corresponding generated SEI data. The SEI data can be used to record the absolute timestamp of the video frame generation, the video frame sequence number, the name of the video sequence to which the video frame belongs, the current node where the video frame is located, and the absolute timestamp when the current node receives the video frame, and other video frame streaming information.

[0037] In some embodiments, optionally, SEI data is serialized into binary during video stream transmission. After receiving the SEI data, each node server deserializes the SEI data to complete parsing and obtain corresponding supplementary enhanced parsing information.

[0038] In some embodiments, reference Figure 2 , Figure 2 This is an optional flowchart of step S101 of the embodiment of the present application. Step S101 may include but is not limited to steps S201 to S203: Step S201, obtaining video sequence data generated by a video encoding end, wherein the video encoding end is a node server end that generates video sequence data using video encoding technology; Step S202, obtaining a preset supplementary enhancement information format; the supplementary enhancement information format at least includes a current node identifier, a current node network protocol address, a current video frame sequence number, and an absolute timestamp of the node parsing or generating the current video frame; Step S203, obtain the node identifier of the video encoding end and the current absolute timestamp of each video frame generated by the video encoding end, and generate the supplementary enhancement information corresponding to each video frame according to the supplementary enhancement information format, the node identifier of the video encoding end and the current absolute timestamp of each video frame generated by the video encoding end.

[0039] In some embodiments, optionally, in the H264 code stream, there are various NALUs, where the SEI type is defined as 6. The following is the definition of the NALU type: NALU: StartCode+NALUHeader+NALUPayload, where StartCode is usually 0x000001 or 0x00000001; NALUHeader is positioned as: forbidden_zero_bit(1bit); nal_ref_idc(2bit); nal_unit_type(5bit); Among them, when nal_unit_type is 6, NALU is of SEI type.

[0040] The internal definition of SEI is shown in Table 1 below: Table 1 SEI internal definition table

[0041] In some embodiments, according to the above SEI internal definition, a supplementary enhancement information format is customized. Exemplarily, the supplementary enhancement information format is shown in Table 2 below: Table 2 Supplementary enhancement information format

[0042] Among them, when the video frame is generated at the video encoding end, the field ts is the absolute timestamp of the video frame generation. When the video frame is pushed to other node servers except the video encoding end, the field ts is the absolute timestamp of the video frame parsed by the current node.

[0043] In step S102 of some embodiments, the node server may include but is not limited to a video live streaming end (such as the above-mentioned video encoding end), an edge server, a transcoding server, a central server, and a video live player. Figure 3 , Figure 3 It is an optional structural diagram of the video stream transmission path in the embodiment of the present application, wherein the video sequence data is generated from the live video streaming end, passes through the edge server, transcoding server and central server in sequence, and is finally streamed to the live video player, i.e., the audience end.

[0044] In some embodiments, reference Figure 4 , Figure 4 This is an optional flowchart of step S102 of the embodiment of the present application. Step S102 may include but is not limited to steps S301 to S303: Step S301, obtaining the encoding generation time of each video frame in the video sequence data, and adding the encoding generation time of each video frame to the supplementary enhancement information corresponding to each video frame; Step S302: for each node server, obtain the current recording time of each video frame parsed by the node server, determine the video frame parsing interval according to the current recording time of the adjacent video frames, obtain the encoding generation time interval of the adjacent video frames according to the supplementary enhanced parsing information corresponding to each node server, and determine the node delay interval corresponding to the node server according to the video frame parsing interval and the encoding generation time interval; Step S303, obtaining the node identifier corresponding to each node server, and generating the node delay information corresponding to each node server according to the node identifier and the node delay interval corresponding to each node server.

[0045] In step S302 of some embodiments, optionally, the node server parses the supplementary enhancement information of each video frame to obtain the encoding generation time interval of adjacent video frames, which specifically includes steps S401 to S402: Step S401, obtaining a first absolute timestamp and a second absolute timestamp according to the supplementary enhanced parsing information obtained after the node server parses the adjacent video frames, wherein the first absolute timestamp is the current absolute timestamp when the previous video frame in the adjacent video frames is generated in the video encoding end, and the second absolute timestamp is the current absolute timestamp when the subsequent video frame in the adjacent video frames is generated in the video encoding end; Step S402: Determine a code generation time interval according to the first absolute timestamp and the second absolute timestamp.

[0046] In some embodiments, reference Figure 5 , Figure 5This is another optional flow chart of a video stream delay observation method provided in an embodiment of the present application. The above method may also include but is not limited to steps S501 to S502: Step S501, generating video frame supplementary enhancement information corresponding to each video frame in each node server according to the supplementary enhancement information format and the node delay information corresponding to each node server; Step S502: for each video frame, the video frame supplementary enhancement information corresponding to the video frame in each node server is added to the supplementary enhancement information corresponding to the video frame.

[0047] In some embodiments, reference Figure 6 , Figure 6 This is an optional flowchart of step S501 of the embodiment of the present application. Step S501 may include but is not limited to steps S601 to S603: Step S601, obtaining the node server to which the video sequence data is currently streaming as the current node server, and obtaining the absolute timestamp corresponding to each video frame at the previous node server according to the supplementary enhanced parsing information obtained after each video frame is parsed at the current node server, wherein the previous node server is the node server that pushes the video frame to the current node server; Step S602, determining the current node absolute timestamp according to the node delay interval corresponding to the current node server and the absolute timestamp corresponding to the video frame at the previous node server, wherein the current node absolute timestamp is the absolute timestamp of the video frame in the current node server; Step S603: Generate video frame supplementary enhancement information of the video frame at the current node server according to the current node absolute timestamp and the node delay information corresponding to the current node server.

[0048] In some embodiments, the above current node absolute timestamp is expressed by the following formula: ; in, is the absolute timestamp of the current node, It is the absolute timestamp of the video frame on the previous node server. The node delay interval corresponding to the current node server.

[0049] In some embodiments, optionally, assuming that there are adjacent video frames A to D, the current node server obtains the first absolute timestamp by parsing the SEI data of video frame A. , by parsing the SEI data of video frame B, obtain the first absolute timestamp , then the encoding generation time interval is , when the current node server completes the SEI data parsing of video frame A, the corresponding current recording time is , when the current node server completes the SEI data parsing of video frame B, the corresponding current recording time is , the video frame resolution interval is , then the node delay time Then, the absolute timestamps corresponding to video frames B to D on the previous node server are obtained. According to the node delay time and the absolute timestamps corresponding to video frames B to D on the previous node server, the absolute timestamps corresponding to video frames B to D on the current node server are determined. The absolute timestamp corresponding to video frame B on the previous node server is , then the absolute timestamp corresponding to video frame B in the current node server is , the absolute timestamp of video frame C on the previous node server is , the absolute timestamp of video frame C in the current node server is , and so on.

[0050] Then, according to the preset supplementary enhancement information format, the node identifier, node network protocol address, node service version, node service name and the absolute timestamp corresponding to each video frame in the current node server are obtained to generate the video frame supplementary enhancement information of video frames B to D in the current node server.

[0051] In some embodiments, each time a video frame passes through a node server, corresponding video frame supplementary enhancement information is generated in the node server and appended to the supplementary enhancement information of each video frame.

[0052] Exemplarily, according to the supplemental enhancement information format, the SEI data corresponding to the video frame at the node server is generated, and the content of the SEI data is as follows: "id": "c565130a-d458-4ae6-bef7-3104148ea326", "version": "v0.1.0"; "name": "client-sdk"; "fr om_ip": "10.127.192.190"; "current_ip": "10.127.192.179"; "ts": 1718867413000; "key_index": 1; Among them, id is the node identifier corresponding to the current node server, version is the node service version corresponding to the current node server, name is the service name of the current node server, from_ip is the IP address corresponding to the previous node server, the video frame is pushed to the current node server through the previous node server, current_ip is the IP address corresponding to the current node server, ts is the absolute timestamp of the video frame resolved by the current node server, and key_index is the serial number of the video frame in the video sequence data.

[0053] In some embodiments, reference Figure 7 , Figure 7 This is an optional flowchart of step S103 of the embodiment of the present application. Step S103 may include but is not limited to steps S701 to S702: Step S701, determining a video stream transmission path corresponding to the video sequence data according to the supplementary enhancement information corresponding to each video frame; Step S702: Generate a video stream transmission link according to the video stream transmission path and the node delay information corresponding to each node server.

[0054] In some embodiments, exemplarily, an array named authors is used to store SEI data of a video frame after completing video stream transmission. The complete SEI data of a video frame is specifically as follows: {"authors":[{"id":"c565130a-d458-4ae6-bef7-3104148ea326","version":"v0.1.0","name":"client-sdk","from_ip":"","current_ip":"10.127.192.179","ts":1718867419000,"key_index":7},{"id":"q565130a-d458-4ae6-bef7-3104148ea111","version":"v0.1.0","name":"edge-sdk","from_ip":"10.127.192.179","current_ip":"10.127.192.190","ts":1718867419200,"key_index":7},{"id":"c565130a-d458-4ae6-bef7-3104148ea331","version":"v0.1.0","name":"transfer-sdk","from_ip":"10.127.192.190","current_ip":"10.127.192.193","ts":1718867419300,"key_index":7},{"id":"4e6b4275-cca3-4096-839b-322baba6f450","version":"v0.1.0","name":"core-sdk","from_ip":"10.127.192.193","current_ip":"10.127.192.195","ts":1718867419500,"key_index":7}, { "id":"b4af4451-690f-4acc-b3ee-adc5d988ea81","version":"v0.1.0","name":"player-sdk","from_ip":"10.127.192.193","current_ip":"192.169.1.23","ts":1718867419700,"key_index":7}] }; Through the SEI data of the above video frame, the video stream transmission path can be obtained, which is: 10.127.192.179->10.127.192.190 ->10.127.192.193 ->10.127.192.195 ->192.169.1.23.

[0055] According to the above video stream transmission path, a video stream transmission link is generated. The delay between the video stream transmission links is: 10.127.192.179->10.127.192.190: 200ms; 10.127.192.190->10.127.192.193: 100ms; 10.127.192.193->10.127.192.195: 200ms; 10.127.192.195->192.169.1.23: 200ms. It can be obtained that the total delay of the video stream transmission link is 200ms+100ms+200ms+200ms=700ms.

[0056] In some embodiments, reference Figure 8 , Figure 8 It is another optional structural diagram of the video transmission path in the embodiment of the present application, which includes a video live broadcast host end, an edge server, a transcoding server, a central server and a video live broadcast viewer end. The video sequence data is generated by video encoding technology at the video live broadcast host end. First, the video sequence data is transmitted from the video live broadcast host end to the edge server. After the edge server parses the SEI data in the video sequence data, the first SEI additional data of the video sequence data in the edge server is generated (such as the video frame supplementary enhancement information corresponding to each video frame mentioned above) and inserted into the SEI data. Then, the video sequence data is transmitted from the edge server to the transcoding server. Through transcoding After the server parses the SEI data in the video sequence data, it generates the second SEI additional data of the video sequence data in the transcoding server (such as the video frame supplementary enhancement information corresponding to each video frame mentioned above) and inserts it into the SEI data. Then, the video sequence data is transmitted from the transcoding server to the central server. After the central server parses the SEI data in the video sequence data, it generates the third SEI additional data of the video sequence data in the central server (such as the video frame supplementary enhancement information corresponding to each video frame mentioned above) and inserts it into the SEI data. Finally, the video sequence data is transmitted from the central server to the live video viewer end, and the video sequence data is played at the live video viewer end.

[0057] Reference Fig. 9 , Fig. 9 : is an optional structural diagram of a video stream delay observation device provided in an embodiment of the present application, the device is used to implement the above-mentioned video stream delay observation method, and the device may include: The first module is used to obtain supplementary enhancement information of video sequence data; The second module is used to push the video sequence data to each node server, obtain the supplementary enhancement parsing information after each node server parses the supplementary enhancement information, and count the node delay information corresponding to each node server according to the supplementary enhancement parsing information corresponding to each node server; The third module is used to generate a video stream transmission link according to the supplementary enhancement information and the node delay information corresponding to each node server, and use the video stream transmission link to observe the video stream delay.

[0058] It can be understood that the contents of the above method embodiments are all applicable to the present device embodiments, the functions specifically implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0059] The embodiment of the present application also provides an electronic device, the electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the above-mentioned video stream delay observation method when executing the computer program. The electronic device can be any smart terminal including a tablet computer.

[0060] It can be understood that the contents of the above method embodiments are all applicable to the present device embodiments, the functions specifically implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0061] See also Fig.10 , Fig.10 The hardware structure of an electronic device of another embodiment is illustrated, and the electronic device includes: The processor 901 may be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application; The memory 902 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 902 can store an operating system and other application programs. When the technical solution provided in the embodiment of this specification is implemented by software or firmware, the relevant program code is stored in the memory 902, and the processor 901 calls and executes the video stream delay observation method of the embodiment of the present application; Input / output interface 903, used to implement information input and output; Communication interface 904, used to realize communication interaction between the device and other devices, which can be realized through wired mode (such as USB, network cable, etc.) or wireless mode (such as mobile network, WIFI, Bluetooth, etc.); A bus 905 that transmits information between various components of the device (e.g., the processor 901, the memory 902, the input / output interface 903, and the communication interface 904); The processor 901 , the memory 902 , the input / output interface 903 and the communication interface 904 are connected to each other in communication within the device via a bus 905 .

[0062] An embodiment of the present application also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the above-mentioned video stream delay observation method is implemented.

[0063] It can be understood that the contents of the above method embodiments are all applicable to the present storage medium embodiments, the functions specifically implemented by the present storage medium embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0064] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0065] The embodiments of the present application provide a method, device, electronic device and storage medium for observing video stream delay, which obtains supplementary enhancement information of video sequence data, pushes the video sequence data to each node server, parses the supplementary enhancement information through each node server, counts the node delay information corresponding to each node server, generates a video stream transmission link based on the supplementary enhancement information and the node delay information corresponding to each node server, and uses the video stream transmission link to observe video stream delay. The embodiments of the present application can provide a reasonable and efficient video stream delay observation scheme, realize accurate video stream delay observation and video stream node tracking, and improve the accuracy and efficiency of video stream delay observation.

[0066] The embodiments described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0067] Those skilled in the art will appreciate that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0068] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0069] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0070] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0071] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0072] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the above units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0073] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0074] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0075] It should be appreciated that embodiments of the present invention may be implemented or enforced by computer hardware, a combination of hardware and software, or by computer instructions stored in a non-transitory computer-readable memory. The method may be implemented in a computer program using standard programming techniques including a non-transitory computer-readable storage medium configured with a computer program, wherein the storage medium so configured causes the computer to operate in a specific and predefined manner - according to the methods and drawings described in the specific embodiments. Each program may be implemented in a high-level procedural or object-oriented programming language to communicate with a computer system. However, if desired, the program may be implemented in an assembly or machine language. In any case, the language may be a compiled or interpreted language. In addition, the program may be run on a programmed ASIC for this purpose.

[0076] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including multiple instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store programs.

[0077] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but the scope of the rights of the present invention is not limited thereto. Any modification, equivalent substitution and improvement made by a person skilled in the art without departing from the scope and essence of the present invention should be within the scope of the rights of the present invention.

Claims

1. A video stream delay observation method, characterized in that: The method comprises the following steps: Obtaining supplementary enhancement information for video sequence data; Push the video sequence data to each node server, obtain the supplementary enhancement parsing information after each node server parses the supplementary enhancement information, and count the node delay information corresponding to each node server according to the supplementary enhancement parsing information corresponding to each node server; A video stream transmission link is generated according to the supplementary enhancement information and the node delay information corresponding to each of the node servers, and the video stream delay observation is performed using the video stream transmission link.

2. The video stream delay observation method according to claim 1, characterized in that: The obtaining of supplementary enhancement information of the video sequence data specifically includes: Acquire the video sequence data generated by a video encoding end, wherein the video encoding end is a node server end that generates the video sequence data using a video encoding technology, and the video sequence data includes a plurality of video frames; Obtaining a preset supplementary enhancement information format; the supplementary enhancement information format at least includes a current node identifier, a current node network protocol address, a current video frame sequence number, and an absolute timestamp of a node parsing or generating a current video frame; Obtain the node identifier of the video encoding end and the current absolute timestamp of each video frame generated by the video encoding end, and generate the supplementary enhancement information corresponding to each video frame according to the supplementary enhancement information format, the node identifier of the video encoding end and the current absolute timestamp of each video frame generated by the video encoding end.

3. The video stream delay observation method according to claim 2, characterized in that: The step of streaming the video sequence data to each node server, obtaining supplementary enhancement parsing information after each node server parses the supplementary enhancement information, and counting node delay information corresponding to each node server according to the supplementary enhancement parsing information corresponding to each node server, specifically includes: Acquire the encoding generation time of each video frame in the video sequence data, and add the encoding generation time of each video frame to the supplementary enhancement information corresponding to each video frame; For each of the node servers, the current recording time of each of the video frames parsed by the node server is obtained, and the video frame parsing interval is determined according to the current recording time of the adjacent video frames, and the encoding generation time interval of the adjacent video frames is obtained according to the supplementary enhanced parsing information corresponding to each of the node servers, and the node delay interval corresponding to the node server is determined according to the video frame parsing interval and the encoding generation time interval; The node identifier corresponding to each of the node servers is obtained, and the node delay information corresponding to each of the node servers is generated according to the node identifier corresponding to each of the node servers and the node delay interval.

4. The video stream delay observation method according to claim 3, characterized in that: The method further comprises: Generate video frame supplementary enhancement information corresponding to each of the video frames in each of the node servers according to the supplementary enhancement information format and the node delay information corresponding to each of the node servers; For each of the video frames, the video frame supplementary enhancement information corresponding to the video frame in each of the node servers is added to the supplementary enhancement information corresponding to the video frame.

5. The video stream delay observation method according to claim 3, characterized in that: The obtaining, according to the supplementary enhanced parsing information corresponding to each of the node servers, the encoding generation time interval of adjacent video frames specifically includes: Obtaining a first absolute timestamp and a second absolute timestamp according to the supplementary enhanced parsing information obtained after the node server parses the adjacent video frames, wherein the first absolute timestamp is a current absolute timestamp when a preceding video frame among the adjacent video frames is generated in the video encoding end, and the second absolute timestamp is a current absolute timestamp when a succeeding video frame among the adjacent video frames is generated in the video encoding end; The encoding generation time interval is determined according to the first absolute timestamp and the second absolute timestamp.

6. The video stream delay observation method according to claim 4, characterized in that: The generating a video stream transmission link according to the supplementary enhancement information and the node delay information corresponding to each of the node servers, and using the video stream transmission link to perform video stream delay observation specifically includes: Determining a video stream transmission path corresponding to the video sequence data according to the supplemental enhancement information corresponding to each of the video frames; The video stream transmission link is generated according to the video stream transmission path and the node delay information corresponding to each of the node servers.

7. The video stream delay observation method according to claim 4, characterized in that: The generating, according to the supplementary enhancement information format and the node delay information corresponding to each of the node servers, the video frame supplementary enhancement information corresponding to each of the video frames in each of the node servers specifically includes: The node server to which the video sequence data is currently streaming is obtained as the current node server, and the absolute timestamp corresponding to each video frame at the previous node server is obtained according to the supplementary enhanced parsing information obtained after each video frame is parsed at the current node server, wherein the previous node server is the node server that pushes the video frame to the current node server; According to the node delay interval corresponding to the current node server and the absolute timestamp corresponding to the video frame at the previous node server, the current node absolute timestamp is determined, wherein the current node absolute timestamp is the absolute timestamp of the video frame in the current node server, and the current node absolute timestamp is expressed by the following formula: ； in, is the absolute timestamp of the current node, It is the absolute timestamp of the video frame on the previous node server. The node delay interval corresponding to the current node server; The video frame supplementary enhancement information of the video frame at the current node server is generated according to the current node absolute timestamp and the node delay information corresponding to the current node server.

8. A video stream delay observation device, characterized in that: The device comprises: The first module is used to obtain supplementary enhancement information of video sequence data; The second module is used to push the video sequence data to each node server, obtain the supplementary enhancement parsing information after each node server parses the supplementary enhancement information, and count the node delay information corresponding to each node server according to the supplementary enhancement parsing information corresponding to each node server; The third module is used to generate a video stream transmission link according to the supplementary enhancement information and the node delay information corresponding to each node server, and use the video stream transmission link to observe the video stream delay.

9. An electronic device, characterized in that: The electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the video stream delay observation method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the video stream delay observation method according to any one of claims 1 to 7 is implemented.

Images