Auxiliary-data inserter and program for smpte st2110

Abstract

To transmit another auxiliary-data RTP stream obtained by adding arbitrary auxiliary date to an original auxiliary-data RTP stream without any frame delay in a SMPTEST2110 standard for transmitting video, audio and auxiliary date in separate RTP streams by using an IP (Internet Protocol) network.SOLUTION: To predict a time stamp value and a packet transmission time of a future frame by using periodicity and continuity of a time stamp value of an RTP packet, and to transmit target auxiliary data.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to an inserter of auxiliary data according to the SMPTE ST2110 standard, and a program therefor. [Background technology]

[0002] The SMPTE ST2110 standard transmits video, audio, and auxiliary data in separate streams over an IP network. The video, audio, and auxiliary data (ancillary) are all separate RTP (Real-time Transport Protocol) streams, and each can be handled individually.

[0003] Among the above standards, the RTP timestamp value of ancillary data in the SMPTE ST2110-40 standard in Non-Patent Document 2 is the same as the RTP timestamp value of each frame (field in the case of interlaced video. Unless otherwise specified, hereafter, when we say frame, it refers to field in the case of interlaced video), and represents the time at the alignment point, which is the boundary between each frame. In other words, it is synonymous with the start time of the first scan line of each frame. If there are multiple ancillary data RTP packets in one frame, they all have the same timestamp value.

[0004] The SMPTE ST2110 standard specifies that the clock frequency for the RTP timestamp value of ancillary data is 90 kHz. Therefore, for a progressive frame of 30 / 1.001 fps (frames per second) (hereafter referred to as "29.97 fps"), the RTP timestamp value of the ancillary data increases in increments of 90,000 / (30 / 1.001) = 3,003 (0 → 3,003 → 6,006 → 9,009 → ...). For an interlaced frame of 29.97 fps, the timestamp value of the second field is not an integer, so the fraction is rounded down (0 → 1,501 → 3,003 → 4,504 → 6,006 → ...).

[0005] The timing for transmitting RTP packets of auxiliary data is not clearly defined, but the RFC8331 standard (Non-Patent Document 4), which is referenced by the SMPTE ST2110-40 standard, states that they should be transmitted as quickly as possible, and that as a guideline, they should be transmitted within 1 millisecond.

[0006] Ancillary data RTP packets may contain multiple pieces of ancillary data. At least one RTP packet must be sent for each frame, and if there is no ancillary data to send, an RTP packet containing only a header is sent. This is called keep-alive. The timing of sending the keep-alive packet is not specified. The last RTP packet in a frame has a flag called the marker bit set to 1. [Prior art documents] [Patent documents]

[0007] [Patent Document 1] Patent application 2024-73266 [Non-patent literature]

[0008] [Non-Patent Document 1] SMPTE ST2110-10 [Non-patent document 2] SMPTE ST2110-40 [Non-patent document 3] SMPTE ST2059-1 [Non-patent document 4] RFC 8331 Summary of the Invention [Problem to be solved by the invention]

[0009] If you want to insert any auxiliary data into an existing RTP stream of SMPTE ST2110 standard auxiliary data, there are no technologies or products that allow this over an IP network, so you have to convert it to SDI (Serial Digital Interface) first and then insert it in SDI format. This means that the benefits of IP cannot be utilized, and not only does it make the system more complex, it also has significant disadvantages such as increased delays, higher costs, more points of failure, and even loss of video and audio in the event of equipment failure.

[0010] Patent Document 1 discloses a technology for receiving an RTP stream conforming to the existing SMPTE ST2110 standard, reusing the RTP timestamp value, and transmitting another RTP stream conforming to the SMPTE ST2110 standard. However, while this technology is applicable to auxiliary data as long as it processes the received auxiliary data, when inserting optional auxiliary data, there are cases where the RTP packet for the frame has not yet been received at the timing when the auxiliary data to be inserted is desired to be sent, making it impossible to reuse the RTP timestamp value, making this technology difficult to apply.

[0011] The present invention has been made in view of the above-mentioned problems, and has as its object to enable insertion of any auxiliary data into an RTP stream of auxiliary data conforming to the SMPTE ST2110 standard without frame delay. [Means for solving the problem]

[0012] In order to solve the above problems, the ST2110 auxiliary data inserter of the present invention comprises: It has receivers and transmitters for RTP packets in the packet format of auxiliary data defined in the SMPTE ST2110-40 standard. The RTP timestamp value and alignment point time of the next frame or field are predicted using the property that the timestamp value of the auxiliary data RTP packet received by the receiving unit continuously increases at regular intervals; The RTP packet of any auxiliary data including the predicted RTP timestamp value is transmitted from the transmitter at a timing obtained by adding a predetermined offset value to the alignment point time, The auxiliary data received by the receiving unit is optionally processed and transmitted by the transmitting unit, or is optionally deleted.

[0013] Furthermore, the ST2110 ancillary data inserter of the present invention is characterized in that the predetermined offset value is estimated using the arrival time of the RTP packet that has arrived at the receiving unit, the time of the timestamp value stamped on the RTP packet, and the characteristics of the RTP packet.

[0014] Furthermore, a general-purpose computer may be configured to have the functions of the above-mentioned device. [Effects of the Invention]

[0015] According to the present invention, it is possible to insert and process any auxiliary data into an RTP stream of auxiliary data conforming to the SMPTE ST2110 standard without frame delay, and this can be achieved with a simple configuration. In addition, a single device can process multiple ST2110 auxiliary data RTP streams, which has significant cost benefits.

[0016] Furthermore, since there is absolutely no interference with the video RTP stream or audio RTP stream, even if the device breaks down, there is no impact on the quality of the video or audio. [Brief explanation of the drawings]

[0017] [Figure 1] FIG. 2 is a block diagram showing the functional configuration of an ST2110 auxiliary data inserter according to an embodiment of the present invention. [Figure 2] 10 is a flowchart illustrating processing by an auxiliary data generator according to an embodiment of the present invention. [Figure 3] 10 is a flowchart showing the processing of a reception auxiliary data processing unit according to an embodiment of the present invention. [Figure 4] 10 is a flowchart showing the processing of an RTP packet sending unit according to one embodiment of the present invention. [Figure 5] 10 is a flowchart showing the process of an offset estimation unit according to an embodiment of the present invention. [Figure 6] 10 is an example of a timing chart showing an offset value estimation performed by the offset estimation unit based on a VANC packet. [Figure 7] 10 is an example of a timing chart showing how the offset estimation unit estimates an offset value based on a keep-alive packet. DETAILED DESCRIPTION OF THE INVENTION

[0018] Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. Note that the same elements are given the same reference numerals, and duplicate explanations will be omitted. Furthermore, the following embodiments are merely examples for explaining the present invention, and are not intended to limit the present invention to only these embodiments. Furthermore, the present invention can be modified in various ways without departing from the gist of the invention.

[0019] FIG. 1 is a block diagram showing the functional configuration of an ST2110 auxiliary data inserter 10 to which one embodiment of the present invention can be applied. ST2110 ancillary data inserter 10 includes an RTP packet receiver 11, a timestamp derivation unit 12, an alignment point calculator 13, ancillary data generator 14, a received ancillary data processor 15, an RTP packet transmitter 16, an offset estimation unit 17, and a time synchronizer 100. The solid arrows within the frame in the diagram of ST2110 ancillary data inserter 10 indicate the flow of processing, and the dotted arrows indicate the notification of information. The time synchronization unit 100 may perform synchronization by a general method using the Precision Time Protocol (PTP) assumed in the SMPTE ST2110 standard, or may use the derived PTP time shown in Patent Document 1. Each device according to the embodiment of the present invention may be configured as a dedicated device, or a general-purpose computer may be configured to have the functions of the device according to the embodiment of the present invention.

[0020] The RTP packet receiving unit 11 receives RTP packets in the packet format of ancillary data defined by the SMPTE ST2110 standard, and outputs them to the timestamp derivation unit 12 , the received ancillary data processing unit 15 , and the offset estimation unit 17 .

[0021] The timestamp derivation unit 12 derives the timestamp value of the frame next to the current frame. The timestamp value Tcurrent of the delivered RTP packet is extracted, and if it has changed from the timestamp value of the previously received RTP packet, the difference d is calculated. In order to calculate the correct difference even when the timestamp value wraps around, the calculation of d is handled as int32. The array D that records the history of timestamp values ​​is shifted (D(n)←D(n-1)), and the timestamp value Tcurrent is substituted into D(0).

[0022] While d is essentially the same as 90,000 / frame rate, depending on the frame rate, it cannot be divided evenly to form an integer, which can cause discrepancies in the rounding down and rounding up of decimal points depending on the timing. Therefore, Table 1 shows the number of timestamp increment patterns m and the timestamp fixed interval increment R that make R = 90,000 / frame rate × m an integer, based on the value of d for each frame rate used in SDI signals specified in the SMPTE ST352 and SMPTE ST2082-10 standards.

[0023] [Table 1]

[0024] Based on d from Table 1, m and R are found, and the timestamp value Tnext of the next frame is derived using the following formula. Note that if d is 938, a determination cannot be made, so the determination is made using the values ​​937 or 939. If d is 750, the determination is made based on whether 750 occurs four times in a row.

[0025] [Number 1] Tnext = D(m-1) + R

[0026] The above formula takes advantage of the property that the increase in timestamp value over m frames is exactly R, and by adding R to the timestamp value m frames before the next frame, the timestamp value of the next frame is derived without being affected by decimal truncation.

[0027] The Tnext derived above is output to the alignment point calculation unit 13.

[0028] The alignment point calculation unit 13 converts the input timestamp value into time. Since the RTP timestamp value is 32 bits and wraps around at regular intervals, it is impossible to determine the exact time based on this value alone. To solve this problem, the SMPTE ST2110 standard uses a common clock, PTP, to synchronize each RTP stream.

[0029] As an example, if the current time obtained from the time synchronization unit 100 is tnow, the time t(T) represented by the timestamp value T can be calculated using the following function, provided that the time represented by the timestamp value T is closer in the past than tnow. Note that "close in the past" means within the wrap-around period (approximately 13.2 hours) of the timestamp value. In this specification, time represents the number of seconds elapsed since SMPTE_EPOCH defined in SMPTE ST2059-1 of Non-Patent Document 3. In the formula, int(A) represents the integer value of A truncated to the nearest integer, and A mod B represents the remainder when A is divided by B. [Number 2] t(T) = ( int(tnow × 90,000) - ((int(tnow × 90,000) mod 2^32 - T) mod 2^32) ) / 90,000

[0030] The t calculated above is based on the 90 kHz RTP clock, so the time resolution is only about 11 microseconds. However, as mentioned above, the timing of sending auxiliary data packets is not clearly defined, so this level of error can be absorbed.

[0031] Using the above-mentioned Equation 2, the alignment point time tnext of the next frame is calculated from the timestamp value Tnext, and Tnext and tnext are output to the auxiliary data generating unit 14.

[0032] Auxiliary data generator 14 waits until the packet transmission time based on values ​​obtained from alignment point calculator 13 and, if necessary, offset estimator 17, and generates an auxiliary data list, which it outputs to RTP packet transmitter 16. If there is a plurality of auxiliary data to be generated, multiple auxiliary data generators 14 may operate simultaneously.

[0033] 2 is a flowchart showing the processing of the auxiliary data generator 14. Specific processing steps will be described.

[0034] In step S1401, a value obtained by adding a predetermined offset value to the alignment point time tnext of the next frame input from the alignment point calculation unit 13 is set as the transmission time of the auxiliary data to be generated, and the process proceeds to step S1402.

[0035] The offset value is the sum of two values: a value (hereinafter referred to as latency) that adjusts the latency between the ST2110 auxiliary data inserter 10 and the transmitter of the RTP packet that the ST2110 auxiliary data inserter 10 is receiving, and a value (hereinafter referred to as transmission adjustment time) that adjusts how much time has passed since the alignment point time of each frame before auxiliary data is transmitted.

[0036] The latency value can be assumed to be 0 in an environment consisting of only a LAN and not via a WAN (Wide Area Network).

[0037] The transmission adjustment time may be preset to a fixed value not exceeding 1 millisecond (for example, 200 microseconds. In this case, the offset value will be 200 microseconds in total). The transmission adjustment time can be set to 0, but the timing of transmitting the additional auxiliary data will be the same as the frame alignment point time, which may cause the receiver to miss reception and malfunction, so this is a short waiting time.

[0038] The offset value may be a value estimated by the offset estimation unit 17. Details of the offset estimation unit 17 will be described later.

[0039] In step S1402, the current time is acquired from the time synchronization unit 100, and the process proceeds to step S1403.

[0040] In step S1403, the acquired current time is compared with the auxiliary data transmission time, and if the current time is after the auxiliary data transmission time (Yes), the process proceeds to step S1404, otherwise (No), the process returns to step S1402.

[0041] In step S1404, zero or more pieces of arbitrary auxiliary data are generated, and an auxiliary data list that stores the data is generated, after which the process proceeds to step S1405.

[0042] In step S1405, the auxiliary data list generated in step S1404 and the value of Tnext input from alignment point calculation unit 13 as the transmission timestamp value are output to RTP packet transmission unit 16, thereby completing the processing of auxiliary data generation unit 14. At this time, even if the auxiliary data list is empty, an empty list is output to RTP packet transmission unit 16.

[0043] The received auxiliary data processor 15 is responsible for retransmitting the received auxiliary data as separate auxiliary data. At this time, the received auxiliary data may be retransmitted as is, or may be optionally processed or deleted. 3 is a flowchart showing the processing of the reception auxiliary data processing unit 15. Specific processing steps will be described.

[0044] In step S1501, the auxiliary data list storing the auxiliary data to be transmitted is cleared, and the process proceeds to step S1502.

[0045] In step S1502, a loop is performed the number of times equal to the number of auxiliary data included in the received RTP packets input from the RTP packet receiving unit 11, the auxiliary data is extracted in order, and the process proceeds to step S1503. If there is no auxiliary data in the received RTP packets, the loop is skipped and the process proceeds to step S1505.

[0046] Step S1503 optionally performs processing on the auxiliary data. If the auxiliary data is to be retained, it is added to the auxiliary data list as is. Alternatively, the auxiliary data may be subjected to any processing before being added to the auxiliary data list. Alternatively, one or more other auxiliary data may be generated based on the content of the auxiliary data and added to the auxiliary data list. Alternatively, the auxiliary data is arbitrarily deleted by not adding it to the auxiliary data list. Furthermore, any combination of the above processes may be performed. Then, the process proceeds to step S1504.

[0047] If the loop termination condition is met in step S1504, the process proceeds to step S1505. If not, the process returns to step S1502 and proceeds to the next loop.

[0048] In step S1505, the marker bit flag included in the header of the received RTP packet is checked, and if it is 1, the process proceeds to step S1506. If it is off, the process proceeds to step S1507.

[0049] In step S1506, the EndRecv flag is set to 1, which indicates that no new RTP packets will arrive for the current frame, and the process proceeds to step S1507.

[0050] In step S1507, the created auxiliary data list and the timestamp value of the received RTP packet as the transmission timestamp value are output to RTP packet transmitting unit 16, and then received auxiliary data processing unit 15 is terminated. At this time, even if the auxiliary data list is empty, the empty list is output to RTP packet transmitting unit 16.

[0051] The RTP packet transmitter 16 converts the auxiliary data list and transmission timestamp value input from the auxiliary data generator 14 or the received auxiliary data processor 15 into RTP packets and transmits them over the network. Figure 4 is a flowchart showing the processing of the RTP packet transmitter 16. Specific processing steps will be explained below. Note that the RTP packet transmitter 16 performs appropriate exclusive control to prevent multiple RTP packet transmitters 16 from operating simultaneously.

[0052] In step S1601, the value of the EndRecv flag is checked, and if it is 1, the process proceeds to step S1602, and if it is 0, the process proceeds to step S1604.

[0053] In step S1602, it is determined whether the RTP packet to be transmitted is the last packet in the frame being processed. If the auxiliary data list has already been input from the auxiliary data generator 14 (or from all auxiliary data generators 14 if multiple auxiliary data generators 14 are operating), it is determined that this is the last packet (Yes), and the process proceeds to step S1603. If not (No), the process proceeds to step S1604.

[0054] In step S1603, the marker bit of the RTP packet to be generated is set to 1. Also, in preparation for processing the next frame, the EndRecv flag is reset to 0. Then, the process proceeds to step S1604.

[0055] In step S1604, the process checks whether the auxiliary data list to be transmitted is empty and the marker bit of the RTP packet to be generated, and if the auxiliary data list is empty and the marker bit is 0 (Yes), it is determined that there is no information to send, and terminates the processing of RTP packet transmitter 16. If not (No), the process proceeds to step S1605.

[0056] In step S1605, binary data of an RTP packet is assembled based on the input auxiliary data list and transmission timestamp value, and the process proceeds to step S1606. Note that information required for the RTP packet, such as the sequence number and SSRC, is written appropriately in accordance with the standard.

[0057] In step S1606, the binary data assembled in step S1605 is transmitted to the network, and the processing of the RTP packet transmitting unit 16 is completed.

[0058] As described above, the ST2110 ancillary data inserter 10 can perform any addition, processing, or deletion on the received ancillary data and then transmit another ancillary data RTP stream.

[0059] A receiving device that receives the auxiliary data stream transmitted by ST2110 auxiliary data inserter 10 receives the original RTP stream for video and audio, and receives the RTP stream output by ST2110 auxiliary data inserter 10 for auxiliary data, thereby being able to use the inserted auxiliary data.

[0060] The offset estimation unit 17 estimates an offset value including reception latency and transmission adjustment time based on the characteristics of the RTP packets received by the RTP packet receiving unit 11, and makes it available to the auxiliary data generating unit .

[0061] The type of RTP packet used to estimate the offset value is preferably auxiliary data of the line numbers of the VANC section (vertical blanking section) from the first line of a frame to the line where the video begins, such as ATC (Ancillary Timecode) or VPID (Video Payload ID), which are commonly used as auxiliary data. Here, the range of the VANC section does not need to be clear, and it can be line numbers 1 to 20. If auxiliary data for the VANC section is not available, keep-alive packets, which are sent when no auxiliary data is available, can be used.

[0062] 5 is a flowchart showing the processing of the offset estimation unit 17. Specific processing steps will be described.

[0063] In step S1701, the contents of the RTP packet input from the RTP packet receiving unit 11 are checked to determine whether the packet is to be used for estimating an offset value. If it is a target packet (Yes), the process proceeds to step S1702. If not (No), the process of the offset estimating unit 17 ends.

[0064] In step S1702, the current time is obtained from the time synchronization unit 100 and is set as the reception time tr of the RTP packet. Furthermore, the frame alignment point time t0 is calculated from the timestamp value of the packet using Equation 2, and the process proceeds to step S1703.

[0065] In step S1703, if the RTP packet is a packet in the VANC section, it can be considered to be an RTP packet transmitted within 1 millisecond from the alignment point, and therefore the offset value is estimated using the following formula. [Number 3] Offset value = tr - t0

[0066] Figure 6 shows an example of the above calculation in the form of a timing chart. When an RTP packet in the VANC section is sent from a remote original transmitter over a low-jitter line such as a dark fiber WAN line, latency occurs before it is received by the ST2110 ancillary data inserter 10. The packet reception time tr is the timing at which the ST2110 ancillary data inserter 10 wishes to send ancillary data for frame 0 in Figure 6, and the time at which it wishes to send the packet for the next frame 1 can be found by adding an offset value to the predicted time t1.

[0067] If the RTP packet is a keep-alive packet, the timing of sending the keep-alive packet is left to the implementation of the transmitter, but assuming that the sending timing is constant for each model and device, it is measured and set in advance. For example, for a device that sends a keep-alive packet 10 milliseconds after the frame alignment point, the keep-alive transmission time dp = 10 milliseconds is set. Based on the above, the latency can be estimated using the following formula: [Number 4] Latency = tr - t0 - dp

[0068] If the latency can be estimated, the offset value can be estimated as a value obtained by adding a transmission adjustment time (for example, 200 microseconds).

[0069] Figure 7 shows an example of the above calculation in the form of a timing chart. There is a latency between the keep-alive RTP packet sent from the transmitter and its reception by the ST2110 ancillary data inserter 10. The latency can be calculated by subtracting the keep-alive transmission time dp of the original transmitter from the value tr - t0. This is added to the transmission adjustment time to obtain the offset value, and the time at which the ST2110 ancillary data inserter 10 wishes to transmit the packet for frame 1 can be calculated by adding the offset value to the predicted time t1.

[0070] From the above, the offset estimation unit 17 can estimate the offset value. In the above example, the offset value is estimated based on the characteristics of the auxiliary data RTP packets in the VANC section and the keep-alive RTP packets, but if other types of auxiliary data with distinguishable characteristics are available, they may also be used.

[0071] Although one embodiment of the present invention has been described above, the present invention is not limited to the above embodiment and can be embodied in various other forms without departing from the spirit of the present invention. Therefore, the above embodiment is merely an example in all respects and should not be interpreted as being restrictive. For example, the offset estimation unit 17 may be used even in an environment consisting of only a LAN without going through a WAN. [Explanation of symbols]

[0072] 10 ST2110 Auxiliary Data Inserter 11 RTP packet receiver 12 Timestamp derivation part 13 Alignment point calculation section 14 Auxiliary Data Generation Unit 15 Reception auxiliary data processing section 16 RTP packet transmitter 17 Offset Estimation Unit 100 Time synchronization unit d The difference between the timestamp value of the received RTP packet and the current timestamp value dp The time from the alignment point when the original ancillary data transmitter sends a keep-alive packet D( ) An array that records the history of timestamp values. D(n) indicates the nth previous timestamp value. m Number of timestamp increment patterns R timestamp increment at regular intervals t0 Alignment point time of the currently processed frame stamped in the RTP packet t1 Alignment point time of the frame after t0 Tcurrent: The timestamp value of the received RTP packet. Tnext: The derived timestamp value of the next frame tnext: The derived alignment point time of the next frame tnow Current time in PTP seconds tr RTP packet reception time t(T) A function that calculates the time t for a timestamp value T.

Claims

1. The present invention includes a receiver and a transmitter for RTP packets in a packet format of auxiliary data defined in the SMPTE ST2110-40 standard, predicting the RTP timestamp value and alignment point time of the next frame or field using the property that the timestamp value of the auxiliary data RTP packet received by the receiving unit continuously increases at regular intervals; and transmitting an RTP packet of any auxiliary data including the predicted RTP timestamp value from the transmitter at a timing obtained by adding a predetermined offset value to the alignment point time, The auxiliary data received by the receiving unit is optionally processed and transmitted by the transmitting unit, or is optionally deleted. ST2110 Ancillary Data Inserter.

2. The ST2110 auxiliary data inserter according to claim 1, characterized in that the predetermined offset value is estimated using the arrival time of the RTP packet that has arrived at the receiving unit, the time of the timestamp value stamped on the RTP packet, and characteristics of the RTP packet.

3. A program that causes a computer to function as the ST2110 auxiliary data inserter according to claim 1 or 2.

Images