Auxiliary data inserter and program related to smpte st2110

Abstract

[Problem] To make it possible, in the SMPTE ST2110 standard for transmitting videos, audio, and auxiliary data in separate RTP streams using an IP network, to transmit an auxiliary data RTP stream which is obtained by adding arbitrary auxiliary data to an original auxiliary data RTP stream without frame delay. [Solution] The timestamp value and the packet transmission time of a future frame are predicted by using the periodicity and continuity of timestamp values of RTP packets, and target auxiliary data is transmitted.

Description

Ancillary data inserter and program according to SMPTE ST2110

[0001] The present invention relates to an inserter and program for auxiliary data according to the SMPTE ST2110 standard.

[0002] The SMPTE ST2110 standard, which transmits video, audio, and auxiliary data in separate streams over an IP network, defines video, audio, and auxiliary data (ancillary) as separate Real-time Transport Protocol (RTP) streams, each of which can be handled separately.

[0003] Among the above standards, the RTP timestamp value of ancillary data in the SMPTE ST2110-40 standard in Non-Patent Document 2 is the same as the RTP timestamp value of each frame (field in the case of interlaced video; hereinafter, unless otherwise specified, when a frame is written, it refers to a field in the case of interlaced video), and represents the time at the alignment point, which is the boundary between each frame. In other words, it is synonymous with the start time of the first scan line of each frame. If there are multiple ancillary data RTP packets in one frame, they all have the same timestamp value.

[0004] The SMPTE ST2110 standard stipulates that the clock frequency of the RTP timestamp value of ancillary data is 90 kHz. Therefore, in the case of progressive 30 / 1.001 fps (frames per second) (hereinafter referred to as "29.97 fps"), the RTP timestamp value of ancillary data increases in increments of 90,000 / (30 / 1.001) = 3,003 (0 → 3,003 → 6,006 → 9,009 → ...). In the case of interlaced 29.97 fps, the timestamp value of the second field is not an integer, so the fraction is rounded down (0 → 1,501 → 3,003 → 4,504 → 6,006 → ...).

[0005] The timing of transmitting RTP packets of auxiliary data is not clearly defined, but the RFC8331 standard (Non-Patent Document 4) that the SMPTE ST2110-40 standard refers to states that the packets should be transmitted as quickly as possible, and that the maximum transmission time is within 1 millisecond as a guideline.

[0006] Ancillary data RTP packets may contain multiple pieces of auxiliary data. At least one RTP packet must be sent for each frame, and if there is no auxiliary data to send, an RTP packet containing only a header is sent. This is called keep-alive. The timing of sending the keep-alive packet is not specified. The last RTP packet in a frame has a flag called a marker bit set to 1.

[0007] Patent application 2024-73266

[0008] SMPTE ST2110-10SMPTE ST2110-40SMPTE ST2059-1RFC 8331

[0009] If you want to insert any auxiliary data into an existing RTP stream of SMPTE ST2110 standard auxiliary data, there are no technologies or products that allow insertion over an IP network, so you have to convert it to SDI (Serial Digital Interface) first and then insert it in SDI format. This means that the benefits of IP cannot be utilized, the system becomes more complex, and there are significant disadvantages such as increased delays, increased costs, more failure points, and even interruptions to video and audio in the event of equipment failure.

[0010] Patent Document 1 discloses a technique for receiving an RTP stream conforming to the existing SMPTE ST2110 standard, reusing the RTP timestamp value, and transmitting another RTP stream conforming to the SMPTE ST2110 standard. With regard to auxiliary data, this technique is applicable if it is for processing received auxiliary data, but when inserting optional auxiliary data, there may be cases where the RTP packet for that frame has not yet been received at the timing when the auxiliary data to be inserted is desired to be sent, making it impossible to reuse the RTP timestamp value, making this technique difficult to apply.

[0011] The present invention has been made in view of the above-mentioned problems, and has as its object to enable insertion of any auxiliary data into an RTP stream of auxiliary data conforming to the SMPTE ST2110 standard without frame delay.

[0012] In order to solve the above problems, the ST2110 ancillary data inserter of the present invention comprises a receiver and transmitter for RTP packets in the packet format of ancillary data defined in the SMPTE ST2110-40 standard, and predicts the RTP timestamp value and alignment point time of the next frame or field using the property that the timestamp value of the ancillary data RTP packet received by the receiver increases continuously at regular intervals, and transmits an RTP packet of any ancillary data containing the predicted RTP timestamp value from the transmitter at a timing when a predetermined offset value is added to the alignment point time, and the ancillary data received by the receiver is optionally processed and transmitted by the transmitter, or optionally deleted.

[0013] Furthermore, the ST2110 auxiliary data inserter of the present invention is characterized in that the predetermined offset value is estimated using the arrival time of the RTP packet that has arrived at the receiving unit, the time of the timestamp value stamped on the RTP packet, and the characteristics of the RTP packet.

[0014] Furthermore, a general-purpose computer may be configured to have the functions of the above-mentioned device.

[0015] According to the present invention, it is possible to insert or process any auxiliary data into an RTP stream of auxiliary data conforming to the SMPTE ST2110 standard without frame delay, and this can be realized with a simple configuration. In addition, a single device can process multiple ST2110 auxiliary data RTP streams, which also offers significant cost benefits.

[0016] Furthermore, since no interference is made with the video RTP stream or the audio RTP stream, even if the device breaks down, the quality of the video and audio is not affected.

[0017] Fig. 1 is a block diagram showing the functional configuration of an ST2110 ancillary data inserter according to one embodiment of the present invention. Fig. 2 is a flowchart showing the processing of an ancillary data generation unit according to one embodiment of the present invention. Fig. 3 is a flowchart showing the processing of a received ancillary data processing unit according to one embodiment of the present invention. Fig. 4 is a flowchart showing the processing of an RTP packet transmission unit according to one embodiment of the present invention. Fig. 5 is a flowchart showing the processing of an offset estimation unit according to one embodiment of the present invention. Fig. 6 is an example timing chart showing the offset estimation unit estimating an offset value based on a VANC packet. Fig. 7 is an example timing chart showing the offset estimation unit estimating an offset value based on a keep-alive packet.

[0018] Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. Note that the same elements are given the same reference numerals, and duplicate explanations will be omitted. Furthermore, the following embodiments are merely examples for explaining the present invention, and are not intended to limit the present invention to only these embodiments. Furthermore, the present invention can be modified in various ways without departing from the gist of the invention.

[0019] FIG. 1 is a block diagram showing the functional configuration of an ST2110 ancillary data inserter 10 to which an embodiment of the present invention can be applied. The ST2110 ancillary data inserter 10 includes an RTP packet receiver 11, a timestamp derivation unit 12, an alignment point calculator 13, ancillary data generator 14, a received ancillary data processor 15, an RTP packet transmitter 16, an offset estimation unit 17, and a time synchronization unit 100. The solid arrows within the frame of the ST2110 ancillary data inserter 10 indicate the flow of processing, while the dotted arrows indicate information notification. The time synchronization unit 100 may synchronize using a general method using the Precision Time Protocol (PTP) assumed in the SMPTE ST2110 standard, or may use the derived PTP time described in Patent Document 1. Each device according to an embodiment of the present invention may be configured as a dedicated device, or a general-purpose computer may be configured to have the functions of the device according to an embodiment of the present invention.

[0020] The RTP packet receiving unit 11 receives RTP packets in the packet format of ancillary data defined by the SMPTE ST2110 standard, and outputs them to the timestamp deriving unit 12 , the received ancillary data processing unit 15 , and the offset estimating unit 17 .

[0021] The timestamp derivation unit 12 derives the timestamp value of the frame following the current frame. It extracts the timestamp value Tcurrent of the delivered RTP packet, and if it has changed from the timestamp value of the previously received RTP packet, calculates the difference d. In order to calculate the correct difference even when the timestamp value wraps around, d is calculated as an int32. The array D, which records the history of timestamp values, is shifted (D(n)←D(n-1)), and the timestamp value Tcurrent is assigned to D(0).

[0022] Although d is essentially the same as 90,000 / frame rate, depending on the frame rate, it cannot be divided evenly and does not become an integer, so there is a discrepancy between rounding down and rounding up depending on the timing. Therefore, Table 1 shows the number of timestamp increment patterns m and the timestamp fixed interval increment R, which make R = 90,000 / frame rate × m an integer, based on the value of d for each frame rate used in SDI signals specified in the SMPTE ST352 standard and the SMPTE ST2082-10 standard.

[0023]

[0024] Based on d from Table 1, m and R are found, and the timestamp value Tnext of the next frame is derived using the following formula. Note that if d is 938, a determination cannot be made, so the determination is made using the value 937 or 939. If d is 750, the determination is made based on whether 750 occurs four times in a row.

[0025] [Equation 1] Tnext = D(m-1) + R

[0026] The above formula takes advantage of the property that the increase in the timestamp value over m frames is exactly R, and by adding R to the timestamp value m frames before the next frame, the timestamp value of the next frame is derived without being affected by truncation.

[0027] The Tnext derived above is output to the alignment point calculation unit 13.

[0028] The alignment point calculation unit 13 converts the input timestamp value into time. Because the RTP timestamp value is 32 bits and wraps around at regular intervals, it is impossible to determine the exact time based on this value alone. To solve this problem, the SMPTE ST2110 standard uses a common clock, PTP, to synchronize each RTP stream.

[0029] As an example, if the current time obtained from the time synchronization unit 100 is tnow, the time t(T) represented by the timestamp value T can be calculated by the following function, provided that the time represented by the timestamp value T is closer in the past than tnow. Note that "close in the past" means within the period (approximately 13.2 hours) in which the timestamp value wraps around. Furthermore, in this specification, time represents the number of seconds elapsed since SMPTE_EPOCH defined in SMPTE ST2059-1 of Non-Patent Document 3. In the formula, int(A) is the integer value of A truncated to the nearest integer, and A mod B represents the remainder when A is divided by B. [Equation 2] t(T) = (int(tnow x 90,000) - ((int(tnow x 90,000) mod 2^32 - T) mod 2^32)) / 90,000

[0030] The t calculated above is based on a 90 kHz RTP clock, so the time resolution is only about 11 microseconds. However, as mentioned above, the timing of packet transmission of auxiliary data is not clearly defined, so this level of error can be absorbed.

[0031] Using the above-mentioned equation 2, the alignment point time tnext of the next frame is calculated from the timestamp value Tnext, and Tnext and tnext are output to the auxiliary data generating unit 14 .

[0032] The auxiliary data generator 14 waits until the packet transmission time based on the values ​​obtained from the alignment point calculator 13 and, if necessary, the offset estimator 17, generates an auxiliary data list, and outputs it to the RTP packet transmitter 16. If there is a plurality of auxiliary data to be generated, the plurality of auxiliary data generators 14 may operate simultaneously.

[0033] 2 is a flowchart showing the processing of the auxiliary data generating unit 14. Specific processing steps will be described.

[0034] In step S1401, a value obtained by adding a predetermined offset value to the alignment point time tnext of the next frame input from the alignment point calculation unit 13 is set as the transmission time of the generated auxiliary data, and the process proceeds to step S1402.

[0035] The offset value is the sum of two values: a value (hereinafter referred to as latency) that adjusts the latency between the ST2110 auxiliary data inserter 10 and the transmitter of the RTP packet that the ST2110 auxiliary data inserter 10 is receiving, and a value (hereinafter referred to as transmission adjustment time) that adjusts how much time has passed since the alignment point time of each frame before auxiliary data is transmitted.

[0036] The latency value may be assumed to be 0 in an environment that does not go through a WAN (Wide Area Network) and is configured only with a LAN.

[0037] The transmission adjustment time may be preset to a fixed value not exceeding 1 millisecond (for example, 200 microseconds. In this case, the offset value will be a total of 200 microseconds). The transmission adjustment time may be set to 0, but this is a time to wait a little while, since the timing of transmitting the additional auxiliary data will be the same as the frame alignment point time, which may cause the receiver to miss reception and malfunction.

[0038] The offset value may be a value estimated by the offset estimation unit 17. Details of the offset estimation unit 17 will be described later.

[0039] In step S1402, the current time is acquired from the time synchronization unit 100, and the process proceeds to step S1403.

[0040] In step S1403, the acquired current time is compared with the auxiliary data transmission time, and if the current time is after the auxiliary data transmission time (Yes), the process proceeds to step S1404. If not (No), the process returns to step S1402.

[0041] In step S1404, zero or more pieces of any auxiliary data are generated, and an auxiliary data list is generated that stores the generated auxiliary data. Then, the process proceeds to step S1405.

[0042] In step S1405, the auxiliary data list generated in step S1404 and the value of Tnext input from alignment point calculation unit 13 as the transmission timestamp value are output to RTP packet transmission unit 16, thereby completing the processing of auxiliary data generation unit 14. At this time, even if the auxiliary data list is empty, an empty list is output to RTP packet transmission unit 16.

[0043] The received auxiliary data processing unit 15 is responsible for retransmitting the received auxiliary data as separate auxiliary data. In this case, the received auxiliary data may be retransmitted as is, or may be optionally processed or deleted. Figure 3 is a flowchart showing the processing of the received auxiliary data processing unit 15. Specific processing steps will now be described.

[0044] In step S1501, the auxiliary data list storing the auxiliary data to be transmitted is cleared, and the process proceeds to step S1502.

[0045] In step S1502, a loop is executed the number of times equal to the number of auxiliary data included in the received RTP packets input from the RTP packet receiving unit 11, the auxiliary data is extracted in order, and the process proceeds to step S1503. If there is no auxiliary data in the received RTP packets, the loop is skipped and the process proceeds to step S1505.

[0046] In step S1503, any processing is performed on the auxiliary data. If the auxiliary data is to be retained, it is added to the auxiliary data list as is. Alternatively, any processing is performed on the auxiliary data before adding it to the auxiliary data list. Alternatively, one or more other auxiliary data are generated based on the contents of the auxiliary data and added to the auxiliary data list. Alternatively, the auxiliary data is deleted by not being added to the auxiliary data list. Alternatively, any combination of the above processing may be performed. Then, the process proceeds to step S1504.

[0047] If the loop termination condition is met in step S1504, the process proceeds to step S1505. If not, the process returns to step S1502 and proceeds to the next loop.

[0048] In step S1505, the marker bit flag included in the header of the received RTP packet is checked, and if it is 1, the process proceeds to step S1506, and if it is off, the process proceeds to step S1507.

[0049] In step S1506, the EndRecv flag is set to 1, which indicates that no new RTP packets for the current frame will arrive, and the process proceeds to step S1507.

[0050] In step S1507, the created auxiliary data list and the timestamp value of the received RTP packet as the transmission timestamp value are output to the RTP packet transmitting unit 16, and the received auxiliary data processing unit 15 is terminated. At this time, even if the auxiliary data list is empty, the empty list is output to the RTP packet transmitting unit 16.

[0051] The RTP packet transmitter 16 converts the auxiliary data list and transmission timestamp values ​​input from the auxiliary data generator 14 or the received auxiliary data processor 15 into RTP packets and transmits them over the network. Figure 4 is a flowchart showing the processing of the RTP packet transmitter 16. Specific processing steps will be described below. Note that the RTP packet transmitter 16 performs appropriate exclusive control to prevent multiple RTP packet transmitters 16 from operating simultaneously.

[0052] In step S1601, the value of the EndRecv flag is checked, and if it is 1, the process proceeds to step S1602, and if it is 0, the process proceeds to step S1604.

[0053] In step S1602, it is determined whether the RTP packet to be transmitted is the last packet in the frame being processed. If an auxiliary data list has already been input from the auxiliary data generator 14 (or from all auxiliary data generators 14 if multiple auxiliary data generators 14 are operating), it is determined that this is the last packet (Yes), and the process proceeds to step S1603. If not (No), the process proceeds to step S1604.

[0054] In step S1603, the marker bit of the RTP packet to be generated is set to 1. Also, in preparation for processing the next frame, the EndRecv flag is reset to 0. After that, the process proceeds to step S1604.

[0055] In step S1604, the process checks whether the auxiliary data list to be transmitted is empty and the marker bit of the RTP packet to be generated, and if the auxiliary data list is empty and the marker bit is 0 (Yes), it is determined that there is no information to be transmitted, and the process of the RTP packet transmission unit 16 is terminated. If not (No), the process proceeds to step S1605.

[0056] In step S1605, binary data of an RTP packet is assembled based on the input auxiliary data list and transmission timestamp value, and the process proceeds to step S1606. Note that information required for the RTP packet, such as sequence number and SSRC, is written appropriately in accordance with the standard.

[0057] In step S1606, the binary data assembled in step S1605 is transmitted to the network, and the process of the RTP packet transmitting unit 16 ends.

[0058] As described above, ST2110 auxiliary data inserter 10 can perform any addition, processing, or deletion on the received auxiliary data and then transmit another auxiliary data RTP stream.

[0059] A receiving device that receives the auxiliary data stream transmitted by ST2110 auxiliary data inserter 10 receives the original RTP stream for video and audio, and receives the RTP stream output by ST2110 auxiliary data inserter 10 for auxiliary data, thereby being able to use the inserted auxiliary data.

[0060] The offset estimation unit 17 estimates an offset value including reception latency and transmission adjustment time based on the characteristics of the RTP packets received by the RTP packet receiving unit 11, and makes the offset value available to the auxiliary data generating unit .

[0061] The type of RTP packet used to estimate the offset value is preferably auxiliary data of the line numbers of the VANC section (vertical blanking section) from the first line of a frame to the line where the video starts, such as ATC (Ancillary Timecode) or VPID (Video Payload ID), which are often used as auxiliary data. Here, the range of the VANC section does not need to be clear, and it may be line numbers 1 to 20. If auxiliary data for the VANC section is not available, keep-alive packets, which are transmitted when no auxiliary data is available, may be used.

[0062] 5 is a flowchart showing the processing of the offset estimation unit 17. Specific processing steps will be described.

[0063] In step S1701, the contents of the RTP packet input from the RTP packet receiving unit 11 are checked to determine whether the packet is to be used for offset value estimation. If the packet is a target packet (Yes), the process proceeds to step S1702. If not (No), the process of the offset estimating unit 17 ends.

[0064] In step S1702, the current time is obtained from the time synchronization unit 100 and is set as the reception time tr of the RTP packet. Furthermore, the frame alignment point time t0 is calculated from the timestamp value of the packet using Equation 2, and the process proceeds to step S1703.

[0065] In step S1703, if the RTP packet is a packet in the VANC section, it can be considered to be an RTP packet transmitted within 1 millisecond from the alignment point, and therefore the offset value is estimated using the following equation: [Equation 3] Offset value = tr - t0

[0066] An example of the above calculation is shown in a timing chart in Figure 6. Latency occurs when an RTP packet in the VANC section is transmitted from a remote original transmitter over a low-jitter line such as a dark fiber WAN line until it is received by ST2110 ancillary data inserter 10. This packet reception time tr is the timing at which ST2110 ancillary data inserter 10 wishes to transmit ancillary data for frame 0 in Figure 6, and the time at which ST2110 wishes to transmit the packet for the next frame 1 can be found by adding an offset value to the predicted time t1.

[0067] If the RTP packet is a keep-alive packet, the timing at which the keep-alive packet is sent is left to the implementation of the transmitter, but assuming that the sending timing is constant for each model and device, it is measured and set in advance. For example, in the case of a device that sends a keep-alive packet 10 milliseconds after the frame alignment point, the keep-alive sending time dp is set to 10 milliseconds. From the above, the latency can be estimated using the following formula: [Mathematical Expression 4] Latency = tr - t0 - dp

[0068] If the latency can be estimated, the offset value can be estimated as a value obtained by adding a transmission adjustment time (for example, 200 microseconds).

[0069] Figure 7 shows an example of the above calculation in the form of a timing chart. There is a latency before the keep-alive RTP packet transmitted from the transmitter is received by ST2110 ancillary data inserter 10. The latency can be calculated by subtracting the keep-alive transmission time dp of the original transmitter from the value tr - t0. This is added to the transmission adjustment time to obtain the offset value, and the time at which ST2110 ancillary data inserter 10 wishes to transmit the packet for frame 1 can be calculated by adding the offset value to the predicted time t1.

[0070] From the above, the offset estimation unit 17 can estimate the offset value. In the above example, the offset value is estimated based on the characteristics of the auxiliary data RTP packets in the VANC section and the keep-alive RTP packets, but if other types of auxiliary data with characteristics can be used, they may also be used.

[0071] Although one embodiment of the present invention has been described above, the present invention is not limited to the above embodiment and can be embodied in various other forms without departing from the spirit of the present invention. Therefore, the above embodiment is merely an example in all respects and should not be interpreted as being restrictive. For example, the offset estimation unit 17 may be used even in an environment that does not go through a WAN and is configured only with a LAN.

[0072] 10 ST2110 ancillary data inserter 11 RTP packet receiving section 12 Time stamp derivation section 13 Alignment point calculation section 14 Ancillary data generation section 15 Received ancillary data processing section 16 RTP packet transmitting section 17 Offset estimation section 100 Time synchronization section d Difference between the timestamp value of the received RTP packet and the current time dp Time from the alignment point when the original ancillary data transmitter transmits a keep-alive packet D( ) Array for recording the history of timestamp values. D(n) indicates the timestamp value n times before. m Number of timestamp increment patterns R Timestamp increment at fixed intervals t0 Alignment point time of the frame currently being processed stamped in the RTP packet t1 Alignment point time of the frame after t0 Tcurrent Timestamp value of the received RTP packet Tnext Derived timestamp value of the next frame tnext Derived alignment point time of the next frame tnow Number of PTP seconds from the current time tr Reception time of the RTP packet t(T) Function to find time t at timestamp value T

Claims

1. An ST2110 ancillary data inserter comprising a receiver and transmitter for RTP packets in the ancillary data packet format defined in the SMPTE ST2110-40 standard, wherein the receiver uses the property that the timestamp value of the received ancillary data RTP packet increases continuously at regular intervals to predict the RTP timestamp value and alignment point time of the next frame or field, and generates an arbitrary transmitted ancillary data RTP packet containing the RTP timestamp value at the timing when a predetermined offset value is added to the alignment point time, and transmits the generated packet from the transmitter, wherein the received ancillary data RTP packet is optionally processed and transmitted by the transmitter, or optionally deleted.

2. The ST2110 auxiliary data inserter of claim 1, characterized in that the predetermined offset value is estimated using the arrival time of the received auxiliary data RTP packet, the time of the timestamp value stamped on the received auxiliary data RTP packet, and the characteristic that the received auxiliary data RTP packet is transmitted at a fixed time from the alignment point time of the frame.

3. A program that causes a computer to function as the ST2110 auxiliary data inserter according to claim 1 or claim 2.

Images