Ancillary data inserter and program for SMPTE ST2110

Abstract

In the SMPTE ST2110 standard for transmitting video, audio, and auxiliary data in separate RTP streams over an IP network, this enables the transmission of a separate auxiliary data RTP stream to which any auxiliary data has been added without frame delay to the original auxiliary data RTP stream. The present invention uses the periodicity and continuity of timestamp values ​​of RTP packets to predict the timestamp values ​​and packet transmission times of future frames, and transmits the target auxiliary data.

Description

[Technical field]

[0001] The present invention relates to an inserter and program for auxiliary data according to the SMPTE ST2110 standard. [Background technology]

[0002] The SMPTE ST2110 standard, which transmits video, audio, and auxiliary data in separate streams over an IP network, has the unique feature that video, audio, and auxiliary data (ancillary) are in separate RTP (Real-time Transport Protocol) streams and can be handled individually.

[0003] Among the above standards, the RTP timestamp value of ancillary data in the SMPTE ST2110-40 standard in Non-Patent Document 2 is the same as the RTP timestamp value of each frame (field in the case of interlaced video. Unless otherwise specified, hereinafter, when frame is written, it refers to field in the case of interlaced video) of the video, and represents the time at the alignment point, which is the boundary between each frame. In other words, it is synonymous with the start time of the first scan line of each frame. If there are multiple ancillary data RTP packets in one frame, they have the same timestamp value.

[0004] In the SMPTE ST2110 standard, the clock frequency of the RTP timestamp value of the ancillary data is set to 90 kHz. Therefore, in the case of progressive 30 / 1.001 fps (frames per second) (hereinafter referred to as "29.97 fps" in this specification), the RTP timestamp value of the ancillary data increases in increments of 90,000 / (30 / 1.001) = 3,003 (0 → 3,003 → 6,006 → 9,009 → ...). In the case of interlaced 29.97 fps, the timestamp value of the second field does not become an integer, so the fraction is rounded down. (0 → 1,501 → 3,003 → 4,504 → 6,006 → ...)

[0005] The timing of transmitting RTP packets of auxiliary data is not clearly defined, but the RFC8331 standard (Non-Patent Document 4) referenced by the SMPTE ST2110-40 standard states that they should be transmitted as soon as possible, and that as a guideline, they should be transmitted within 1 millisecond.

[0006] Ancillary data RTP packets may contain multiple pieces of ancillary data. At least one RTP packet must be sent for each frame, and if there is no ancillary data to send, an RTP packet containing only the header is sent. This is called keep-alive. The timing of sending the keep-alive packet is not specified. The last RTP packet in a frame has a flag called the marker bit set to 1. [Prior art documents] [Patent documents]

[0007] [Patent Document 1] Patent application 2024-73266 [Non-patent literature]

[0008] [Non-Patent Document 1] SMPTE ST2110-10 [Non-Patent Document 2] SMPTE ST2110-40 [Non-Patent Document 3] SMPTE ST2059-1 [Non-Patent Document 4] RFC 8331 Summary of the Invention [Problem to be solved by the invention]

[0009] If you want to insert any auxiliary data into an existing RTP stream of SMPTE ST2110 standard auxiliary data, there is no technology or product that allows insertion over an IP network, so you have to convert it to SDI (Serial Digital Interface) first and then insert it in SDI form. This means that the benefits of IP cannot be utilized, and not only does it make the system more complicated, it also has many disadvantages such as increased delays, increased costs, more points of failure, and even loss of video and audio in the event of equipment failure.

[0010] Patent Document 1 discloses a technology for receiving an existing SMPTE ST2110 standard RTP stream and reusing the RTP timestamp value to transmit another SMPTE ST2110 standard RTP stream. However, while this technology is applicable to auxiliary data if it is only processing of received auxiliary data, when inserting arbitrary auxiliary data, there are cases where the RTP packet for the frame to be inserted has not yet been received at the time when the auxiliary data to be inserted is to be sent and the RTP timestamp value cannot be reused, making this technology difficult to apply.

[0011] The present invention has been made in view of the above problems, and has an object to enable insertion of any auxiliary data into an RTP stream of auxiliary data conforming to the SMPTE ST2110 standard without frame delay. [Means for solving the problem]

[0012] In order to solve the above problems, the ST2110 auxiliary data inserter of the present invention comprises: Equipped with receiver and transmitter of RTP packets in the packet format of auxiliary data defined in the SMPTE ST2110-40 standard, The RTP timestamp value and alignment point time of the next frame or field are predicted using the property that the timestamp value of the auxiliary data RTP packet received by the receiving unit increases continuously at a constant interval; The present invention is characterized in that an RTP packet of any auxiliary data including the predicted RTP timestamp value is transmitted from the transmitting unit at a timing obtained by adding a predetermined offset value to the alignment point time, The auxiliary data received by the receiving section is optionally processed and then transmitted by the transmitting section, or is optionally deleted.

[0013] In addition, in the ST2110 auxiliary data inserter of the present invention, the predetermined offset value is estimated using the arrival time of the RTP packet that has arrived at the receiving unit, the time of the timestamp value stamped on the RTP packet, and characteristics of the RTP packet.

[0014] Furthermore, a general-purpose computer may be configured to have the functions of the above-mentioned device. Effect of the Invention

[0015] According to the present invention, it is possible to insert or process any auxiliary data into an RTP stream of auxiliary data conforming to the SMPTE ST2110 standard without frame delay, and this can be realized with a simple configuration. In addition, a single device can process multiple ST2110 auxiliary data RTP streams, which also offers significant cost benefits.

[0016] In addition, since there is absolutely no interference with the video RTP stream or the audio RTP stream, even if the device breaks down, there is no impact on the quality of the video and audio. [Brief description of the drawings]

[0017] [Figure 1] FIG. 2 is a block diagram showing the functional configuration of an ST2110 auxiliary data inserter according to an embodiment of the present invention. [Diagram 2] 10 is a flowchart showing a process of an auxiliary data generating unit according to an embodiment of the present invention. [Diagram 3] 5 is a flowchart showing the process of a reception auxiliary data processing unit according to an embodiment of the present invention. [Figure 4] 10 is a flowchart showing a process of an RTP packet transmitting unit according to an embodiment of the present invention. [Diagram 5] 5 is a flowchart showing a process of an offset estimation unit according to an embodiment of the present invention. [Figure 6] 13 is an example of a timing chart showing an offset value estimation performed by the offset estimating unit based on a VANC packet. [Figure 7] 13 is an example of a timing chart showing an offset value estimation performed by the offset estimating unit based on a keep-alive packet. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0018] Hereinafter, the embodiments of the present invention will be described in detail with reference to the drawings. The same elements are given the same reference numerals, and duplicated explanations will be omitted. The following embodiments are merely examples for explaining the present invention, and are not intended to limit the present invention to the embodiments. Furthermore, the present invention can be modified in various ways without departing from the gist of the invention.

[0019] FIG. 1 is a block diagram showing the functional configuration of an ST2110 auxiliary data inserter 10 to which an embodiment of the present invention can be applied. ST2110 ancillary data inserter 10 includes an RTP packet receiver 11, a time stamp derivation unit 12, an alignment point calculator 13, ancillary data generator 14, received ancillary data processor 15, an RTP packet transmitter 16, an offset estimation unit 17, and a time synchronization unit 100. The solid arrows within the frame of the diagram of ST2110 ancillary data inserter 10 indicate the flow of processing, and the dotted arrows indicate the notification of information. The time synchronization unit 100 may perform synchronization by a general method using the Precision Time Protocol (PTP) envisaged in the SMPTE ST2110 standard, or may use the derived PTP time shown in Patent Document 1. Each device according to the embodiment of the present invention may be configured as a dedicated device, or a general-purpose computer may be configured to have the functions of the device according to the embodiment of the present invention.

[0020] The RTP packet receiver 11 receives RTP packets in the packet format of ancillary data defined by the SMPTE ST2110 standard, and outputs them to the time stamp derivation unit 12 , the received ancillary data processor 15 , and the offset estimation unit 17 .

[0021] The time stamp derivation unit 12 derives a time stamp value of the frame next to the current frame. The timestamp value Tcurrent of the delivered RTP packet is taken, and if it has changed from the timestamp value of the previously received RTP packet, the difference d is calculated. In order to calculate the correct difference even when the timestamp value wraps around, the calculation of d is handled as int32. The array D, which records the history of timestamp values, is shifted (D(n)←D(n-1)), and the timestamp value Tcurrent is substituted for D(0).

[0022] Although d is essentially the same as 90,000 / frame rate, depending on the frame rate it cannot be divided evenly to form an integer, and so there is a discrepancy between rounding up and down depending on the timing. Therefore, Table 1 shows the number of timestamp increment patterns m and the timestamp fixed interval increment R that make R = 90,000 / frame rate × m an integer, based on the value of d for each frame rate used in SDI signals specified in the SMPTE ST352 and SMPTE ST2082-10 standards.

[0023] [Table 1]

[0024] Based on d from Table 1, m and R are found, and the timestamp value Tnext of the next frame is derived using the following formula. Note that if d is 938, a judgment cannot be made, so the judgment is made based on the value of 937 or 939. Also, if d is 750, the judgment is made based on whether it is 750 four times in a row.

[0025] [Number 1] Tnext = D(m-1) + R

[0026] The above formula takes advantage of the property that the increase in the timestamp value over m frames is exactly R, and derives the timestamp value of the next frame by adding R to the timestamp value m frames before the next frame, without being affected by truncation.

[0027] The Tnext derived above is output to the alignment point calculation unit 13.

[0028] The alignment point calculation unit 13 converts the input timestamp value into time. Since the RTP timestamp value is 32 bits and wraps around at a fixed time, it is impossible to determine the exact time based on this value alone. To solve this problem, the SMPTE ST2110 standard uses a common clock, PTP, to synchronize each RTP stream.

[0029] As an example, if the current time obtained from the time synchronization unit 100 is tnow, the time t(T) represented by the time stamp value T can be calculated by the following function, provided that the time represented by the time stamp value T is closer to the past than tnow. Note that the "close past" refers to within the period (approximately 13.2 hours) at which the time stamp value wraps around. In this specification, time indicates the number of seconds elapsed from SMPTE_EPOCH defined in SMPTE ST2059-1 of Non-Patent Document 3. In the formula, int(A) indicates the integer value of A truncated to 1, and A mod B indicates the remainder when A is divided by B. [Number 2] t(T) = ( int(tnow × 90,000) - ((int(tnow × 90,000) mod 2^32 - T) mod 2^32) ) / 90,000

[0030] The t calculated above has a time resolution of only about 11 microseconds because it is based on the 90 kHz RTP clock. However, as mentioned above, the timing of sending auxiliary data packets is not clearly defined, so this level of error can be absorbed.

[0031] Using the above-mentioned formula 2, the alignment point time tnext of the next frame is calculated from the time stamp value Tnext, and Tnext and tnext are output to the auxiliary data generating unit 14.

[0032] Auxiliary data generator 14 waits until the packet transmission time based on values ​​obtained from alignment point calculator 13 and, if necessary, offset estimator 17, generates an auxiliary data list, and outputs it to RTP packet transmitter 16. When there are multiple pieces of auxiliary data to be generated, multiple auxiliary data generators 14 may operate simultaneously.

[0033] 2 is a flowchart showing the processing of the auxiliary data generating unit 14. Specific processing steps will be described.

[0034] In step S1401, a value obtained by adding a predetermined offset value to the alignment point time tnext of the next frame input from alignment point calculation unit 13 is set as the transmission time of the auxiliary data to be generated, and the process proceeds to step S1402.

[0035] The offset value is the sum of two values: a value (hereinafter referred to as latency) that adjusts the latency between the ST2110 auxiliary data inserter 10 and the transmitter of the RTP packet that the ST2110 auxiliary data inserter 10 is receiving, and a value (hereinafter referred to as transmission adjustment time) that adjusts how much time has passed since the alignment point time of each frame before auxiliary data is transmitted.

[0036] The latency value can be assumed to be 0 in an environment that consists only of a LAN and does not go through a WAN (Wide Area Network).

[0037] The transmission adjustment time may be preset to a fixed value not exceeding 1 millisecond (e.g., 200 microseconds. In this case, the offset value will be 200 microseconds in total). The transmission adjustment time can be 0, but since the timing of transmitting the additional auxiliary data is simultaneous with the frame alignment point time, there is a possibility that the receiver may fail to receive the data and malfunction, so this is a slight waiting time.

[0038] The offset value may be a value estimated by the offset estimation unit 17. Details of the offset estimation unit 17 will be described later.

[0039] In step S1402, the current time is obtained from the time synchronization unit 100, and the process proceeds to step S1403.

[0040] In step S1403, the acquired current time is compared with the auxiliary data transmission time, and if the current time is after the auxiliary data transmission time (Yes), the process proceeds to step S1404, otherwise (No), the process returns to step S1402.

[0041] In step S1404, zero or more pieces of arbitrary auxiliary data are generated, and an auxiliary data list is generated in which the auxiliary data are stored. Then, the process proceeds to step S1405.

[0042] In step S1405, the auxiliary data list generated in step S1404 and the value of Tnext input from alignment point calculation unit 13 as the transmission timestamp value are output to RTP packet transmission unit 16, and the processing of auxiliary data generation unit 14 is terminated. At this time, even if the auxiliary data list is empty, an empty list is output to RTP packet transmission unit 16.

[0043] The received auxiliary data processor 15 is responsible for retransmitting the received auxiliary data as separate auxiliary data. At this time, instead of simply retransmitting the data as is, the data may be optionally processed or deleted. 3 is a flow chart showing the processing of the reception auxiliary data processing unit 15. Specific processing steps will be described.

[0044] In step S1501, the auxiliary data list that stores the auxiliary data to be transmitted is cleared, and the process proceeds to step S1502.

[0045] In step S1502, a loop is performed the number of times equal to the number of auxiliary data included in the received RTP packet input from the RTP packet receiving unit 11, the auxiliary data is extracted in order, and the process proceeds to step S1503. If there is no auxiliary data in the received RTP packet, the loop is skipped and the process proceeds to step S1505.

[0046] Step S1503 optionally performs processing on the auxiliary data. If the auxiliary data is to be held, it is added directly to the auxiliary data list. Alternatively, the auxiliary data is subjected to any processing before being added to the auxiliary data list. Alternatively, one or more other pieces of auxiliary data are generated based on the content of the auxiliary data, and are added to the auxiliary data list. Alternatively, the auxiliary data is arbitrarily deleted by not adding it to the auxiliary data list. Moreover, any combination of the above processes may be performed. Then, the process transitions to step S1504.

[0047] In step S1504, if the loop end condition is satisfied, the process proceeds to step S1505, otherwise the process returns to step S1502 and proceeds to the next loop.

[0048] In step S1505, the marker bit flag included in the header of the received RTP packet is checked, and if it is 1, the process proceeds to step S1506, or if it is off, the process proceeds to step S1507.

[0049] In step S1506, the EndRecv flag is set to 1, which indicates that no new RTP packets will arrive for the current frame, and the process proceeds to step S1507.

[0050] In step S1507, the created auxiliary data list and the timestamp value of the received RTP packet as the transmission timestamp value are output to RTP packet transmitting unit 16, and received auxiliary data processing unit 15 is terminated. At this time, even if the auxiliary data list is empty, an empty list is output to RTP packet transmitting unit 16.

[0051] The RTP packet transmitter 16 converts the auxiliary data list and transmission timestamp value input from the auxiliary data generator 14 or the received auxiliary data processor 15 into an RTP packet and transmits it over the network. Figure 4 is a flow chart showing the processing of the RTP packet transmitter 16. Specific processing steps will be described. Note that the RTP packet transmitter 16 performs appropriate exclusive control so that multiple RTP packet transmitters 16 do not operate simultaneously.

[0052] In step S1601, the value of the EndRecv flag is checked, and if it is 1, the process proceeds to step S1602, and if it is 0, the process proceeds to step S1604.

[0053] In step S1602, it is determined whether the RTP packet to be transmitted is the last packet in the frame being processed. If the auxiliary data list has already been input from the auxiliary data generator 14 (or from all of the auxiliary data generators 14 if multiple auxiliary data generators 14 are operating), it is determined that it is the last packet (Yes), and the process proceeds to step S1603. If not (No), the process proceeds to step S1604.

[0054] In step S1603, the marker bit of an RTP packet to be generated is set to 1. Also, in preparation for processing the next frame, the EndRecv flag is reset to 0. Then, the process proceeds to step S1604.

[0055] In step S1604, the process checks whether the auxiliary data list to be transmitted is empty and the marker bit of the RTP packet to be generated, and if the auxiliary data list is empty and the marker bit is 0 (Yes), it is determined that there is no information to be transmitted, and the process of RTP packet transmitting unit 16 is terminated. If not (No), the process proceeds to step S1605.

[0056] In step S1605, binary data of an RTP packet is assembled based on the input auxiliary data list and transmission time stamp value, and the process proceeds to step S1606. Note that information required for the RTP packet, such as sequence number and SSRC, is appropriately described in accordance with the standard.

[0057] In step S1606, the binary data assembled in step S1605 is transmitted to the network, and the process of the RTP packet transmitting unit 16 ends.

[0058] As a result, ST2110 ancillary data inserter 10 can perform any addition, processing, or deletion on the received ancillary data and then transmit another ancillary data RTP stream.

[0059] A receiving device that receives the auxiliary data stream transmitted by ST2110 auxiliary data inserter 10 receives the original RTP stream for video and audio, and receives the RTP stream output by ST2110 auxiliary data inserter 10 for auxiliary data, thereby being able to use the inserted auxiliary data.

[0060] The offset estimation unit 17 estimates an offset value including a reception latency and a transmission adjustment time based on the characteristics of the RTP packet received by the RTP packet receiving unit 11, and makes the offset value available to the auxiliary data generating unit .

[0061] The type of RTP packet used to estimate the offset value is preferably auxiliary data of the line numbers of the VANC section (vertical blanking section) from the first line of a frame to the line where the video starts, such as ATC (Ancillary Timecode) or VPID (Video Payload ID), which are often used as auxiliary data. Here, the range of the VANC section does not need to be clear, and it may be in the form of targeting line numbers 1 to 20. If auxiliary data for the VANC section is not available, keep-alive packets that are transmitted when there is no auxiliary data may be used.

[0062] 5 is a flowchart showing the process of the offset estimation unit 17. Specific process steps will be described.

[0063] In step S1701, the contents of the RTP packet input from the RTP packet receiving unit 11 are checked to determine whether the packet is to be used for estimating the offset value. If the packet is a target (Yes), the process proceeds to step S1702. If not (No), the process of the offset estimating unit 17 ends.

[0064] In step S1702, the current time is obtained from the time synchronization unit 100 and is set as the reception time tr of the RTP packet. Furthermore, the alignment point time t0 of the frame is calculated from the timestamp value of the packet using Equation 2, and the process proceeds to step S1703.

[0065] In step S1703, if the RTP packet is a packet in the VANC section, it can be regarded as an RTP packet transmitted within 1 millisecond from the alignment point, so that the offset value is estimated by the following formula. [Number 3] Offset value = tr - t0

[0066] An example of the above calculation is shown in a timing chart in Figure 6. Latency occurs when an RTP packet in the VANC section is sent from an original transmitter in a remote location via a low-jitter line such as a dark fiber WAN line until it is received by the ST2110 ancillary data inserter 10. This packet reception time tr is the timing at which the ST2110 ancillary data inserter 10 wishes to send ancillary data for frame 0 in Figure 6, and the time at which the ST2110 ancillary data inserter 10 wishes to send the next packet for frame 1 can be found by adding an offset value to the predicted time t1.

[0067] If the RTP packet is a keep-alive packet, the timing of sending the keep-alive packet is left to the implementation of the transmitter, but it is assumed that the sending timing is constant for each model and device, and is measured and set in advance. For example, for a device that sends a keep-alive packet 10 milliseconds after the frame alignment point, the keep-alive sending time dp = 10 milliseconds is set. Based on the above, the latency can be estimated using the following formula: [Number 4] Latency = tr - t0 - dp

[0068] If the latency can be estimated, the offset value can be estimated by adding a transmission adjustment time (eg, 200 microseconds).

[0069] An example of the above calculation is shown in a timing chart in Figure 7. There is a latency in the keep-alive RTP packet sent from the transmitter before it is received by ST2110 ancillary data inserter 10. The latency is found by subtracting the keep-alive transmission time dp of the original transmitter from the value of tr - t0. This is added to the transmission adjustment time to obtain the offset value, and the time at which ST2110 ancillary data inserter 10 wishes to transmit a packet for frame 1 is found by adding the offset value to the predicted time t1.

[0070] From the above, offset estimation unit 17 can estimate the offset value. In the above example, the offset value is estimated based on the characteristics of the auxiliary data RTP packets in the VANC section and the keep-alive RTP packets, but if other types of auxiliary data with characteristics can be used, they may be used.

[0071] Although one embodiment of the present invention has been described above, the present invention is not limited to the above embodiment, and can be embodied in various other forms without departing from the scope of the present invention. Therefore, the above embodiment is merely an example in every respect, and should not be interpreted as being restrictive. For example, the offset estimation unit 17 may be used even in an environment that does not go through a WAN and is composed only of a LAN. [Explanation of symbols]

[0072] 10 ST2110 Auxiliary Data Inserter 11 RTP packet receiver 12 Timestamp derivation part 13 Alignment point calculation section 14 Auxiliary Data Generation Unit 15 Reception auxiliary data processing section 16 RTP packet transmitter 17 Offset Estimation Unit 100 Time Synchronization Unit d The difference between the timestamp value of the received RTP packet and the current time dp The time from the alignment point when the original ancillary data transmitter sends a keep-alive packet D( ) is an array that records the history of timestamp values. D(n) indicates the timestamp value n years ago. m Number of timestamp increment patterns R timestamp increment at regular intervals t0 Alignment point time of the currently processed frame stamped in the RTP packet t1 Alignment point time of the frame next to t0 Tcurrent The timestamp value of the received RTP packet. Tnext The derived timestamp value of the next frame tnext: The derived alignment point time of the next frame tnow Current time in PTP seconds tr RTP packet reception time t(T) A function that calculates the time t for a timestamp value T.

Claims

1. The present invention includes a receiving unit and a transmitting unit for an RTP packet having a packet format of auxiliary data defined in the SMPTE ST2110-40 standard, predicting the RTP timestamp value and alignment point time of the next frame or field using the property that the timestamp value of the reception auxiliary data RTP packet received by the receiving unit increases continuously at a constant interval; generating an arbitrary transmission auxiliary data RTP packet including the RTP time stamp value at a timing obtained by adding a predetermined offset value to the alignment point time, and transmitting the packet from the transmission unit; The received auxiliary data RTP packet is arbitrarily processed and transmitted by the transmitting unit, or arbitrarily deleted. ST2110 Auxiliary Data Inserter.

2. The ST2110 auxiliary data inserter according to claim 1, characterized in that the predetermined offset value is estimated using the arrival time of the received auxiliary data RTP packet, the time of the timestamp value stamped on the received auxiliary data RTP packet, and the characteristic that the received auxiliary data RTP packet is transmitted at a fixed time from the alignment point time of the frame.

3. A program for causing a computer to function as the ST2110 auxiliary data inserter according to claim 1 or 2.

Images