Delay time management server and timing adjustment method

Abstract

To provide a delay time management server that can manage more accurate delay time and vertical synchronization shift amount in transmitting image data from an image transmitting device to an image receiving device via a network.SOLUTION: An image transmitting device including a first camera 11 and an image receiving device including a second camera 11 and a display 22 are connected with each other via a network 30. A delay time measuring unit 28 measures a delay time when the image receiving device receives first image data that is generated by the first camera 11 and transmitted to the image receiving device by the image transmitting device and displays the data on the display 22. A vertical synchronization shift amount measuring unit (vertical synchronization delay circuit 123) measures a vertical synchronization shift amount indicating the time in which the vertical synchronization between the first image data and second image data generated by the second camera and displayed on the display 22 is shifted. A delay time management server 50 stores the delay time and the vertical synchronization shift amount transmitted from the image receiving device via the network 30.SELECTED DRAWING: Figure 13

Description

Technical Field

[0001] The present invention relates to a delay time management server and a timing adjustment method.

Background Art

[0002] Image data may be transmitted from an image transmission device arranged at at least one site to an image reception device via a network, and the image transmitted from the image transmission device may be displayed on a display device provided in the image reception device. At this time, with the practical implementation of the fifth-generation mobile communication system (5G), image data can be transmitted with low latency, so the delay due to the transmission of image data is negligibly small.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, the delay due to internal processing in the image transmission device and the image reception device cannot be ignored. In addition, when image transmission devices are arranged at a plurality of sites and image data is transmitted from the plurality of image transmission devices to the image reception device, there is variation in the delay time due to internal processing in the plurality of image transmission devices, so the timing of the images displayed on the display device may vary, causing a sense of discomfort. There is a need to grasp the delay time including the delay due to internal processing in the image transmission device and the image reception device when transmitting image data from the image transmission device to the image reception device via the network.

[0005] An object of the present invention is to provide a delay time management server capable of managing more accurate delay time and vertical synchronization deviation amount when transmitting image data from an image transmission device to an image reception device via a network. Another object of the present invention is to provide a timing adjustment method capable of adjusting timing including the timing of a frame of image data so that the timings match when displaying image data obtained by photographing a subject that performs a predetermined operation separately at a transmission-side base point and a reference base point at the reference base point.

Means for Solving the Problems

[0006] The present invention provides a delay time management server that is connected via a network between an image transmission device including a first camera and an image reception device including a second camera and a display device. The image reception device includes a delay time measurement unit that measures a delay time when the image reception device receives first image data generated by the first camera and transmitted by the image transmission device to the image reception device and displays the received data on the display device. a delay time measurement unit that calculates, as the delay time, a difference between the time when pixel data of a reference pixel, which is a pixel at a specific position in a specific row within a frame, is generated by an imaging element included in the first camera and the time when the reference pixel is displayed on the display device The image reception device further includes a vertical synchronization deviation amount measurement unit that measures a vertical synchronization deviation amount indicating a time during which the vertical synchronization between the first image data and second image data generated by the second camera and displayed on the display device is deviated. The delay time management server stores the delay time and the vertical synchronization deviation amount transmitted from the image reception device via the network.

[0007] In the present invention, a first camera included in an image transmission device arranged at a transmission-side base point play a musical instrument transmits first image data generated by photographing a first subject that performs an operation to an image reception device arranged at a reference base point via a network, and the delay time when the image reception device displays the received first image data on a display device calculating, as the delay time, a difference between the time when pixel data of a reference pixel, which is a pixel at a specific position in a specific row within a frame, is generated by an imaging element included in the first camera and the time when the reference pixel is displayed on the display device between the first image data and a second camera included in the image reception device play a musical instrumentMeasure a vertical synchronization deviation amount indicating the time when the vertical synchronization is shifted from the second image data generated by photographing a second subject that performs an operation, and set a delay time correction value based on the delay time obtained by adding the delay time and the vertical synchronization deviation amount to a timing instruction unit arranged at the transmission-side base. Then, the timing instruction unit notifies the first subject of the timing to start the operation at a time earlier by the time indicated by the delay time correction value from the time when the first subject is instructed to start the operation. Provide a timing adjustment method for sending a signal to notify the first subject of the timing to start the operation. play a musical instrument from the time when the first subject is instructed to start the operation until the time earlier by the time indicated by the delay time correction value play a musical instrument Provide a timing adjustment method for sending a signal to notify the first subject of the timing to start the operation.

Effect of the Invention

[0008] According to the delay time management server of the present invention, it is possible to manage a more accurate delay time and vertical synchronization deviation amount when transmitting image data from an image transmission device to an image reception device via a network. According to the timing adjustment method of the present invention, it is possible to adjust the timing including the timing of the frame of the image data so that the timings when displaying the image data photographed by the subject performing a predetermined operation separately performed at the transmission-side base and the reference base match at the reference base.

Brief Description of the Drawings

[0009]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

[0010] Hereinafter, an image transmission apparatus, an image reception apparatus, a delay time measurement apparatus, a delay time management server, and a timing adjustment method according to one or more embodiments will be described with reference to the accompanying drawings.

[0011] First, a concept of an image transmission system that transmits image data from a plurality of image transmission apparatuses to an image reception apparatus via a network will be described with reference to FIG. 1. The image data is moving image data. Note that the image transmission system may transmit audio data in addition to the image data. Processing of transmission and reception of audio data will be omitted from illustration.

[0012] In FIG. 1, image transmission apparatuses 10A to 10C are arranged at three different locations. An image reception apparatus 20 is arranged at a location different from the locations where the image transmission apparatuses 10A to 10C are arranged. The image transmission apparatuses 10A to 10C include a camera 11 that captures a subject, and a transmission unit 12 that transmits image data generated by the camera 11 capturing the subject.

[0013] The image data transmitted from the image transmission apparatuses 10A to 10C is transmitted to the image reception apparatus 20 via the network 30. The network 30 is typically the Internet. The image reception apparatus 20 includes a reception unit 21 that receives the image data, and a display device 22 that displays a moving image based on the image data. The display device 22 may display the moving images from the image transmission apparatuses 10A to 10C simultaneously, or may display while switching the moving images from the image transmission apparatuses 10A to 10C. An image transmission apparatus that does not specify any of the image transmission apparatuses 10A to 10C is referred to as an image transmission apparatus 10.

[0014] <Specific configuration examples of the image transmission apparatus, the image reception apparatus, and the delay time measurement apparatus> FIG. 2 shows a configuration example of an image transmission device 10 and an image reception device 20 configured to measure the delay time from the time when the camera 11 in the image transmission device 10 starts shooting a moving image to the time when the display device 22 in the image reception device 20 starts displaying the moving image. The image transmission device 10 and the image reception device 20 shown in FIG. 2 have a configuration suitable for grasping the delay time including the delay due to their internal processing.

[0015] The delay time measurement unit 28 provided in the image reception device 20 constitutes a delay time measurement device according to one or more embodiments, and measures a more accurate delay time including the delay due to the internal processing in the image transmission device 10 and the image reception device 20 as described below. The image transmission device 10 is an image transmission device according to one or more embodiments. The image reception device 20 is an image reception device according to one or more embodiments.

[0016] As shown in FIG. 2, the image transmission device 10 includes, in addition to the camera 11, a GNSS satellite radio wave clock 13, a clock generation unit 14, an image memory 15, and a communication unit 16. The camera 11 has a control unit 111, an imaging element 112, and an image processing circuit 113. The image memory 15 includes a write / read control unit 150. The communication unit 16 functions as the transmission unit 12 in FIG. 1.

[0017] The image reception device 20 includes, in addition to the display device 22, a GNSS satellite radio wave clock 23, a clock generation unit 24, a communication unit 25, an image processing circuit 26, an image memory 27, and a delay time measurement unit 28. The display device 22 has a control unit 221, a drive circuit 222, and a liquid crystal panel 223. The image memory 27 includes a write / read control unit 270. The communication unit 25 functions as the reception unit 21 in FIG. 1.

[0018] The GNSS satellite radio clock 13 receives radio waves from a satellite 40 for a Global Navigation Satellite System (GNSS), and outputs a 1PPS signal which is a pulse at 1-second intervals shown in Fig. 3(a), and a 10 MHz clock (hereinafter, 10 MHz clock) shown in Fig. 3(b). As an example, GNSS is GPS (Global Positioning System).

[0019] Based on the input 1PPS signal and 10 MHz clock, the clock generation unit 14 generates a 148.5 MHz clock (hereinafter, 148.5 MHz clock) shown in Fig. 3(c), and supplies it to the camera 11, the image memory 15, and the communication unit 16. For the sake of illustration, the 148.5 MHz clock shown in Fig. 3(c) is shown with the period of one clock being significantly lengthened. Note that, in addition to 148.5 MHz, the clock generation unit 14 may generate clocks of other frequencies for video synchronization such as 297 MHz, 74.25 MHz, 27 MHz, etc.

[0020] The image transmission device 10 may be provided with a clock generation unit that generates a clock including time information based on radio waves received from a satellite 40 for GNSS. The 10 MHz clock may be supplied to the camera 11, the image memory 15, and the communication unit 16 so that the camera 11, the image memory 15, and the communication unit 16 operate with the 10 MHz clock. In this case, the GNSS satellite radio clock 13 serves as the clock generation unit that generates a clock including time information. The frequency of the clock supplied to the camera 11, the image memory 15, and the communication unit 16 is not limited.

[0021] Assume that the GNSS satellite radio clock 13 generates a pulse of the 1PPS signal shown in Fig. 3(a) at time 0:00:00 and the next pulse at time 0:00:01. The clock generation unit 14 outputs a 148.5 MHz clock including time information, such as clocks numbered 0 to 148.5×10 -6 -1 based on time 0:00:00, and clocks numbered 0 to 148.5×10 -6 -1 based on time 0:00:01.

[0022] The control unit 111 of the camera 11 controls the electronic shutter in the imaging device 112, and the imaging device 112 images the subject. The control unit 111 may be a central processing unit included in the camera 11. The image data generated by the imaging device 112 imaging the subject is subjected to various image processes by the image processing circuit 113 and supplied to the image memory 15.

[0023] The image process in the image processing circuit 113 is one or more of defective pixel interpolation processing, black level processing, white balance processing, demosaicing processing, shading correction processing, and compression encoding processing. The image process in the image processing circuit 113 includes at least compression encoding processing.

[0024] The control unit 111 controls to superimpose, as metadata, the clock number of the 148.5 MHz clock indicating the time t0 when the imaging device 112 generates pixel data by turning off the electronic shutter of a predetermined reference pixel, on the blanking period of the image data. The reference pixel is preferably the first pixel in the first row within the frame. That is, the clock number included in the image data output by the camera 11 indicates the time when the camera 11 starts generating the image data of each frame. As shown in FIG. 3(c), assume that the clock number Cn0 indicating the time t0 included in the image data output by the camera 11 is, for example, clock number 0.

[0025] The write / read control unit 150 writes the image data to the image memory 15. At this time, the write / read control unit 150 writes, as metadata, the clock number of the 148.5 MHz clock indicating the time t1 when the pixel data of the reference pixel of each frame is written to the image memory 15, to the image memory 15 in a state where it is superimposed on the blanking period of the image data. As shown in FIG. 3(c), assume that the clock number indicating the time t1 included in the image data written to the image memory 15 is Cn1. The clock number Cn0 and the clock number Cn1 are superimposed on the image data written to the image memory 15.

[0026] The writing / reading control unit 150 reads out the image data stored in the image memory 15 and supplies it to the communication unit 16. The communication unit 16 packetizes the image data read from the image memory 15 and transmits it to the image receiving device 20 via the network 30. At this time, the communication unit 16 superimposes the clock number of the 148.5 MHz clock indicating the time t2 when the pixel data of the reference pixel of each frame is transmitted, as metadata, on the blanking period of the image data, and packetizes the image data. As shown in FIG. 3(c), assume that the clock number indicating the time t2 when the pixel data of the reference pixel is transmitted is Cn2. As a result, the clock number Cn2 indicates the time t2 when the pixel data of the reference pixel was transmitted.

[0027] In addition, the communication unit 16 adds clock frequency information to the image data and packetizes it in order to cause the image receiving device 20 to transmit the camera 11, the image memory 15, and the communication unit 16 to operate at a 148.5 MHz clock. The packet data transmitted from the image transmitting device 10 to the image receiving device 20 via the network 30 includes image data and clock frequency information, and the clock numbers Cn0, Cn1, and Cn2 are superimposed on the image data.

[0028] The GNSS satellite radio wave clock 23 in the image receiving device 20 also receives radio waves from the satellite 40 and outputs a 1PPS signal that is a pulse at 1-second intervals shown in FIG. 4(a) and a 10 MHz clock shown in FIG. 4(b), similar to the GNSS satellite radio wave clock 13.

[0029] In FIG. 2, the satellite 40 that transmits the radio waves received by the GNSS satellite radio wave clock 13 and the GNSS satellite radio wave clock 23 is the same common satellite 40. The satellite 40 that transmits the radio waves received by the GNSS satellite radio wave clock 13 and the satellite 40 that transmits the radio waves received by the GNSS satellite radio wave clock 23 may be different. The GNSS satellite radio wave clocks 13 and 23 may receive radio waves from any of the plurality of satellites 40 operated by a single global navigation satellite system.

[0030] The communication unit 25 receives packet data including image data and clock frequency information, supplies the clock frequency information to the clock generation unit 24, and supplies the image data to the image processing circuit 26. Since the clock frequency information indicating 148.5 MHz is input to the clock generation unit 24, based on the input 1PPS signal and 10 MHz clock, the 148.5 MHz clock shown in Fig. 4(c) is generated. The 148.5 MHz clock is supplied to the display device 22, the communication unit 25, the image memory 27, and the delay time measurement unit 28.

[0031] The 148.5 MHz clock shown in Fig. 4(c) is the same clock (same phase and same frequency) as the 148.5 MHz clock shown in Fig. 3(c).

[0032] The clock that the clock generation unit 24 supplies to the display device 22, the communication unit 25, the image memory 27, and the delay time measurement unit 28 is a 148.5 MHz clock including time information, as described above.

[0033] As shown in Fig. 4(c), the communication unit 25 superimposes, as metadata, the clock number of the 148.5 MHz clock indicating the time t3 when the pixel data of the reference pixel of each frame is received from the network 30 during the blanking period of the image data. Assume that the clock number indicating the time t3 when the pixel data of the reference pixel is received is Cn3. Therefore, the clock numbers Cn0, Cn1, Cn2, and Cn3 are superimposed on the image data output from the communication unit 25.

[0034] The image data output from the communication unit 25 is subjected to various image processes by the image processing circuit 26 and then supplied to the image memory 27. The image process in the image memory 27 is at least decompression decoding. The image processing circuit 26 supplies the clock numbers Cn0, Cn1, Cn2, and Cn3 included in the image data to the delay time measurement unit 28.

[0035] The write / read control unit 270 writes the image data supplied to the image memory 27 into the image memory 27, reads out the image data stored in the image memory 27, and supplies it to the drive circuit 222 of the display device 22. At this time, the write / read control unit 270 supplies the clock number of the 148.5 MHz clock indicating the time t4 when the pixel data of the reference pixel of each frame is read out to the delay time measurement unit 28. The time t4 is also the time when the pixel data of the reference pixel is supplied to the drive circuit 222. As shown in Fig. 4(c), assume that the clock number indicating the time t4 when the image data is read out from the image memory 27 is Cn4.

[0036] The control unit 221 of the display device 22 controls the drive circuit 222 to display the image data supplied to the drive circuit 222 on the liquid crystal panel 223. The control unit 221 may be a central processing unit provided in the display device 22. The control unit 221 supplies the clock number of the 148.5 MHz clock indicating the time t5 when the pixel data of the reference pixel of each frame is displayed on the liquid crystal panel 223 to the delay time measurement unit 28. As shown in Fig. 4(c), assume that the clock number indicating the time t5 when the pixel data of the reference pixel is displayed on the liquid crystal panel 223 is Cn5.

[0037] Note that the display device 22 does not necessarily need to operate at a 148.5 MHz clock. The reason for supplying a 148.5 MHz clock to the display device 22 is to supply the clock number Cn5 indicating the time t5 at the 148.5 MHz clock to the delay time measurement unit 28.

[0038] When the display device 22 is not a direct-view display device but a projection display device, the liquid crystal panel 223 is a liquid crystal display element for a projection display device. The display device 22 may include a display panel (display element) other than the liquid crystal panel.

[0039] The delay time measurement unit 28 obtains the delay time between the image transmission device 10 and the image reception device 20 according to the processes shown in the flowcharts of Figs. 5 to 7, and determines a delay time correction value for correcting the delay due to the delay time.

[0040] In FIG. 5, the delay time measurement unit 28 sets the shooting time in the image transmission device 10 at step S1 and sets the display time in the image reception device 20 at step S2. FIG. 6 shows the detailed processing of step S1. In FIG. 6, the delay time measurement unit 28 determines at step S11 whether the clock number Cn0 (i.e., time t0) exists. If the clock number Cn0 exists (YES), the delay time measurement unit 28 sets the time t0 indicated by the clock number Cn0 as the shooting time at step S13.

[0041] If the clock number does not exist at step S11 (NO), the delay time measurement unit 28 determines at step S12 whether the clock number Cn1 (i.e., time t1) exists. If the clock number Cn1 exists (YES), the delay time measurement unit 28 sets the time t1 indicated by the clock number Cn1 as the shooting time at step S14. If the clock number Cn1 does not exist (NO), the delay time measurement unit 28 sets the time t2 indicated by the clock number Cn2 as the shooting time at step S15.

[0042] FIG. 7 shows the detailed processing of step S2. In FIG. 7, the delay time measurement unit 28 determines at step S21 whether the clock number Cn5 (i.e., time t5) exists. If the clock number Cn5 exists (YES), the delay time measurement unit 28 sets the time t5 as the display time at step S23.

[0043] If the clock number Cn5 does not exist at step S21 (NO), the delay time measurement unit 28 determines at step S22 whether the clock number Cn4 (i.e., time t4) exists. If the clock number Cn4 exists (YES), the delay time measurement unit 28 sets the time t4 as the display time at step S24. If the clock number Cn4 does not exist (NO), the delay time measurement unit 28 sets the time t3 indicated by the clock number Cn3 as the display time at step S25.

[0044] In FIG. 6, the reason for determining whether or not the clock number Cn0 exists is that there may be a case where the camera 11 is not configured to superimpose the clock number Cn0 indicating the time t0 when the generation of the image data for each frame is started on the image data. Also, the reason for determining whether or not the clock number Cn1 exists is that there may be a case where the image memory 15 is not configured to superimpose the clock number Cn1 indicating the time t1 when the image data is written on the image data.

[0045] In FIG. 7, the reason for determining whether or not the clock number Cn5 exists is that there may be a case where the display device 22 is not configured to supply the clock number Cn5 indicating the time t5 when the image data is displayed on the liquid crystal panel 223 to the delay time measurement unit 28. Also, the reason for determining whether or not the clock number Cn4 exists is that there may be a case where the image memory 27 is not configured to supply the clock number Cn4 indicating the time t4 when the image data is read to the delay time measurement unit 28.

[0046] Returning to FIG. 5, the delay time measurement unit 28 calculates, in step S3, the delay time between the image transmission device 10 and the image reception device 20, which is the difference between the shooting time and the display time. In step S4, the delay time measurement unit 28 determines whether or not the shooting time is time t2 and the display time is time t3. If the shooting time is time t2 and the display time is time t3, the time from time t2 to time t3 is the transmission time itself of the image data by the network 30. Therefore, it does not represent the delay time from when the image transmission device 10 starts generating the image data until the image reception device 20 starts displaying the image data.

[0047] Therefore, if in step S4 the shooting time is time t2 and the display time is time t3 (YES), the delay time measurement unit 28 determines, in step S5, whether or not a fixed correction value measured in advance is set in the delay time measurement unit 28. The fixed correction value is the time obtained by adding one or both of the average time from time t0 to time t2 and the average time from time t3 to time t5.

[0048] If the fixed correction value measured in advance in step S5 is set (YES), the time obtained by adding the fixed correction value to the delay time obtained in step S3 is determined as the delay time correction value, and the process ends.

[0049] If the shooting time is not time t2 and the display time is not time t3 in step S4 (NO), or if the fixed correction value measured in advance in step S5 is not set (NO), the delay time measurement unit 28 determines, in step S7, the delay time obtained in step S3 as the delay time correction value as it is, and ends the process.

[0050] As described above, the image transmission device 10 includes a camera 11, a clock generation unit 14, an image memory 15, and a communication unit 16. The clock generation unit 14 generates a first clock (for example, a 148.5 MHz clock) including time information based on radio waves received from the GNSS satellite 40. The camera 11 operates according to the first clock, and the imaging element 112 generates image data by imaging a subject. The image memory 15 operates according to the first clock, and writes and reads image data based on the control by the write / read control unit 150.

[0051] The communication unit 16 transmits the image data read from the image memory 15 and having at least one of the first to third clock numbers superimposed thereon. The first to third clock numbers are clock numbers Cn0 to Cn2.

[0052] The first clock number indicates the time when the camera 11 generates the pixel data of the reference pixel in the frame by the imaging element 112. If the camera 11 is configured to execute a process of superimposing the first clock number on the image data, the image data including the first clock number is transmitted.

[0053] The second clock number indicates the time when the write / read control unit 150 writes the pixel data of the reference pixel to the image memory 15. If the write / read control unit 150 is configured to execute a process of superimposing the second clock number on the image data, the image data including the second clock number is transmitted. The third clock number indicates the time when the communication unit 16 transmits the pixel data of the reference pixel. If the communication unit 16 is configured to execute a process of superimposing the third clock number on the image data, the image data including the third clock number is transmitted.

[0054] The image receiving device 20 includes a communication unit 25 that receives image data on which at least one of the first clock number, the second clock number, and the third clock number is superimposed. If the communication unit 25 is the first communication unit, the communication unit 16 is the second communication unit. If the communication unit 16 is the first communication unit, the communication unit 25 is the second communication unit.

[0055] The first clock number is the clock number in the first clock generated by the clock generation unit 14 (the first clock generation unit), and indicates the time when the imaging element 112 of the camera 11 generates the pixel data of the reference pixel in the frame. The second clock number is the clock number in the first clock, and indicates the time when the write / read control unit 150 (the first write / read control unit) writes the pixel data of the reference pixel to the image memory 15 (the first image memory). The third clock number is the clock number in the first clock, and indicates the time when the communication unit 16 (the second communication unit) transmits the pixel data of the reference pixel.

[0056] The image receiving device 20 further includes a clock generation unit 24 (the second clock generation unit), an image memory 27 (the second image memory), a display device 22, and a delay time measurement unit 28. The clock generation unit 24 generates a second clock identical to the first clock. The image memory 27 operates according to the second clock, and writes and reads image data based on the control by the write / read control unit 270 (the second write / read control unit). The display device 22 displays the image data read from the image memory 27.

[0057] The delay time measurement unit 28 measures a delay time indicating a difference between any one of the first to third clock numbers and any one of the fourth to sixth clock numbers. The fourth clock number is the clock number at the second clock and indicates the time when the communication unit 25 receives the pixel data of the reference pixel. The fifth clock number is the clock number at the second clock and indicates the time when the write / read control unit 270 reads the pixel data of the reference pixel from the image memory 27. The sixth clock number is the clock number at the second clock and indicates the time when the display device 22 displays the pixel data of the reference pixel.

[0058] When the delay time measurement unit 28 acquires the first clock number (clock number Cn0), it is preferable to measure the delay time using the first clock number. When the delay time measurement unit 28 cannot acquire the first clock number and acquires the second clock number (clock number Cn1), it is preferable to measure the delay time using the second clock number. When the delay time measurement unit 28 acquires the sixth clock number (clock number Cn5), it is preferable to measure the delay time using the sixth clock number. When the delay time measurement unit 28 cannot acquire the sixth clock number and acquires the fifth clock number (clock number Cn4), it is preferable to measure the delay time using the fifth clock number.

[0059] The delay time measurement unit 28 is a delay time measurement device according to one or more embodiments. As described above, the delay time measurement unit 28 obtains a difference between any one of the first to third clock numbers and any one of the fourth to sixth clock numbers. Thereby, the delay time measurement unit 28 measures the delay time when the image receiving device 20 receives and displays the image data generated by the image transmitting device 10 and transmitted to the image receiving device 20.

[0060] According to the image transmission device, image reception device, and delay time measurement device of one or more embodiments, when transmitting image data from the image transmission device 10 to the image reception device 20 via the network 30, it is possible to grasp a more accurate delay time.

[0061] <Delay time management server> In FIG. 8, at bases B1 to B3, image transmission / reception devices 120A to 120C are respectively arranged. The image transmission / reception devices 120A to 120C communicate with each other bidirectionally via the network 30. An image transmission / reception device that does not specify any of the image transmission / reception devices 120A to 120C is referred to as the image transmission / reception device 120. The image transmission / reception device 120 has the configurations of both the image transmission device 10 and the image reception device 20 shown in FIG. 2.

[0062] As shown in FIG. 8, schematically, the image transmission / reception device 120 includes a camera 11, a display device 22, a delay time measurement unit 28, a control unit 122, and a transmission / reception unit 121. Although not shown in FIG. 8, the image transmission / reception device 120 includes a GNSS satellite radio wave clock 13 (or 23) and a clock generation unit 14 (or 24), and a 148.5 MHz clock is supplied to the camera 11 and the display device 22. The transmission / reception unit 121 corresponds to the communication units 16 and 25. The control unit 122 corresponds to the control units 111 and 221.

[0063] The image transmission device 10 shown in FIG. 2 details the configuration when the image transmission / reception device 120 shown in FIG. 8 operates as an image transmission device. The image reception device 20 shown in FIG. 2 details the configuration when the image transmission / reception device 120 shown in FIG. 8 operates as an image reception device.

[0064] The network 30 is connected to a delay time management server 50. The delay time management server 50 communicates bidirectionally with the image transmission / reception apparatuses 120A to 120C. As shown in FIG. 9, the delay time management server 50 stores the delay time when the image transmission / reception apparatuses 120A to 120C arranged at the bases B1 to B3 are on the transmission side and the other image transmission / reception apparatuses 120 are on the reception side. FIG. 9 also shows information indicating which delay time from any of the times t0 to t2 to any of the times t3 to t5 the delay time is.

[0065] The delay time management server 50 may update the delay time shown in FIG. 9 each time the delay time measurement unit 28 of the image transmission / reception apparatuses 120A to 120C measures the delay time. The delay time management server 50 may update the delay time shown in FIG. 9 to the latest delay time measured by the delay time measurement unit 28 each time a predetermined time elapses. The image transmission / reception apparatus 120 can read out the delay time stored in the delay time management server 50.

[0066] As described above, the delay time management server 50, which is one or more delay time management servers, is connected to the network 30. At least two image transmission / reception apparatuses 120, i.e., the first and second image transmission / reception apparatuses, are connected to the network 30 so as to communicate bidirectionally with each other. Any two of the image transmission / reception apparatuses 120A to 120C are the first and second image transmission / reception apparatuses.

[0067] The first image transmission / reception apparatus includes a first delay time measurement unit (delay time measurement unit 28) that measures a first delay time when the first image transmission / reception apparatus receives and displays image data generated by the second image transmission / reception apparatus and transmitted to the first image transmission / reception apparatus. The second image transmission / reception apparatus includes a second delay time measurement unit (delay time measurement unit 28) that measures a second delay time when the second image transmission / reception apparatus receives and displays image data generated by the first image transmission / reception apparatus and transmitted to the second image transmission / reception apparatus.

[0068] The delay time management server 50 stores the first delay time transmitted from the first image transmission / reception device via the network 30 and the second delay time transmitted from the second image transmission / reception device via the network 30.

[0069] According to the delay time management server 50 of one or more embodiments, when transmitting image data from the image transmission / reception device 120 operating as an image transmission device to the image transmission / reception device 120 operating as an image reception device via the network 30, it is possible to manage a more accurate delay time.

[0070] <Timing adjustment method at the time of receiving image data> As an example, in FIG. 10, consider a case where the image transmission / reception device 120B at the base B2 where the supervisor of the performance is located receives the image data transmitted from the image transmission / reception devices 120A and 120C while musical instruments are being played at each of the bases B1 to B3. The performance of the musical instrument is an example of a predetermined action performed by a person who is the subject. The subject performing the musical instrument is the performer.

[0071] The camera 11 of the image transmission / reception device 120A photographs the performer at the base B1, and the transmission / reception unit 121 transmits the image data to the image transmission / reception device 120B. The camera 11 of the image transmission / reception device 120C photographs the performer at the base B3, and the transmission / reception unit 121 transmits the image data to the image transmission / reception device 120B. The camera 11 of the image transmission / reception device 120B photographs the performer at the base B2, and the display device 22 displays a moving image. The display device 22 of the image transmission / reception device 120B may display the moving images generated by the image transmission / reception devices 120A to 120C simultaneously, or may display the moving images generated by the image transmission / reception devices 120A to 120C while switching them.

[0072] The image transmission / reception devices 120A to 120C are each provided with a timing instruction unit 29 for giving a signal to the performers located at the bases B1 to B3 to start playing. As shown in FIG. 11, the timing instruction unit 29 includes a control unit 291, a clock 292, an operation unit 293, a delay time correction value holding unit 294, and a display unit 295. The clock 292 may be a GNSS satellite radio clock 13 (or 23). The display unit 295 may be a liquid crystal panel or an LED.

[0073] When the timing instruction unit 29 sets the time to start playing to the control unit 291 by the operation unit 293, the control unit 291 monitors the time measured by the clock 292 and is configured to display a signal notifying the timing to start playing on the display unit 295. For example, assuming that the performance starts at 13:00:00 at each of the bases B1 to B3, since there is the above-mentioned delay time until the image data is transmitted from the bases B1 and B3 to the base B2 and displayed on the display device 22, the timings of all the performances at the bases B1 to B3 displayed at the base B2 do not match.

[0074] Therefore, based on the control by the control unit 122, the transmission / reception unit 121 reads the delay time shown in FIG. 9 from the delay time management server 50 and supplies it to the control unit 291 of the timing instruction unit 29. The control unit 291 causes the delay time correction value holding unit 294 to hold a delay time correction value based on the delay time. Here, the case where the delay time is directly used as the delay time correction value is taken as an example.

[0075] In the example shown in FIG. 10, the delay time correction value holding unit 294 of the timing instruction unit 29 provided in the image transmission / reception device 120A holds a delay time of 10.3 ms with the base B1 as the transmission side and the base B2 as the reception side. The delay time correction value holding unit 294 of the timing instruction unit 29 provided in the image transmission / reception device 120C holds a delay time of 10.2 ms with the base B3 as the transmission side and the base B2 as the reception side.

[0076] There is no need for the delay time correction value holding unit 294 of the timing instruction unit 29 provided in the image transmission / reception device 120B to hold the delay time correction value. In FIG. 10, since the delay time is not supplied from the transmission / reception unit 121 to the timing instruction unit 29, the area between the transmission / reception unit 121 and the timing instruction unit 29 is indicated by a dashed arrow line.

[0077] The control unit 291 of the timing instruction unit 29 provided in the image transmission / reception device 120A causes the display unit 295 to display a signal indicating the timing to start the performance at a time 10.3 ms before the time of 13:00:00, which is the time to start the performance. The control unit 291 of the timing instruction unit 29 provided in the image transmission / reception device 120C causes the display unit 295 to display a signal indicating the timing to start the performance at a time 10.2 ms before the time of 13:00:00, which is the time to start the performance.

[0078] The control unit 291 of the timing instruction unit 29 provided in the image transmission / reception device 120B causes the display unit 295 to display a signal indicating the timing to start the performance at the time of 13:00:00, which is the time to start the performance.

[0079] The performers at each of the bases B1 to B3 start the performance in accordance with the signal displayed on the display unit 295 of the timing instruction unit 29 at each of the bases B1 to B3. As a result, the image data displayed on the display device 22 at the base B2 is displayed in a state where the timings of all the performances at the bases B1 to B3 are synchronized.

[0080] The processes executed by the image transmission / reception devices 120A to 120C for synchronizing the reception timings of the image data will be described using the flowchart shown in FIG. 12. In FIG. 12, the control unit 122 of the image transmission / reception device 120B sets the image transmission / reception device 120B as the reference base in step S31. The control units 122 of the image transmission / reception devices 120A and 120C, which are the transmission-side bases, acquire the corresponding delay time from the delay time management server 50 in step S32.

[0081] The control units 291 of the image transmission / reception apparatuses 120A and 120C set the delay time as a delay time correction value in the delay time correction value holding unit 294 in step S33. Here, the delay time is directly used as the delay time correction value. The control units 291 of the image transmission / reception apparatuses 120A to 120C set the performance start time based on the operation input by the operation unit 293 in step S34.

[0082] The control units 122 of the image transmission / reception apparatuses 120A and 120C, which are the transmission-side bases, set the time that is earlier than the performance start time by the delay time correction value as the performance start timing in step S351. The control unit 122 of the image transmission / reception apparatus 120B, which is the reference base, sets the performance start time as the performance start timing in step S352.

[0083] The control units 122 of the image transmission / reception apparatuses 120A and 120C determine whether the current time has reached the performance start timing in step S361. If the current time has not reached the performance start timing (NO), the control unit 122 repeats the process of step S361. If the current time has reached the performance start timing (YES), the control unit 122 transfers the process to step S371.

[0084] The control unit 122 of the image transmission / reception apparatus 120B determines whether the current time has reached the performance start timing in step S362. If the current time has not reached the performance start timing (NO), the control unit 122 repeats the process of step S362. If the current time has reached the performance start timing (YES), the control unit 122 transfers the process to step S372.

[0085] The control units 122 of the image transmission / reception apparatuses 120A and 120C control to display a signal notifying the start of performance at the performance start timing on the display unit 295 in step S371, and end the process. The control unit 122 of the image transmission / reception apparatus 120B controls to display a signal notifying the start of performance at the performance start timing on the display unit 295 in step S372, and end the process.

[0086] As described above, in the image transmission system shown in FIG. 10, the image transmission device (image transmission / reception device 120A or 120C) arranged at the transmission-side base station includes a camera 11. The image transmission device transmits the image data generated by the camera 11 capturing a subject performing a predetermined operation to the image reception device (image transmission / reception device 120B) arranged at the reference base station via the network 30.

[0087] The timing adjustment method executed in the image transmission system measures the delay time when the image reception device displays the image data received on the display device 22. The timing adjustment method sets a delay time correction value based on the delay time in the timing instruction unit 29 arranged at the transmission-side base station. The timing adjustment method signals the subject to start the predetermined operation at a time that is the time indicated by the delay time correction value before the time when the subject is instructed to start the predetermined operation by the timing instruction unit 29.

[0088] According to the timing adjustment method of one or more embodiments, it is possible to adjust so as to match the timing when displaying the image data captured by the subject performing the predetermined operation individually performed at the transmission-side base station and the reference base station at the reference base station.

[0089] <Timing Adjustment Method of Frame at the Time of Image Data Reception> Even if the timing of the start of performance is adjusted according to the configuration shown in FIG. 10, the timing of the frames of the image data generated by the image transmission / reception devices 120A to 120C is deviated, and image noise may occur when switching and displaying the moving images generated by the image transmission / reception devices 120A to 120C.

[0090] FIG. 13 shows an image transmission / reception device 120 configured to correct the deviation of the timing of the frames of the image data, and a delay time management server 50 suitable for correcting the deviation of the timing of the frames of the image data. In FIG. 13, similar to FIG. 10, the image transmission / reception device 120B receives the image data transmitted from the image transmission / reception devices 120A and 120C.

[0091] The image transmission / reception device 120B has a vertical synchronization delay circuit 123. The vertical synchronization delay circuit 123 receives the image data transmitted from the image transmission / reception devices 120A and 120C received by the transmission / reception unit 121 and the image data output from the camera 11. It is assumed that the vertical synchronization signals of the image data transmitted from the image transmission / reception devices 120A and 120C are delayed with respect to the vertical synchronization signal of the image data output from the camera 11.

[0092] The vertical synchronization delay circuit 123 delays the vertical synchronization of the image data output from the camera 11 so as to reduce the deviation of the vertical synchronization signal of the image data transmitted from the image transmission / reception devices 120A and 120C with respect to the vertical synchronization signal of the image data output from the camera 11. When the deviation amounts of the vertical synchronization signals of the image data transmitted from the image transmission / reception devices 120A and 120C are different, the vertical synchronization delay circuit 123 delays the vertical synchronization of the image data with the smaller deviation amount to match the one with the larger deviation amount. The vertical synchronization delay circuit 123 matches the vertical synchronizations of the three pieces of image data by delaying the image data from the camera 11 according to the deviation amount.

[0093] The vertical synchronization delay circuit 123 supplies the transmission / reception unit 121 with a first delay time of the vertical synchronization signal of the image data transmitted from the image transmission / reception device 120A and a second delay time of the vertical synchronization signal of the image data transmitted from the image transmission / reception device 120C with respect to the vertical synchronization signal of the image data output from the camera 11. The vertical synchronization delay circuit 123 functions as a vertical synchronization deviation amount measurement unit that measures the first and second delay times.

[0094] Based on the control by the control unit 122, the transmission / reception unit 121 transmits the first and second delay times to the delay time management server 50 via the network 30. Assume that the first delay time is 3.2 ms and the second delay time is 0.3 ms.

[0095] As shown in FIG. 14, when the delay time management server 50 sets Site B1 as the transmission side and Site B2 as the reception side, it stores the delay time obtained by adding the first delay time of 3.2 ms to the delay time of 10.3 ms. Further, when the delay time management server 50 sets Site B3 as the transmission side and Site B2 as the reception side, it stores the delay time obtained by adding the second delay time of 0.3 ms to the delay time of 10.2 ms.

[0096] The delay time correction value holding units 294 of the timing instruction units 29 in the image transmission / reception apparatuses 120A and 120C respectively hold the delay times (10.3 + 3.2) ms and (10.2 + 0.3) ms as delay time correction values.

[0097] The control unit 291 of the timing instruction unit 29 in the image transmission / reception apparatus 120A causes the display unit 295 to display a signal notifying the timing to start playing at a time (10.3 + 3.2) ms before the time to start playing. The control unit 291 of the timing instruction unit 29 in the image transmission / reception apparatus 120C causes the display unit 295 to display a signal notifying the timing to start playing at a time (10.2 + 0.3) ms before the time to start playing.

[0098] As a result, not only is the image data displayed on the display device 22 at Site B2 displayed with the timings of all the performances at Sites B1 to B3 being synchronized, but also the deviation in the frame timings of the image data generated by the image transmission / reception apparatuses 120A to 120C is corrected.

[0099] As time passes, if the vertical synchronization signal of the image data transmitted from the image transmission / reception apparatus 120A or 120C is deviated from the vertical synchronization signal of the image data output from the camera 11, the vertical synchronization of the image data output from the camera 11 can be delayed by the vertical synchronization delay circuit 123 so as to reduce the deviation of the vertical synchronization signal.

[0100] Furthermore, the first delay time of the vertical synchronization signal of the image data transmitted from the image transmission / reception device 120A with respect to the vertical synchronization signal of the image data output from the camera 11, and the second delay time of the vertical synchronization signal of the image data transmitted from the image transmission / reception device 120C are transmitted to the delay time management server 50 and updated to the latest first and second delay times. Therefore, the timing of the frames of the image data generated by the image transmission / reception devices 120A to 120C is always corrected to an optimal state at predetermined intervals.

[0101] As described above, in the image transmission system shown in FIG. 13, an image transmission device (image transmission / reception device 120A or 120C) including a first camera (camera 11) and an image reception device including a second camera (camera 11) and a display device 22 are connected via a network 30. The image reception device (image transmission / reception device 120B) includes a delay time measurement unit 28 and a vertical synchronization delay circuit 123 that functions as a vertical synchronization deviation amount measurement unit.

[0102] The delay time measurement unit 28 measures the delay time when the image reception device receives the first image data generated by the first camera and transmitted by the image transmission device and displays it on the display device 22. The vertical synchronization delay circuit 123 measures the vertical synchronization deviation amount indicating the time when the vertical synchronization of the first image data and the second image data generated by the second camera and displayed on the display device 22 is deviated. The delay time management server 50 stores the delay time and the vertical synchronization deviation amount transmitted from the image reception device via the network 30.

[0103] In the image transmission system shown in FIG. 13, the first image data generated by the first camera included in the image transmission device arranged at the transmission-side base point by photographing a first subject performing a predetermined operation is transmitted to the image reception device arranged at the reference base point via the network 30. The timing adjustment method executed in the image transmission system measures the delay time when the image reception device displays the first image data received on the display device 22.

[0104] The timing adjustment method executed in the image transmission system measures the vertical synchronization deviation amount indicating the time when the vertical synchronization between the first image data and the second image data generated by the second camera provided in the image receiving device by photographing a second subject performing a predetermined operation is shifted. The timing adjustment method executed in the image transmission system sets, in the timing instruction unit 29 arranged at the transmission-side base, a delay time correction value based on the delay time obtained by adding the delay time and the vertical synchronization deviation amount.

[0105] The timing adjustment method executed in the image transmission system signals the first subject to start the predetermined operation at a time that is the time indicated by the delay time correction value before the time when the first subject is instructed by the timing instruction unit 29 to start the predetermined operation.

[0106] According to the delay time management server of one or more embodiments, it is possible to manage more accurate delay time and vertical synchronization deviation amount when transmitting image data from the image transmission / reception device 120 operating as an image transmission device to the image transmission / reception device 120 operating as an image receiving device via the network 30. According to the timing adjustment method of one or more embodiments, it is possible to adjust the timing including the timing of the frame of the image data so as to match the timing when displaying the image data photographed by the subject performing the predetermined operation separately performed at the transmission-side base and the reference base at the reference base.

[0107] The present invention is not limited to one or more of the embodiments described above, and various modifications can be made without departing from the gist of the present invention.

Explanation of Reference Numerals

[0108] 10, 10A~10C Image transmission device 11 Camera 12 Transmission unit 13,23 GNSS satellite radio clock 14,24 Clock generation unit 15,27 Image memory 16,25 Communication unit 20 Image receiving device 21 Receiver 22 Display device 26,113 Image processing circuit 28 Delay time measurement unit 29 Timing instruction unit 30 Network 40 Satellite 50 Delay time management server 111,122,221 Control unit 112 Imaging device 120A~120C Image transceiver 121 Transceiver 123 Vertical synchronization delay circuit (vertical synchronization deviation measurement unit) 150,270 Write / read control unit 222 Driving circuit 223 Liquid crystal panel

Claims

【Claim 1】 The first camera included in the image transmission device arranged at the transmission-side base captures a first subject performing an operation of playing a musical instrument, and transmits the generated first image data to the image reception device arranged at the reference base via a network. As a delay time when the image reception device displays the received first image data, the time when the pixel element included in the first camera generates pixel data of a reference pixel which is a pixel at a specific position in a specific row within a frame, and the time when the display device displays the reference pixel are used to calculate the difference, a vertical synchronization deviation amount indicating the time when the vertical synchronization between the first image data and the second image data generated by the second camera included in the image reception device capturing a second subject performing an operation of playing a musical instrument is deviated is measured, a delay time correction value based on the delay time obtained by adding the delay time and the vertical synchronization deviation amount is set in the timing instruction unit arranged at the transmission-side base, and a timing adjustment method is provided, in which the timing instruction unit signals the first subject to start the operation of playing the musical instrument at a time earlier by the time indicated by the delay time correction value from the time when the first subject is instructed to start the operation of playing the musical instrument.

Images