Image transmitting device, image receiving device, and delay time measuring device

Abstract

To provide an image transmitting device favorable to grasp an accurate delay time when transmitting image data from an image transmitting device to an image receiving device via a network.SOLUTION: A clock generation unit 14 generates a clock including time information based on an electric wave received from a satellite 40 for a whole earth navigation satellite system. A communication unit 16 transmits image data superposed with at least one of first to third clock numbers read from an image memory. The first clock number indicates a time when an imaging element 112 generates pixel data of a reference pixel in a frame. The second clock number indicates a time when a writing / reading control unit 150 writes the pixel data of the reference pixel in the image memory 15. The third clock number indicates a time when the communication unit 16 transmits the pixel data of the reference pixel.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present invention relates to an image transmitting device, an image receiving device, and a delay time measuring device. [Background technology]

[0002] Image data is transmitted from an image transmitting device located at at least one base to an image receiving device via a network, and the image transmitted from the image transmitting device is displayed on a display device provided in the image receiving device. In this case, with the practical application of the fifth generation mobile communication system (5G), image data can be transmitted with low latency, so the delay due to the transmission of image data is negligible. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Publication No. 11-88305 Summary of the Invention [Problem to be solved by the invention]

[0004] However, delays due to internal processing in the image transmitting device and the image receiving device cannot be ignored. In addition, when image transmitting devices are installed at multiple locations and image data is transmitted from the multiple image transmitting devices to the image receiving device, delay times due to internal processing in the multiple image transmitting devices vary, which can cause timing variations in images displayed on the display device, resulting in an uncomfortable feeling. There is a need to understand the delay times, including delays due to internal processing in the image transmitting device and the image receiving device, when transmitting image data from the image transmitting device to the image receiving device via a network.

[0005] The present invention aims to provide an image transmitting device, an image receiving device, and a delay time measuring device that are suitable for determining a more accurate delay time when transmitting image data from an image transmitting device to an image receiving device via a network. [Means for solving the problem]

[0006] The present invention provides a camera that includes a clock generating unit that generates a clock including time information based on radio waves received from a satellite for a global navigation satellite system, a camera that operates according to the clock and generates image data by an imaging element capturing an image of a subject, an image memory that operates according to the clock and writes and reads out the image data under the control of a write / read control unit, and a communication unit that transmits the image data read out from the image memory, and the camera is configured such that the imaging element captures an image within a frame. A pixel at a specific position in a specific row The present invention provides an image transmitting device in which the communication unit transmits the image data on which at least one of the first to third clock numbers has been superimposed by at least one of the following processes: a process of superimposing a first clock number in the clock indicating the time when pixel data of a reference pixel was generated on the image data; a process of superimposing a second clock number in the clock indicating the time when the write / read control unit wrote the pixel data of the reference pixel into the image memory on the image data; and a process of superimposing a third clock number in the clock indicating the time when the communication unit transmitted the pixel data of the reference pixel on the image data.

[0007] The present invention provides a method for transmitting an image data on a display device, the method comprising: receiving an image data on which at least one of a first clock number, a second clock number, and a third clock number is superimposed, the first clock number being a clock number in a first clock that includes time information based on radio waves received from a satellite for a global navigation satellite system and that is generated by a first clock generating unit included in the image data transmitting device; A pixel at a specific position in a specific rowa second clock generating unit that generates a second clock identical to the first clock, the second clock number being a clock number in the first clock and indicating a time when a first write / read control unit included in the image transmitting device wrote the pixel data of the reference pixel into a first image memory; the third clock number being a clock number in the first clock and indicating a time when a second communication unit included in the image transmitting device transmitted the pixel data of the reference pixel, the second clock number including time information based on radio waves received from a satellite for the global navigation satellite system; and a second clock generating unit that operates by the second clock and writes the image data under the control of a second write / read control unit. a display device that displays the image data read from the second image memory; and a delay time measurement unit that measures a delay time indicating a difference between one of the clock numbers among the first to third clock numbers, a fourth clock number in the second clock that indicates the time when the first communication unit received the pixel data of the reference pixel, a fifth clock number in the second clock that indicates the time when the second write / read control unit reads the pixel data of the reference pixel from the second image memory, and a sixth clock number in the second clock that indicates the time when the display device displayed the pixel data of the reference pixel.

[0008] The present invention relates to a method for transmitting an image data from an image transmitting device, in which at least one of a first clock number, a second clock number, and a third clock number is superimposed on the image data transmitted from the image transmitting device, the first clock number being a clock number in a first clock including time information based on radio waves received from a satellite for a global navigation satellite system, the first clock number being generated by a first clock generating unit provided in the image transmitting device, and the image capturing element of a camera provided in the image transmitting device being superimposed on the image data within a frame. A pixel at a specific position in a specific rowthe second clock number is a clock number in the first clock and indicates a time when a first write / read control unit included in the image transmitting device wrote the pixel data of the reference pixel into a first image memory; the third clock number is a clock number in the first clock and indicates a time when a first communication unit included in the image transmitting device transmitted the pixel data of the reference pixel; an image receiving device that receives the image data generates at least one of a fourth clock number, a fifth clock number, and a sixth clock number, the fourth clock number being a clock number in the second clock that is the same as the first clock and including time information based on radio waves received from a satellite for the global navigation satellite system; the fifth clock number is a clock number in the second clock and indicates the time when a second write / read control unit included in the image receiving device writes the image data into a second image memory and reads out the pixel data of the reference pixel from the second image memory; the sixth clock number is a clock number in the second clock and indicates the time when a display device included in the image receiving device displays the pixel data of the reference pixel read out from the second image memory; and a delay time measuring device is provided that measures the delay time when the image receiving device receives and displays the image data generated by the image transmitting device and transmitted to the image receiving device by calculating the difference between any one of the first to third clock numbers and any one of the fourth to sixth clock numbers. [Effects of the Invention]

[0009] The image transmitting device, image receiving device, and delay time measuring device of the present invention make it possible to grasp a more accurate delay time when transmitting image data from the image transmitting device to the image receiving device via a network. [Brief explanation of the drawings]

[0010] [Figure 1]FIG. 1 is a block diagram conceptually showing an image transmission system in which image data is transmitted from a plurality of image transmitting devices to an image receiving device via a network. [Figure 2] FIG. 2 is a block diagram showing an example configuration of an image transmitting device and an image receiving device configured to measure the delay time from the time when a camera in the image transmitting device starts capturing a moving image to the time when a display device in the image receiving device starts displaying the moving image. [Figure 3] FIG. 3 is a diagram showing the 1 PPS signal and the 10 MHz clock generated by the GNSS satellite wave clock 13 in FIG. 2, and the 148.5 MHz clock generated by the clock generating unit 14. In FIG. [Figure 4] FIG. 4 is a diagram showing the 1PPS signal and 10 MHz clock generated by the GNSS satellite wave clock 23 in FIG. 2, and the 148.5 MHz clock generated by the clock generating unit 24. [Figure 5] FIG. 5 is a flowchart showing the process executed by the delay time measurement unit 28 in FIG. [Figure 6] FIG. 6 is a flowchart showing the detailed processing of step S1 in FIG. [Figure 7] FIG. 7 is a flowchart showing the detailed processing of step S2 in FIG. [Figure 8] FIG. 8 is a block diagram showing a delay time management server that manages delay times when image data is transmitted and received bidirectionally between image transmitting and receiving devices located at three locations. [Figure 9] FIG. 9 is a diagram illustrating an example of delay times stored in the delay time management server illustrated in FIG. [Figure 10] FIG. 10 is a block diagram showing an image transmitting / receiving device that adjusts the timing of giving a signal to start a predetermined operation based on the delay time stored in the delay time management server. [Figure 11] FIG. 11 is a block diagram showing a specific example of the configuration of the timing instruction section 29 in FIG. [Figure 12]FIG. 12 is a flowchart showing the process executed by the image transmitting / receiving devices arranged at the three locations shown in FIG. 10 to synchronize the reception timing of image data. [Figure 13] FIG. 13 is a block diagram showing an image transmitting device configured to correct a timing shift between image data frames, and a delay time management server suitable for correcting the timing shift between image data frames. [Figure 14] FIG. 14 is a diagram showing an example of delay times stored in the delay time management server shown in FIG. DETAILED DESCRIPTION OF THE INVENTION

[0011] Hereinafter, an image transmitting device, an image receiving device, a delay time measuring device, a delay time management server, and a timing adjustment method according to one or more embodiments will be described with reference to the accompanying drawings.

[0012] First, using Figure 1, we will conceptually explain an image transmission system that transmits image data from multiple image transmitting devices to an image receiving device via a network. The image data is moving image data. Note that the image transmission system may also transmit audio data in addition to the image data. The process of transmitting and receiving audio data is not shown in the figure.

[0013] 1, image transmitting devices 10A to 10C are installed at three different locations. An image receiving device 20 is installed at a location different from the locations where the image transmitting devices 10A to 10C are installed. Each of the image transmitting devices 10A to 10C includes a camera 11 that captures an image of a subject, and a transmitting unit 12 that transmits image data generated by the camera 11 capturing an image of the subject.

[0014] The image data transmitted from the image transmitting devices 10A to 10C is transmitted via the network 30. The image data is transmitted to the image receiving device 20. The network 30 is typically the Internet. The image receiving device 20 includes a receiving unit 21 that receives image data and a display device 22 that displays a moving image based on the image data. The display device 22 may simultaneously display the moving images from the image transmitting devices 10A to 10C, or may switch between the moving images from the image transmitting devices 10A to 10C. An image transmitting device that does not specify which of the image transmitting devices 10A to 10C is referred to as the image transmitting device 10.

[0015] <Specific configuration examples of the image transmitting device, the image receiving device, and the delay time measuring device> Fig. 2 shows an example of the configuration of the image transmitting device 10 and the image receiving device 20 configured to measure the delay time from the time when the camera 11 in the image transmitting device 10 starts capturing a moving image to the time when the display device 22 in the image receiving device 20 starts displaying the moving image. The image transmitting device 10 and the image receiving device 20 shown in Fig. 2 have a suitable configuration for grasping the delay time including the delay due to their internal processing.

[0016] The delay time measurement unit 28 included in the image receiving device 20 constitutes a delay time measurement device of one or more embodiments, and measures a more accurate delay time, including delays due to internal processing in the image transmitting device 10 and the image receiving device 20, as described below. The image transmitting device 10 is an image transmitting device of one or more embodiments. The image receiving device 20 is an image receiving device of one or more embodiments.

[0017] 2, the image transmitting device 10 includes a camera 11, as well as a GNSS satellite wave clock 13, a clock generating unit 14, an image memory 15, and a communication unit 16. The camera 11 has a control unit 111, an image sensor 112, and an image processing circuit 113. The image memory 15 includes a write / read control unit 150. The communication unit 16 functions as the transmission unit 12 in FIG.

[0018] In addition to the display device 22, the image receiving device 20 includes a GNSS satellite wave clock 23, a clock generation unit 24, a communication unit 25, an image processing circuit 26, an image memory 27, and a delay time measurement unit 28. The display device 22 has a control unit 221, a drive circuit 222, and a liquid crystal panel 223. The image memory 27 includes a write / read control unit 270. The communication unit 25 functions as the receiving unit 21 in FIG. 1.

[0019] The GNSS satellite wave clock 13 receives radio waves from a satellite 40 for the Global Navigation Satellite System (GNSS) and outputs a 1PPS signal, which is a pulse at one-second intervals as shown in (a) of Fig. 3, and a 10 MHz clock (hereinafter referred to as 10 MHz clock) as shown in (b) of Fig. 3. An example of the GNSS is the Global Positioning System (GPS).

[0020] The clock generating unit 14 generates a 148.5 MHz clock (hereinafter referred to as the 148.5 MHz clock) shown in Fig. 3(c) based on the input 1 PPS signal and 10 MHz clock, and supplies it to the camera 11, image memory 15, and communication unit 16. For convenience of illustration, the 148.5 MHz clock shown in Fig. 3(c) is shown with one clock period significantly extended. Note that the clock generating unit 14 may generate clocks of other video synchronization frequencies other than 148.5 MHz, such as 297 MHz, 74.25 MHz, 27 MHz, etc.

[0021] The image transmitting device 10 may include a clock generating unit that generates a clock including time information based on radio waves received from a GNSS satellite 40. A 10 MHz clock may be supplied to the camera 11, image memory 15, and communication unit 16 to operate the camera 11, image memory 15, and communication unit 16 on the 10 MHz clock. In this case, the GNSS satellite wave clock 13 serves as the clock generating unit that generates the clock including time information. The frequency of the clock supplied to the camera 11, image memory 15, and communication unit 16 is not limited.

[0022] Assume that the GNSS satellite wave clock 13 generates a pulse of the 1PPS signal shown in (a) of FIG. 3 at time 0:00:00 and generates the next pulse at time 0:00:01. The clock generation unit 14 generates clock numbers 0 to 148.5×10 based on time 0:00:00. -6 -1 clock, clock number 0 to 148.5 x 10 based on time 0:00:01 -6 Like the -1 clock, it outputs a 148.5MHz clock that includes time information.

[0023] A control unit 111 of the camera 11 controls an electronic shutter in the image sensor 112, and the image sensor 112 captures an image of a subject. The control unit 111 may be a central processing unit provided in the camera 11. Image data generated by the image sensor 112 capturing an image of a subject is subjected to various types of image processing by an image processing circuit 113 and is supplied to an image memory 15.

[0024] The image processing in the image processing circuit 113 includes one or more of defective pixel interpolation processing, black level processing, white balance processing, demosaic processing, shading correction processing, and compression encoding processing. The image processing in the image processing circuit 113 includes at least compression encoding processing.

[0025] The control unit 111 controls the image sensor 112 to superimpose, as metadata, the clock number of the 148.5 MHz clock, which indicates the time t0 when the image sensor 112 released the electronic shutter for a predetermined reference pixel and generated pixel data, onto the blanking period of the image data. The reference pixel is preferably the first pixel in the first row within a frame. In other words, the clock number included in the image data output by the camera 11 indicates the time when the camera 11 started generating image data for each frame. As shown in FIG. 3(c), the clock number included in the image data output by the camera 11 and indicating the time t0 is, for example, clock number 0, Cn0.

[0026] The write / read control unit 150 writes image data into the image memory 15. At this time, the write / read control unit 150 writes image data into the image memory 15 in such a state that the clock number of the 148.5 MHz clock indicating the time t1 at which pixel data of the reference pixel of each frame was written into the image memory 15 is superimposed as metadata onto the blanking period of the image data. As shown in FIG. 3(c), it is assumed that the clock number indicating the time t1 contained in the image data written into the image memory 15 is Cn1. The image data written into the image memory 15 has the clock numbers Cn0 and Cn1 superimposed thereon.

[0027] The write / read control unit 150 reads image data stored in the image memory 15 and supplies the data to the communication unit 16. The communication unit 16 packetizes the image data read from the image memory 15 and transmits the packetized data to the image receiving device 20 via the network 30. At this time, the communication unit 16 packetizes the image data with the clock number of the 148.5 MHz clock indicating the time t2 at which pixel data of the reference pixel of each frame is transmitted superimposed as metadata on the blanking period of the image data. As shown in FIG. 3(c), the clock number indicating the time t2 at which pixel data of the reference pixel is transmitted is assumed to be Cn2. Consequently, the clock number Cn2 indicates the time t2 at which the pixel data of the reference pixel is transmitted.

[0028] Furthermore, communication unit 16 adds clock frequency information to image data and packetizes the image data to inform image receiving device 20 that camera 11, image memory 15, and communication unit 16 are operating at a 148.5 MHz clock. Packet data transmitted from image transmitting device 10 to image receiving device 20 via network 30 includes image data and clock frequency information, and clock numbers Cn0, Cn1, and Cn2 are superimposed on the image data.

[0029] The GNSS satellite wave clock 23 in the image receiving device 20 also receives radio waves from the satellite 40 in the same manner as the GNSS satellite wave clock 13, and outputs a 1 second pulse as shown in FIG. It outputs a PPS signal and a 10 MHz clock shown in FIG. 4(b).

[0030] 2, the satellite 40 that transmits radio waves received by the GNSS satellite radio clock 13 and the GNSS satellite radio clock 23 is a single common satellite 40. The satellite 40 that transmits radio waves received by the GNSS satellite radio clock 13 may be different from the satellite 40 that transmits radio waves received by the GNSS satellite radio clock 23. The GNSS satellite radio clocks 13 and 23 may receive radio waves from any satellite 40 among multiple satellites 40 operated by one global navigation satellite system.

[0031] The communication unit 25 receives packet data including image data and clock frequency information, supplies the clock frequency information to the clock generation unit 24, and supplies the image data to the image processing circuit 26. Since the clock frequency information indicating 148.5 MHz has been input to the clock generation unit 24, the clock generation unit 24 generates the 148.5 MHz clock shown in Fig. 4(c) based on the input 1 PPS signal and 10 MHz clock. The 148.5 MHz clock is supplied to the display device 22, the communication unit 25, the image memory 27, and the delay time measurement unit 28.

[0032] The 148.5 MHz clock shown in FIG. 4(c) is the same clock (same phase and same frequency) as the 148.5 MHz clock shown in FIG. 3(c).

[0033] The clock supplied by the clock generating unit 24 to the display device 22, the communication unit 25, the image memory 27, and the delay time measuring unit 28 is a 148.5 MHz clock including time information, as in the above.

[0034] 4(c), the communication unit 25 superimposes, as metadata, the clock number of the 148.5 MHz clock indicating the time t3 at which pixel data of the reference pixel of each frame was received from the network 30 onto the blanking period of the image data. The clock number indicating the time t3 at which pixel data of the reference pixel was received is assumed to be Cn3. Therefore, the image data output from the communication unit 25 has the clock numbers Cn0, Cn1, Cn2, and Cn3 superimposed thereon.

[0035] The image data output from the communication unit 25 is subjected to various image processing by the image processing circuit 26 and then supplied to the image memory 27. The image processing in the image memory 27 includes at least a compression decoding process. The image processing circuit 26 supplies the clock numbers Cn0, Cn1, Cn2, and Cn3 included in the image data to the delay time measurement unit 28.

[0036] The write / read control unit 270 writes the image data supplied to the image memory 27 into the image memory 27, and reads out the image data stored in the image memory 27 and supplies it to the drive circuit 222 of the display device 22. At this time, the write / read control unit 270 supplies the clock number of the 148.5 MHz clock indicating the time t4 at which the pixel data of the reference pixel of each frame is read out to the delay time measurement unit 28. The time t4 is also the time at which the pixel data of the reference pixel is supplied to the drive circuit 222. As shown in FIG. 4(c), the clock number indicating the time t4 at which the image data is read out from the image memory 27 is assumed to be Cn4.

[0037] The control unit 221 of the display device 22 controls the drive circuit 222 so as to display the image data supplied to the drive circuit 222 on the liquid crystal panel 223. The control unit 221 may be a central processing unit included in the display device 22. The control unit 221 supplies the clock number of the 148.5 MHz clock indicating the time t5 at which the pixel data of the reference pixel of each frame was displayed on the liquid crystal panel 223 to the delay time measurement unit 28. As shown in (c) of Figure 4, the clock number indicating the time t5 at which the pixel data of the reference pixel was displayed on the liquid crystal panel 223 is assumed to be Cn5.

[0038] It is not necessary for the display device 22 to operate on the 148.5 MHz clock. The 148.5 MHz clock is supplied to the display device 22 in order to supply the delay time measurement unit 28 with clock number Cn5 indicating time t5 in the 148.5 MHz clock.

[0039] When the display device 22 is a projection type display device rather than a direct-view type display device, the liquid crystal panel 223 is a liquid crystal display element for a projection type display device. The display device 22 may include a display panel (display element) other than a liquid crystal panel.

[0040] The delay time measurement unit 28 obtains the delay time between the image transmitting device 10 and the image receiving device 20 according to the processing shown in the flowcharts of FIGS. 5 to 7, and determines a delay time correction value for correcting the delay due to the delay time.

[0041] 5, in step S1, the delay time measurement unit 28 sets the shooting time in the image transmitting device 10, and in step S2, sets the display time in the image receiving device 20. FIG. 6 shows the detailed processing of step S1. In FIG. 6, in step S11, the delay time measurement unit 28 determines whether or not clock number Cn0 (i.e., time t0) exists. If clock number Cn0 exists (YES), in step S13, the delay time measurement unit 28 sets the time t0 indicated by clock number Cn0 as the shooting time.

[0042] If the clock number does not exist in step S11 (NO), the delay time measurement unit 28 determines whether or not the clock number Cn1 (i.e., time t1) exists in step S12. If the clock number Cn1 exists (YES), the delay time measurement unit 28 sets the time t1 indicated by the clock number Cn1 as the shooting time in step S14. If the clock number Cn1 does not exist (NO), the delay time measurement unit 28 sets the time t2 indicated by the clock number Cn2 as the shooting time in step S15.

[0043] Fig. 7 shows the detailed processing of step S2. In Fig. 7, the delay time measurement unit 28 determines whether or not clock number Cn5 (i.e., time t5) exists in step S21. If clock number Cn5 exists (YES), the delay time measurement unit 28 sets time t5 as the display time in step S23.

[0044] If clock number Cn5 does not exist in step S21 (NO), the delay time measurement unit 28 determines in step S22 whether clock number Cn4 (i.e., time t4) exists. If clock number Cn4 exists (YES), the delay time measurement unit 28 sets time t4 as the display time in step S24. If clock number Cn4 does not exist (NO), the delay time measurement unit 28 sets time t3 indicated by clock number Cn3 as the display time in step S25.

[0045] 6, the presence or absence of clock number Cn0 is determined because the camera 11 may not be configured to superimpose clock number Cn0, which indicates time t0 when the camera 11 starts generating image data for each frame, on the image data.The presence or absence of clock number Cn1 is determined because the camera 11 may not be configured to superimpose clock number Cn1, which indicates time t1 when the image memory 15 writes the image data, on the image data.

[0046] 7, the presence or absence of clock number Cn5 is determined because the display device 22 may not be configured to supply the delay time measurement unit 28 with clock number Cn5 indicating time t5 at which image data was displayed on the liquid crystal panel 223. The presence or absence of clock number Cn4 is determined because the display device 22 may not be configured to supply the delay time measurement unit 28 with clock number Cn4 indicating time t4 at which image memory 27 reads out image data. This is because there may be cases where the information has not been provided.

[0047] Returning to Fig. 5, in step S3, delay time measurement unit 28 calculates the delay time between image transmitting device 10 and image receiving device 20, which is the difference between the shooting time and the display time. In step S4, delay time measurement unit 28 determines whether the shooting time is time t2 and the display time is time t3. If the shooting time is time t2 and the display time is time t3, the time from time 2 to time t3 is the transmission time of the image data over network 30 itself. Therefore, it does not represent the delay time from when image transmitting device 10 starts generating image data to when image receiving device 20 starts displaying the image data.

[0048] Therefore, if the shooting time is time t2 and the display time is time t3 in step S4 (YES), the delay time measurement unit 28 determines in step S5 whether a pre-measured fixed correction value has been set in the delay time measurement unit 28. The fixed correction value is the sum of either or both of the average time from time t0 to time t2 and the average time from time t3 to time t5.

[0049] If a pre-measured fixed correction value is set in step S5 (YES), the delay time obtained in step S3 is added to the fixed correction value, and the delay time correction value is determined, and the process ends.

[0050] If the shooting time is not time t2 and the display time is not time t3 in step S4 (NO), or if a pre-measured fixed correction value is not set in step S5 (NO), the delay time measurement unit 28 determines the delay time obtained in step S3 as the delay time correction value in step S7, and ends the process.

[0051] As described above, the image transmitting device 10 includes the camera 11, the clock generating unit 14, the image memory 15, and the communication unit 16. The clock generating unit 14 generates a first clock (for example, a 148.5 MHz clock) that includes time information based on radio waves received from a GNSS satellite 40. The camera 11 operates according to the first clock, and generates image data by the image sensor 112 capturing an image of a subject. The image memory 15 operates according to the first clock, and writes and reads image data under the control of the write / read control unit 150.

[0052] The communication unit 16 transmits image data that has been read out from the image memory 15 and on which at least one of the first to third clock numbers has been superimposed. The first to third clock numbers are clock numbers Cn0 to Cn2.

[0053] The first clock number indicates the time when the image sensor 112 of the camera 11 generated pixel data for a reference pixel in a frame. If the camera 11 is configured to execute a process of superimposing the first clock number on image data, the image data including the first clock number is transmitted.

[0054] The second clock number indicates the time when the write / read control unit 150 wrote the pixel data of the reference pixel into the image memory 15. If the write / read control unit 150 is configured to execute a process of superimposing the second clock number on the image data, the image data including the second clock number is transmitted. The third clock number indicates the time when the communication unit 16 transmitted the pixel data of the reference pixel. If the communication unit 16 is configured to execute a process of superimposing the third clock number on the image data, the image data including the third clock number is transmitted.

[0055] The image receiving device 20 includes a communication unit 25 that receives image data on which at least one of the first clock number, the second clock number, and the third clock number is superimposed. If communication unit 25 is the first communication unit, communication unit 16 is the second communication unit. If communication unit 16 is the first communication unit, communication unit 25 is the second communication unit.

[0056] The first clock number is a clock number in the first clock generated by the clock generation unit 14 (first clock generation unit) and indicates the time when the image sensor 112 of the camera 11 generated pixel data for a reference pixel in a frame. The second clock number is a clock number in the first clock and indicates the time when the write / read control unit 150 (first write / read control unit) wrote the pixel data for the reference pixel into the image memory 15 (first image memory). The third clock number is a clock number in the first clock and indicates the time when the communication unit 16 (second communication unit) transmitted the pixel data for the reference pixel.

[0057] The image receiving device 20 further includes a clock generating unit 24 (second clock generating unit), an image memory 27 (second image memory), a display device 22, and a delay time measuring unit 28. The clock generating unit 24 generates a second clock that is the same as the first clock. The image memory 27 operates according to the second clock, and writes and reads image data under the control of a write / read control unit 270 (second write / read control unit). The display device 22 displays the image data read from the image memory 27.

[0058] The delay time measurement unit 28 measures the delay time indicating the difference between any one of the first to third clock numbers and any one of the fourth to sixth clock numbers. The fourth clock number is a clock number in the second clock and indicates the time when the communication unit 25 received the pixel data of the reference pixel. The fifth clock number is a clock number in the second clock and indicates the time when the write / read control unit 270 reads the pixel data of the reference pixel from the image memory 27. The sixth clock number is a clock number in the second clock and indicates the time when the display device 22 displayed the pixel data of the reference pixel.

[0059] When the delay time measurement unit 28 acquires the first clock number (clock number Cn0), it is preferable that the delay time measurement unit 28 measures the delay time using the first clock number. When the delay time measurement unit 28 is unable to acquire the first clock number and acquires the second clock number (clock number Cn1), it is preferable that the delay time measurement unit 28 measures the delay time using the second clock number. When the delay time measurement unit 28 acquires the sixth clock number (clock number Cn5), it is preferable that the delay time measurement unit 28 measures the delay time using the sixth clock number. When the delay time measurement unit 28 is unable to acquire the sixth clock number and acquires the fifth clock number (clock number Cn4), it is preferable that the delay time measurement unit 28 measures the delay time using the fifth clock number.

[0060] The delay time measurement unit 28 is a delay time measurement device according to one or more embodiments. As described above, the delay time measurement unit 28 calculates the difference between any one of the first to third clock numbers and any one of the fourth to sixth clock numbers. In this way, the delay time measurement unit 28 measures the delay time when the image receiving device 20 receives and displays image data generated by the image transmitting device 10 and transmitted to the image receiving device 20.

[0061] According to one or more embodiments of the image transmitting device, image receiving device, and delay time measuring device, it is possible to grasp a more accurate delay time when transmitting image data from the image transmitting device 10 to the image receiving device 20 via the network 30.

[0062] <Delay time management server> In FIG. 8, image transmitting / receiving devices 120A to 120C are installed at the locations B1 to B3, respectively. The image transmitting / receiving devices 120A to 120C communicate with each other bidirectionally via a network 30. An image transmitting / receiving device without specifying which of the image transmitting / receiving devices 120A to 120C is called an image transmitting / receiving device 120. The image transmitting / receiving device 120 is the same as the image transmitting device shown in FIG. The image receiving device 10 has the configuration of both the image receiving device 10 and the image receiving device 20 .

[0063] As shown in Fig. 8, the image transmitting and receiving device 120 generally includes a camera 11, a display device 22, a delay time measurement unit 28, a control unit 122, and a transmitting and receiving unit 121. Although not shown in Fig. 8, the image transmitting and receiving device 120 also includes a GNSS satellite wave clock 13 (or 23) and a clock generation unit 14 (or 24), and a 148.5 MHz clock is supplied to the camera 11 and the display device 22. The transmitting and receiving unit 121 corresponds to the communication units 16 and 25. The control unit 122 corresponds to the control units 111 and 221.

[0064] The image transmitting device 10 shown in Fig. 2 is a detailed description of the configuration when the image transmitting / receiving device 120 shown in Fig. 8 is operating as an image transmitting device. The image receiving device 20 shown in Fig. 2 is a detailed description of the configuration when the image transmitting / receiving device 120 shown in Fig. 8 is operating as an image receiving device.

[0065] A delay time management server 50 is connected to the network 30. The delay time management server 50 communicates bidirectionally with the image transmitting and receiving devices 120A to 120C. As shown in Fig. 9, the delay time management server 50 stores delay times when the image transmitting and receiving devices 120A to 120C located at the bases B1 to B3 act as the transmitting side and the other image transmitting and receiving devices 120 act as the receiving side. In Fig. 9, the delay times are listed together with information indicating from which time among times t0 to t2 the delay time extends to which time among times t3 to t5.

[0066] The delay time management server 50 may update the delay time shown in Fig. 9 every time the delay time measurement unit 28 of the image transmitting and receiving devices 120A to 120C measures the delay time. The delay time management server 50 may update the delay time shown in Fig. 9 to the latest delay time measured by the delay time measurement unit 28 every time a predetermined time has elapsed. The image transmitting and receiving devices 120 can read out the delay time stored in the delay time management server 50.

[0067] As described above, one or more delay time management servers 50 are connected to the network 30. At least two image transmitting / receiving devices 120, namely, first and second image transmitting / receiving devices, are connected to the network 30 so as to communicate with each other bidirectionally. Any two image transmitting / receiving devices 120 among the image transmitting / receiving devices 120A to 120C are the first and second image transmitting / receiving devices.

[0068] The first image transmitting / receiving device includes a first delay time measurement unit (delay time measurement unit 28) that measures a first delay time when the first image transmitting / receiving device receives and displays image data generated by the second image transmitting / receiving device and transmitted to the first image transmitting / receiving device. The second image transmitting / receiving device includes a second delay time measurement unit (delay time measurement unit 28) that measures a second delay time when the second image transmitting / receiving device receives and displays image data generated by the first image transmitting / receiving device and transmitted to the second image transmitting / receiving device.

[0069] The delay time management server 50 stores a first delay time transmitted from a first image transmitting / receiving device via the network 30 and a second delay time transmitted from a second image transmitting / receiving device via the network 30.

[0070] According to one or more embodiments of the delay time management server 50, it is possible to more accurately manage the delay time when transmitting image data from an image transmitting / receiving device 120 operating as an image transmitting device to an image transmitting / receiving device 120 operating as an image receiving device via a network 30.

[0071] <Timing adjustment method when receiving image data> As an example, consider the case in Fig. 10 where musical instruments are played at each of the locations B1 to B3, and image transmitting / receiving device 120B at location B2, where the performance supervisor is located, receives image data transmitted from image transmitting / receiving devices 120A and 120C. Playing a musical instrument is an example of a predetermined action performed by a person who is a subject. The subject playing the musical instrument is a performer.

[0072] The camera 11 of the image transmitting / receiving device 120A captures an image of the performer at location B1, and the transmitting / receiving unit 121 transmits the image data to the image transmitting / receiving device 120B. The camera 11 of the image transmitting / receiving device 120C captures an image of the performer at location B3, and the transmitting / receiving unit 121 transmits the image data to the image transmitting / receiving device 120B. The camera 11 of the image transmitting / receiving device 120B captures an image of the performer at location B2, and the display device 22 displays a video. The display device 22 of the image transmitting / receiving device 120B may simultaneously display the video generated by the image transmitting / receiving devices 120A to 120C, or may switch between the video images generated by the image transmitting / receiving devices 120A to 120C.

[0073] The image transmitting and receiving devices 120A to 120C each include a timing instruction unit 29 for signaling the start of performance to the performers located at the respective locations B1 to B3. As shown in Fig. 11, the timing instruction unit 29 includes a control unit 291, a clock 292, an operation unit 293, a delay time correction value storage unit 294, and a display unit 295. The clock 292 may be the GNSS satellite wave clock 13 (or 23). The display unit 295 may be a liquid crystal panel or an LED.

[0074] The timing instruction unit 29 is configured so that when the time to start performance is set in the control unit 291 by the operation unit 293, the control unit 291 monitors the time measured by the clock 292 and displays a signal informing the timing to start performance on the display unit 295. For example, if performance starts at each of the locations B1 to B3 at 13:00:00, there is the delay time described above until the image data is transmitted from the locations B1 and B3 to the location B2 and displayed on the display device 22, and therefore the timing of the performances of all the locations B1 to B3 displayed at the location B2 will not match.

[0075] 9 from the delay time management server 50 under the control of the control unit 122, and supplies the delay time to the control unit 291 of the timing instruction unit 29. The control unit 291 stores a delay time correction value based on the delay time in the delay time correction value storage unit 294. Here, the case where the delay time is used as the delay time correction value as it is will be taken as an example.

[0076] 10, the delay time correction value storage unit 294 of the timing instruction unit 29 included in the image transmitting and receiving device 120A stores a delay time of 10.3 ms between location B1 as the transmitting side and location B2 as the receiving side. The delay time correction value storage unit 294 of the timing instruction unit 29 included in the image transmitting and receiving device 120C stores a delay time of 10.2 ms between location B3 as the transmitting side and location B2 as the receiving side.

[0077] There is no need to store a delay time correction value in the delay time correction value storage unit 294 of the timing instruction unit 29 provided in the image transmission / reception device 120B. In Fig. 10, since a delay time is not supplied from the transmission / reception unit 121 to the timing instruction unit 29, the connection between the transmission / reception unit 121 and the timing instruction unit 29 is indicated by a dashed arrow.

[0078] The control unit 291 of the timing instruction unit 29 included in the image transmitting and receiving device 120A displays on the display unit 295 a cue informing the user of the timing to start playing at a delay time of 10.3 ms before 13:00:00, which is the time when the playing is to start. The control unit 291 of the timing instruction unit 29 included in the image transmitting and receiving device 120C displays on the display unit 295 a cue informing the user of the timing to start playing at a delay time of 10.2 ms before 13:00:00, which is the time when the playing is to start. Display at 295.

[0079] The control unit 291 of the timing instruction unit 29 included in the image transmitting / receiving device 120B displays on the display unit 295 a cue informing the user of the timing to start playing at 13:00:00, which is the time when the playing is to start.

[0080] The performers at each of the locations B1 to B3 start playing in sync with the cues displayed on the display units 295 of the timing indicators 29 at each of the locations B1 to B3. As a result, the image data displayed on the display device 22 at the location B2 is displayed in a state where the performance timings of all the performers at the locations B1 to B3 are synchronized.

[0081] The process executed by the image transmitting and receiving devices 120A to 120C to synchronize the reception timing of image data will be described using the flowchart shown in Fig. 12. In Fig. 12, the control unit 122 of the image transmitting and receiving device 120B sets the image transmitting and receiving device 120B as a reference location in step S31. The control units 122 of the image transmitting and receiving devices 120A and 120C, which are transmission-side locations, acquire the corresponding delay times from the delay time management server 50 in step S32.

[0082] In step S33, the control unit 291 of the image transmitting and receiving devices 120A and 120C sets the delay time as a delay time correction value in the delay time correction value storage unit 294. Here, the delay time is used as the delay time correction value as is. In step S34, the control unit 291 of the image transmitting and receiving devices 120A to 120C sets the performance start time based on an operation input from the operation unit 293.

[0083] In step S351, the control units 122 of the image transmitting and receiving devices 120A and 120C, which are transmission-side bases, set the performance start timing to a time earlier than the performance start time by the delay time correction value. In step S352, the control unit 122 of the image transmitting and receiving device 120B, which is the reference base, sets the performance start timing to the performance start time.

[0084] In step S361, the control unit 122 of the image transmitting / receiving devices 120A and 120C determines whether the current time has reached the performance start timing. If the current time has not reached the performance start timing (NO), the control unit 122 repeats the process of step S361. If the current time has reached the performance start timing (YES), the control unit 122 proceeds to step S371.

[0085] In step S362, the control unit 122 of the image transmitting / receiving device 120B determines whether the current time has reached the performance start timing. If the current time has not reached the performance start timing (NO), the control unit 122 repeats the process of step S362. If the current time has reached the performance start timing (YES), the control unit 122 proceeds to step S372.

[0086] In step S371, the control units 122 of the image transmitting and receiving devices 120A and 120C control the display unit 295 to display a cue notifying the start of performance at the performance start timing, and then terminate the processing. In step S372, the control unit 122 of the image transmitting and receiving device 120B controls the display unit 295 to display a cue notifying the start of performance at the performance start timing, and then terminates the processing.

[0087] 10, the image transmitting device (image transmitting / receiving device 120A or 120C) located at the transmitting side base includes the camera 11. The image transmitting device transmits image data generated by the camera 11 capturing an image of a subject performing a predetermined action to the image receiving device (image transmitting / receiving device 120B) located at the reference base via the network 30.

[0088] The timing adjustment method executed in the image transmission system measures the delay time when the image receiving device displays the received image data on the display device 22. The timing adjustment method sets a delay time correction value based on the delay time in the timing instruction unit 29 located at the transmitting side base. The timing adjustment method causes the timing instruction unit 29 to signal the subject to notify the subject of the timing to start a predetermined action at a time indicated by the delay time correction value before the time the subject is instructed to start the predetermined action.

[0089] According to one or more embodiments of the timing adjustment method, it is possible to adjust the timing at which image data of a subject performing a predetermined action, which is performed separately at the transmitting location and the reference location, is displayed at the reference location to match.

[0090] <Frame timing adjustment method when receiving image data> Even if the timing of the start of performance is adjusted using the configuration shown in FIG. 10, the timing of the frames of image data generated by the image transmitting / receiving devices 120A to 120C is misaligned, and image noise may occur when switching between and displaying the moving images generated by the image transmitting / receiving devices 120A to 120C.

[0091] Fig. 13 shows an image transmitting / receiving device 120 configured to correct a timing shift between image data frames, and a delay time management server 50 suitable for correcting the timing shift between image data frames. In Fig. 13, similar to Fig. 10, the image transmitting / receiving device 120B receives image data transmitted from the image transmitting / receiving devices 120A and 120C.

[0092] The image transmitting / receiving device 120B has a vertical synchronization delay circuit 123. The vertical synchronization delay circuit 123 receives the image data transmitted from the image transmitting / receiving devices 120A and 120C and received by the transmitting / receiving unit 121, as well as the image data output from the camera 11. It is assumed that the vertical synchronization signals of the image data transmitted from the image transmitting / receiving devices 120A and 120C are delayed with respect to the vertical synchronization signal of the image data output from the camera 11.

[0093] The vertical synchronization delay circuit 123 delays the vertical synchronization of the image data output from the camera 11 so as to reduce the deviation of the vertical synchronization signal of the image data transmitted from the image transmitting and receiving devices 120A and 120C from the vertical synchronization signal of the image data output from the camera 11. If the deviation amounts of the vertical synchronization signals of the image data transmitted from the image transmitting and receiving devices 120A and 120C are different, the vertical synchronization delay circuit 123 delays the vertical synchronization of the image data with the smaller deviation amount to match the one with the larger deviation amount. The vertical synchronization delay circuit 123 synchronizes the vertical synchronization of the three pieces of image data by delaying the image data from the camera 11 to match the deviation amount.

[0094] The vertical synchronization delay circuit 123 supplies the transmitter / receiver unit 121 with a first delay time of the vertical synchronization signal of the image data transmitted from the image transmitter / receiver 120A and a second delay time of the vertical synchronization signal of the image data transmitted from the image transmitter / receiver 120C, relative to the vertical synchronization signal of the image data output from the camera 11. The vertical synchronization delay circuit 123 functions as a vertical synchronization deviation amount measuring unit that measures the first and second delay times.

[0095] The transmitting / receiving unit 121 transmits the first and second delay times to the delay time management server 50 via the network 30 under the control of the control unit 122. It is assumed that the first delay time is 3.2 ms and the second delay time is 0.3 ms.

[0096] As shown in FIG. 14, the delay time management server 50 is configured to have the site B1 as the sending side and the site B2 as the receiving side. The delay time management server 50 stores a delay time obtained by adding a first delay time of 3.2 ms to a delay time of 10.3 ms when site B3 is the sending side and site B2 is the receiving side. The delay time management server 50 also stores a delay time obtained by adding a second delay time of 0.3 ms to a delay time of 10.2 ms when site B3 is the sending side and site B2 is the receiving side.

[0097] The delay time correction value holding unit 294 of the timing instruction unit 29 in the image transmitting / receiving devices 120A and 120C holds the delay times (10.3+3.2) ms and (10.2+0.3) ms as delay time correction values, respectively.

[0098] The control unit 291 of the timing instruction unit 29 in the image transmitting and receiving device 120A displays on the display unit 295 a cue informing the user of the timing to start performance at a time that is (10.3+3.2) ms before the time when performance is to start, which is the delay time correction value. The control unit 291 of the timing instruction unit 29 in the image transmitting and receiving device 120C displays on the display unit 295 a cue informing the user of the timing to start performance at a time that is (10.2+0.3) ms before the time when performance is to start, which is the delay time correction value.

[0099] As a result, the image data displayed on the display device 22 at location B2 is not only displayed in a state where the timing of the performances at all locations B1 to B3 is synchronized, but also the timing discrepancies between the frames of the image data generated by the image transmitting / receiving devices 120A to 120C are corrected.

[0100] If, over time, the vertical synchronization signal of the image data transmitted from the image transmitting / receiving device 120A or 120C deviates from the vertical synchronization signal of the image data output from the camera 11, the vertical synchronization of the image data output from the camera 11 can be delayed by the vertical synchronization delay circuit 123 to reduce the deviation of the vertical synchronization signal.

[0101] Furthermore, a first delay time of the vertical synchronization signal of the image data transmitted from the image transmitting / receiving device 120A and a second delay time of the vertical synchronization signal of the image data transmitted from the image transmitting / receiving device 120C relative to the vertical synchronization signal of the image data output from the camera 11 are transmitted to the delay time management server 50, and are updated to the latest first and second delay times. Therefore, the timing of the frames of the image data generated by the image transmitting / receiving devices 120A to 120C is always corrected to an optimal state at predetermined time intervals.

[0102] 13, an image transmitting device (image transmitting / receiving device 120A or 120C) equipped with a first camera (camera 11) and an image receiving device equipped with a second camera (camera 11) and display device 22 are connected via network 30. The image receiving device (image transmitting / receiving device 120B) includes a delay time measuring unit 28 and a vertical synchronization delay circuit 123 that functions as a vertical synchronization deviation amount measuring unit.

[0103] The delay time measurement unit 28 measures the delay time when the first image data generated by the first camera and transmitted by the image transmitting device to the image receiving device is received by the image receiving device and displayed on the display device 22. The vertical synchronization delay circuit 123 measures the vertical synchronization deviation amount indicating the time by which vertical synchronization is deviated between the first image data and the second image data generated by the second camera and displayed on the display device 22. The delay time management server 50 stores the delay time and the vertical synchronization deviation amount transmitted from the image receiving device via the network 30.

[0104] 13, a first camera provided in an image transmitting device located at a transmitting side site captures an image of a first subject performing a predetermined action, and the first image data is generated by the first camera and transmitted to an image receiving device located at a reference site via a network 30. A timing adjustment method executed in the image transmission system measures a delay time when the first image data received by the image receiving device is displayed on a display device 22.

[0105] The timing adjustment method performed in the image transmission system measures a vertical synchronization deviation amount indicating a time by which vertical synchronization is deviated between first image data and second image data generated by a second camera provided in the image receiving device by capturing an image of a second subject performing a predetermined action. The timing adjustment method performed in the image transmission system sets a delay time correction value based on a delay time obtained by adding the delay time and the vertical synchronization deviation amount in a timing instruction unit 29 arranged at the transmitting side base.

[0106] The timing adjustment method executed in the image transmission system is such that a timing instruction unit 29 signals the first subject to inform the timing of starting the predetermined action at a time indicated by a delay time correction value before the time when the first subject is instructed to start the predetermined action.

[0107] According to the delay time management server of one or more embodiments, it is possible to more accurately manage the delay time and the amount of vertical synchronization deviation when transmitting image data from the image transmitting / receiving device 120 operating as an image transmitting device to the image transmitting / receiving device 120 operating as an image receiving device via the network 30. According to the timing adjustment method of one or more embodiments, it is possible to adjust the timing, including the timing of the frames of the image data, so that they coincide when image data obtained by capturing a subject performing a predetermined action that is performed separately at the transmitting site and the reference site is displayed at the reference site.

[0108] The present invention is not limited to one or more of the above-described embodiments, and various modifications can be made without departing from the spirit of the present invention. [Explanation of symbols]

[0109] 10, 10A~10C Image transmitting device 11 Camera 12 Transmitter 13,23 GNSS satellite radio clock 14,24 Clock generation unit 15,27 Image memory 16,25 Communications Department 20 Image receiving device 21 Receiving unit 22 Display device 26,113 Image processing circuit 28 Delay time measurement unit 29 Timing indicator 30 Network 40 satellites 50 Delay Time Management Server 111,122,221 Control unit 112 Image sensor 120A~120C Image transmission / reception device 121 Transmitter / Receiver 123 Vertical sync delay circuit (vertical sync deviation measurement unit) 150,270 Write / read control section 222 Drive circuit 223 LCD panel

Claims

[Claim 1] a clock generating unit that generates a clock including time information based on radio waves received from a satellite for a global navigation satellite system; a camera that operates according to the clock and generates image data by capturing an image of a subject with an image sensor; an image memory that operates according to the clock and writes and reads the image data under the control of a write / read control unit; a communication unit that transmits the image data read from the image memory; Equipped with The communication unit transmits the image data on which at least one of the first to third clock numbers has been superimposed by at least one of the following processes: a process in which the camera superimposes, on the image data, a first clock number in the clock indicating the time when the image sensor released an electronic shutter for a reference pixel, which is a pixel at a specific position in a specific row within a frame, to generate pixel data; a process in which the write / read control unit superimposes, on the image data, a second clock number in the clock indicating the time when the write / read control unit wrote the pixel data of the reference pixel into the image memory; and a process in which the communication unit superimposes, on the image data, a third clock number in the clock indicating the time when the communication unit transmitted the pixel data of the reference pixel. Image transmission device.

Images