COMMUNICATION DEVICE, COMMUNICATION DEVICE CONTROL METHOD, AND PROGRAM

Abstract

To provide a communication device, a control method and a program of the communication device which transmits a background image to a target device even when a data amount of a foreground image increases.SOLUTION: In a sensor system 110-2, a transmission unit 207 of a camera adapter 112-2 generates one or more image packets including a foreground image or a background image from a photographed image, receives the one or more image packets including the foreground image or background image from another first communication device, and transmits the generated image packets and the received image packets to another second communication device according to a prescribed priority sequence. The prescribed priority sequence is the priority sequence on the basis of whether or not the image packet is the generated image packet or received image packet and includes the foreground image or background image.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present invention relates to a communication device, a control method for a communication device, and a program. [Background technology]

[0002] Recently, technology for time synchronization between multiple devices connected via a network has been used in a wide range of fields. For example, there is a virtual viewpoint image generation system in which multiple cameras installed in different locations capture images from multiple viewpoints in a synchronized manner, and virtual viewpoint content is generated using the multiple viewpoint images obtained by the capture (Patent Document 1). In such a system, highly accurate synchronization of the capture timing is required to generate high-quality virtual viewpoint images.

[0003] The Precision Time Protocol (PTP), an IEEE1588 standard, is known as a protocol for time synchronization using a network. In PTP, a master device that has accurate time transmits a PTP packet containing that time information, and a slave device that receives the PTP packet performs time synchronization. In a virtual viewpoint image generation system that uses PTP, multiple cameras capture images in synchronization and transmit the image data obtained via a network to an image processing device that generates a virtual viewpoint image. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Publication No. 2017-211827 Summary of the Invention [Problem to be solved by the invention]

[0005] Image data is classified into image types: foreground images (images cut out from the foreground) and background images (images of the background). Each of the multiple devices constituting the virtual viewpoint image generation system generates an image of one of the image types, relays the images, and transmits them to the image processing device. Here, each device prioritizes the processing of relayed packets to prevent buffer overflow in each device. Background images must be reliably transmitted to generate high-quality virtual viewpoint content. However, if the number of devices generating background images is smaller than the number of devices generating foreground images, and the amount of data of relayed foreground images increases, a situation may arise in which the background images do not reach the image processing device. As a result, high-quality virtual viewpoint video cannot be generated.

[0006] The present invention has been made in consideration of the above-mentioned problems, and aims to provide a technology for reliably transmitting a background image to a target device even when the data volume of the foreground image increases. [Means for solving the problem]

[0007] As one means for achieving the above object, a communication device of the present invention has the following configuration: the communication device includes a generating means for generating one or more image packets containing a foreground image or a background image from a captured image, a receiving means for receiving one or more image packets containing a foreground image or a background image from a first other communication device, and a transmitting means for transmitting the image packets generated by the generating means and the image packets received by the receiving means to a second other communication device in accordance with a predetermined priority, the predetermined priority being based on whether the image packet is an image packet generated by the generating means or an image packet received by the receiving means, and whether it contains a foreground image or a background image. The receiving means has a receiving buffer for temporarily storing received image packets, and when the total amount of image packets stored in the receiving buffer exceeds a second threshold and is equal to or less than a third threshold that is greater than the second threshold, and when image packets containing background images are stored in the receiving buffer, the generating means stops generating packets containing background images. do. [Effects of the Invention]

[0008] According to the present invention, even if the amount of data of the foreground image increases, the background image can be reliably transmitted to the destination device. [Brief explanation of the drawings]

[0009] [Figure 1] 1 shows an example of the configuration of a synchronous imaging system. [Figure 2] FIG. 2 is a block diagram showing an example of the configuration of a camera adapter. [Figure 3] FIG. 2 is a block diagram showing an example of the configuration of a transmission unit. [Figure 4] 10 is a flowchart of a process executed by a transmission control unit of the camera adapter. [Figure 5] 10 is a flowchart of an image packet generation process performed by a transmission control unit of a camera adapter. [Figure 6] 10 is a flowchart of a process executed by a forward unit of the camera adapter. [Figure 7] 10 is a flowchart of a process executed by an arbitration unit of the camera adapter. [Figure 8] (a) shows an exemplary reference signal configuration, and (b) shows an exemplary image packet configuration. [Figure 9] 1 shows an example of transmitting only packets of a foreground image in the prior art. [Figure 10] 1 shows an example of packet transmission of foreground and background images under normal conditions in the prior art. [Figure 11] 10 shows an example of transmission of packets of foreground images and background images when the total data amount of the foreground images increases in the prior art. DETAILED DESCRIPTION OF THE INVENTION

[0010] Hereinafter, with reference to the accompanying drawings, a detailed description will be given of an embodiment for carrying out the present invention. Note that the embodiment described below is an example of a means for realizing the present invention, and should be appropriately modified or changed depending on the configuration of the device to which the present invention is applied and various conditions. The present invention is not limited to the following embodiment. Furthermore, not all of the combinations of features described in the present embodiment are necessarily essential to the solution of the present invention.

[0011] [Configuration of synchronized imaging system] Fig. 1 shows an example of the configuration of a synchronized shooting system according to this embodiment. The synchronized shooting system 100 shown in Fig. 1 is a virtual viewpoint image generation system that generates virtual viewpoint content using multiple viewpoint images. The synchronized shooting system 100 includes sensor systems 110-1 to 110-n, a hub 130, a control device 140, a time server 150, an image processing device 160, and a user device 170.

[0012] 1, sensor systems 110-1 to 110-n (n>1) each include cameras 111-1 to 111-n and camera adapters 112-1 to 112-n, connected via connections 113-1 to 113-n, respectively. In the following, unless otherwise specified, sensor systems 110-1 to 110-n will be collectively referred to as sensor system 110. Similarly, cameras 111-1 to 111-n will be collectively referred to as cameras 111, camera adapters 112-1 to 112-n will be collectively referred to as camera adapters 112, and connections 113-1 to 113-n will be collectively referred to as connections 113.

[0013] In the sensor system 110, an image (image data) captured by the camera 111 is converted into image data by the camera adapter 112 undergoing image processing, which will be described later. 1 shows only one connection 113, the number of connections is not limited to this. For example, multiple signal lines for controlling the camera 111 and transmitting image data captured by the camera 111 may be connected between the camera 111 and the camera adapter 112.

[0014] The number of sensor systems 110 in the synchronized imaging system 100 is not limited to a specific number as long as it is plural. Furthermore, the sensor systems 110 do not need to have the same configuration; for example, each may be composed of a different model of device. Furthermore, the sensor system 110 may include an audio device such as a microphone, a camera platform for controlling the camera orientation, etc. Furthermore, the sensor system 110 may be composed of one camera 111 and multiple camera adapters 112. Furthermore, the camera 111 and the camera adapter 112 may be configured as an integrated unit. Furthermore, at least some of the functions of the camera adapter 112 may be provided by the image processing device 160.

[0015] The sensor systems 110-1 to 110-n are connected in a daisy chain via transmission paths 120-1 to 120-n. This has the effect of reducing the number of connecting cables and saving labor for wiring work when the volume of image data increases with the increase in resolution of captured images to 4K, 8K, etc. and the increase in frame rate. In the following description, the transmission paths 120-1 to 120-n may be collectively referred to as the transmission path 120.

[0016] In this embodiment, unless otherwise specified, the term "image" will be described as including the concepts of moving images and still images. That is, the synchronized photography system 100 of this embodiment is capable of processing both still images and moving images.

[0017] In this embodiment, PTP (Precision Time Protocol) is used as a protocol for time synchronization. As described above, in PTP, a master device having accurate time transmits a PTP packet containing information about the time (time information), and a slave device that receives the PTP packet performs time synchronization. In PTP, the master device and slave device each have a clock to write the accurate time into the PTP packet or to accurately know the time at which the PTP packet was received. Furthermore, PTP requires that a hub located between the master device and slave device be equipped with a transparent clock (TC) function. The TC function is a function for adding information such as the time it takes to relay a PTP packet when relaying the packet. This function allows a slave device to accurately obtain the delay time between itself and the master device, enabling highly accurate time adjustment.

[0018] 1, the master device in PTP corresponds to the time server 150, the slave device corresponds to the camera adapter 112 in the sensor system 110, and the hub corresponds to the hub 130. In this embodiment, the hub 130 is further configured to be able to send and receive other packets. The PTP packets transmitted from the time server 150 are transmitted to the sensor system 110 via the transmission path 120. The camera adapter 112 in the sensor system 110 receives the PTP packets via the transmission path 120 and outputs a Genlock signal (corresponding to an image capture synchronization signal) to the camera 111 based on the time information contained in the PTP packets. The camera 111 captures an image in synchronization with the received Genlock signal.

[0019] In this embodiment, the camera adapter 112 is assumed to have both a TC function and an OC (Ordinary Clock) function in PTP. For example, when transferring a received PTP packet, the camera adapter 112-1 measures the residence time within the camera adapter 112-1, adds the measured time to the PTP packet, and then transfers it to the subsequent camera adapter 112-2. The camera adapter 112-1 also synchronizes with the time of the time server 150 based on the received PTP packet. All of the camera adapters 110-1 to 110-n in the synchronized photography system 100 have both the TC function and the OC function, thereby enabling highly accurate time synchronization with the time server 150. The camera adapter 112, which is time-synchronized based on the time server 150, can synchronize the photography timing of the camera 111.

[0020] Note that, since the synchronized imaging system 100 may fail if a malfunction occurs in the time server 150, redundancy may be achieved by providing multiple time servers 150. In this case, it is necessary to synchronize the times of the multiple time servers 150 using, for example, a global positioning system (GPS) so that the times are synchronized.

[0021] The control device 140 controls the operation status and parameter settings of each block constituting the synchronized imaging system 100 via a network. The control device 140 also issues instructions to the sensor system 110 to capture images (imaging instructions) and controls the image processing device 160 to specify a (virtual) viewpoint for generating a virtual viewpoint image. Control information for these controls is transmitted and received via a network mainly using communication packets (hereinafter referred to as control packets) defined by TCP / IP. The network is, for example, GbE (Gigabit Ethernet), 10 GbE, or 100 GbE, which is IEEE standard-compliant, such as Ethernet (registered trademark, omitted hereinafter). It may also be configured by combining an interconnect (such as Infiniband), industrial Ethernet, etc. The network is not limited to these, and other types of networks may also be used.

[0022] Next, a description will be given of an operation of transmitting an image packet containing image data generated by the sensor systems 110-1 to 110-n to the image processing device 160. Note that the term "image packet" can be replaced with other terms as long as it is a unit of data that contains image data and can be transmitted over a transmission path. In the sensor system 110-n, an image (image capture data) captured by the camera 111-n is converted into image data by image processing (described below) in the camera adapter 112-n. An image packet containing the image data is transmitted to the camera adapter 112-(n-1) of the sensor system 110-(n-1) via the transmission path 120-n. The sensor system 110-(n-1) transmits an image packet (or a separate image packet) containing the image data generated by the camera adapter 112-(n-1) and the image data acquired from the sensor system 110-n to the adjacent sensor system 110-(n-2).

[0023] By continuing such operations, image packets containing image data generated by the sensor systems 110-1 to 110-n are transmitted to the hub 130 via the transmission path 120-1, and then transmitted to the image processing device 160 via the transmission path 131. The transmission path 131 is configured to be capable of communication at a communication band wider than the communication band of the transmission path 120.

[0024] Next, the operation of the image processing device 160 will be described. The image processing device 160 of this embodiment processes image data acquired from the sensor systems 110-1 to 110-n. First, the image processing device 160 acquires image packets containing image data from the sensor systems 110-1 to 110-n via the hub 130 and the transmission path 131. Next, the image processing device 160 reconstructs the image data of the acquired image packets, converts the data format, and then stores the image data in an internal storage unit according to the camera identifier, data type, and frame number. Then, the image processing device 160 accepts a designation of a viewpoint from the control device 140, reads corresponding image data from the stored information based on the accepted viewpoint, and performs rendering processing to generate a virtual viewpoint image. Note that at least some of the functions of the image processing device 160 may be possessed by the control device 140, the sensor system 110, or the user device 170.

[0025] The generated virtual viewpoint image is transmitted from the image processing device 160 to the user device 170, and the user operating the user device 170 can view the image from the viewpoint specified. That is, the image processing device 160 generates the virtual viewpoint image based on the captured images (multiple viewpoint images) captured by the multiple cameras 111-1 to 111-n and viewpoint information. Note that in this embodiment, the virtual viewpoint image is generated by the image processing device 160, but is not limited to this. For example, the virtual viewpoint image may be generated by the control device 140 or the user device 170.

[0026] 1 shows a configuration in which all of the sensor systems 110-1 to 110-n are daisy-chained, but the present invention is not limited to this. For example, the sensor systems 110-1 to 110-n may be divided into several groups, and the sensor systems 110 may be daisy-chained for each of the divided groups. This configuration is effective, for example, in a stadium. A stadium generally has multiple floors, and it is conceivable that a sensor system 110 is installed on each floor. In this case, input to the image processing device 160 can be performed for each floor or for each half circumference of the stadium. This simplifies installation and makes the system more flexible, even in locations where it is difficult to install wiring to connect all of the sensor systems 110-1 to 110-n in a single daisy chain.

[0027] [Example of packet transmission of foreground and background images in conventional technology] Next, an example of packet transmission (transmission) of foreground and background images in the prior art will be described with reference to Figures 9 to 11. For the purpose of explanation, refer to the synchronized shooting system shown in Figure 1, where the number of sensor systems 110 is assumed to be four (i.e., sensor systems 110-1 to 110-4 are present). Each of camera adapters 112-1 to 112-4 transmits an image packet including image data obtained by shooting (capturing) images using each of cameras 111-1 to 111-4.

[0028] (Example of transmitting only foreground image packets) 9 shows an example of transmission of only foreground image packets in the prior art. All camera adapters 112-1 to 112-4 are time-synchronized using PTP. Cameras 111-1 to 111-4 capture images in synchronization with identically timed image capture synchronization signals S1 to S3 generated within each camera adapter (at the timing of image capture synchronization signals S1 to S3). Packets of foreground images obtained by cutting out a subject portion such as a human body from an image obtained by capturing an image are transmitted to image processing device 160 in order, starting with camera adapter 112-4, which is farthest from image processing device 160. For example, starting with image capture synchronization signal S1, foreground image packets P14, P13, P12, and P11 are transmitted to image processing device 160 in order from camera adapters 112-4, 112-3, 112-2, and 112-1. Similarly, packets P24 to P21 and P34 to P31 of the foreground image are transmitted to the image processing device 160 starting from the shooting synchronization signals S2 and S3. Although each foreground image packet is shown as a single block, it is actually made up of multiple packets. For example, the block for foreground image packet P14 shows that camera adapter 112-4 fragments the captured image and transmits the packets continuously until all of the fragmented image information has been transmitted.

[0029] (Example of transmission of packets for foreground and background images (normal)) FIG. 10 shows an example of normal transmission of foreground image and background image packets in conventional technology. To generate virtual viewpoint content, not only foreground images but also background images are required. FIG. 10 shows an example in which camera adapters 112-2 to 112-4 transmit foreground image packets, and camera adapter 112-1 transmits background image packets to image processing device 160. To improve the image quality of the subject of the virtual viewpoint content, the frame rate of the background image is slower than that of the foreground image, thereby reducing the amount of background image data in the total amount of image transmission data. Furthermore, to level the communication bandwidth of the entire system, the background image packets are transmitted at intervals. In the example of FIG. 10, camera adapter 112-1 transmits background image packets B11 to B18 at intervals. Note that background image packets B11 to B18 are obtained by dividing an image captured at the timing of the capture synchronization signal S1. Furthermore, in consideration of delays due to image processing performed after capturing an image, a configuration is possible in which an image captured at a timing prior to the timing of the capture synchronization signal S1 and then divided and transmitted after image processing.

[0030] 10, camera adapters 112-2 to 112-4 generate three capture synchronization signals, S1 to S3, and transmit packets of foreground images in accordance with each capture synchronization signal. On the other hand, camera adapter 112-1 generates two capture synchronization signals, S1 and S3, and transmits packets of background images in accordance with each capture synchronization signal. As such, camera adapter 112-1 generates capture synchronization signals less frequently than the other camera adapters. This is because the frame rate of background images is lower than that of foreground images because there is little image difference between frames and because a transmission bandwidth needs to be allocated to the foreground images.

[0031] The camera adapter 112-1 starts transmitting background image packets at the timing of the capture synchronization signal S1, and the transmission interval is preset so that packets B11 to B18 are sent by the time of the next capture synchronization signal S3. While the camera adapter 112 is transmitting packets it has generated, it is unable to transmit image packets received from other camera adapters, and the packets accumulate in the buffer. If the camera adapter 112 prioritizes packets it has generated, packets received from other camera adapters will continue to accumulate in its limited buffer, eventually causing the buffer to overflow and unable to accommodate the packets. To prevent this buffer overflow, the camera adapter 112 prioritizes relaying received image packets over transmitting image packets it has generated itself. In the example of FIG. 10, due to this priority control, the camera adapter 112-1 waits for the background image packets it generates before transmitting them. Therefore, the intervals between transmissions of background image packets are not always equal.

[0032] (Example of transmission of packets for foreground and background images (when the total data volume of the foreground image increases)) The total data volume of foreground images increases due to a deterioration in the compression rate of captured images and an increase in the number of subjects. Fig. 11 shows an example of transmission of foreground image and background image packets when the total data volume of foreground images increases in conventional technology. As with Fig. 10, Fig. 11 shows an example in which camera adapters 112-2 to 112-4 transmit foreground image packets and camera adapter 112-1 transmits background image packets to image processing device 160. However, the data volume of the foreground image packets in Fig. 11 is larger than that of the foreground image packets in Fig. 10.

[0033] As in FIG. 10 , camera adapter 112-1 generates two capture synchronization signals, S1 and S3. Due to the drawbacks of the priority control described above, the transmission of background image packets by camera adapter 112-1, which began with capture synchronization signal S1, is unable to send all image information by the time of the next capture synchronization signal S3. Therefore, background image packet B18 is transmitted after foreground image packet P34, which camera adapter 112-4 began transmitting with capture synchronization signal S3. Once image packet transmission has begun, it is not interrupted until all fragmented image information has been sent. Not only camera adapter 112-1, but each camera adapter 112 does not generate or transmit an image packet associated with that capture synchronization signal if the timing of the next capture synchronization signal arrives before all image information has been sent after the camera adapter 112 has started transmitting a generated image packet. This is to limit the communication bandwidth available for image packets transmitted in response to that capture synchronization signal and prevent system failure.

[0034] 11, the camera adapter 112-1 started transmitting background image packets in response to the capture synchronization signal S1, but did not finish transmitting them by the time of the next capture synchronization signal S3. Therefore, the camera adapter 112-1 does not generate and transmit background image packets in response to the capture synchronization signal S3. The camera adapter 112-1 then generates and transmits image packets in response to the capture synchronization signal S5 (not shown) that follows the capture synchronization signal S3. Therefore, the image processing device 160 receives background image packets less frequently, and is unable to generate high-quality virtual viewpoint images.

[0035] In this embodiment, the camera adapter is configured so that even if the total data amount of the foreground image increases, packets of the background image can be reliably transmitted to image processing device 160. The configuration and operation of camera adapter 112 according to this embodiment will be described below.

[0036] [Camera Adapter 112 Configuration] Next, the configuration of camera adapter 112 according to this embodiment will be described with reference to Fig. 2. Fig. 2 is a block diagram showing an example of the configuration of camera adapter 112. Note that, although the camera adapter 112-2 in Fig. 1 will be described as an example here, other camera adapters 112 may have a similar configuration.

[0037] The camera adapter 112-2 includes a CPU 201, a memory unit 202, a first communication I / F unit 203a, a second communication I / F unit 203b, a time synchronization control unit 204, an imaging control unit 205, an image processing unit 206, a transmission unit 207, and a system bus 208.

[0038] The CPU (Central Processing Unit) 201 is a control unit (processing unit) that controls the entire camera adapter 112-2, as well as controls the transmission and reception of PTP packets transmitted from the time server 150 and the transmission and reception of control packets transmitted from the control device 140. This control can be performed by the CPU 201 executing a control program stored in the storage unit 202.

[0039] The storage unit 202 is a memory that stores the control program executed by the CPU 201, PTP packets and control packets to be transmitted and received, etc. The storage unit 202 also stores, as image data, data that has been subjected to image processing by the image processing unit 206 on the captured image data received from the camera 111-2. Note that although only one storage unit 202 is shown in FIG. 2, the storage unit 202 may be divided into multiple storage units depending on the application. The type of memory that constitutes the storage unit 202 is not limited to a specific one.

[0040] A first communication I / F (interface) unit 203a and a second communication I / F unit 203b transmit and receive communication packets (PTP packets, control packets, image packets, etc.) between the camera adapter 112-2 and an external device. DMA (Direct Memory Access) transfer instructions for transmitting and receiving packets are issued by the CPU 201 and a transmission unit 207.

[0041] The following describes the operations of the first communication I / F unit 203a and the second communication I / F unit 203b in response to a DMA transfer instruction. For the sake of explanation, unless otherwise specified, the first communication I / F unit 203b and the second communication I / F unit 203b will be collectively referred to as the communication I / F unit 203. First, a general operation of the communication I / F unit 203 will be described. The DMA transfer instruction is accompanied by information specifying an area to read from or an area to write to in storage unit 202. When communication I / F unit 203 transmits a packet, it reads the packet from the area in storage unit 202 specified in the DMA transfer instruction and transmits it to external camera adapter 112. On the other hand, when communication I / F unit 203 receives a packet from camera adapter 112 external to camera adapter 112-2, it writes the received packet to the area in storage unit 202 specified in the DMA transfer instruction.

[0042] Next, the transmission and reception process of PTP packets will be described. The PTP packet transmitted by the time server 150 is received by the first communication I / F unit 203a via the camera adapter 112-1, and is transferred to a designated area in the storage unit 202 in Fig. 2 in response to a DMA transfer instruction from the CPU 201. Then, after protocol processing is performed on the PTP packet within the camera adapter 112-2, the packet is read from the designated area in response to a DMA transfer instruction from the CPU 201, and is transmitted to the adjacent camera adapter 112-3 via the second communication I / F unit 203b. 2 and is transferred once to the storage unit 202. Then, after protocol processing is performed on the PTP packets, the PTP packets are transferred from a designated area of the storage unit 202 to the first communication I / F unit 203a in response to a DMA transfer instruction from the CPU 201, and are finally transmitted to the time server 150. In addition, PTP packets generated by the camera adapter 112-2 itself are similarly stored in the storage unit 202 and transmitted to the time server 150 via the first communication I / F unit 203a. The process of transmitting and receiving image packets will be described later. Since it is not known when a packet will be received, the first communication I / F unit 203 and the second communication I / F unit 203b can be configured to buffer multiple transfer instructions in advance.

[0043] The communication I / F unit 203 has a function (hereinafter referred to as a timestamp function) that maintains the time on an internal clock when a PTP packet is sent or received. The CPU 201 uses this function for time synchronization calculations. The timestamp function makes it possible to accurately determine the time when a PTP packet is sent or received and the time that the PTP packet remains in the camera adapter 112-2.

[0044] The time synchronization control unit 204 has a clock that synchronizes with the clock in the communication I / F unit 203. Furthermore, the time synchronization control unit 204 has a function of outputting a signal (for example, 1 Hz) synchronized with its own clock as a synchronization signal to the imaging control unit 205. Since the communication I / F unit 203 has a total of two clocks, the clock synchronization control unit 204 also has two clocks. The synchronization signal output by the time synchronization control unit 204 is synchronized with one of the internal clocks. In order to measure the accurate residence time, the clocks in the first communication I / F unit 203a and the second communication I / F unit 203b are synchronized with each other before starting time synchronization using PTP.

[0045] The imaging control unit 205 controls the camera 111-2. Upon receiving an imaging instruction from the control device 140, the CPU 201 sets the imaging control unit 205 and starts outputting a Genlock signal and time code to the camera 111-2. The Genlock signal and time code are generated based on a synchronization signal received from the time synchronization control unit 204. Furthermore, the imaging control unit 205 outputs a part of the time code as a trigger signal to the image processing unit 206. The trigger signal is synchronized with the synchronization signal and, in this embodiment, is generated at a cycle equal to the imaging frame rate of the camera 111-2. For example, if the camera adapter 112 that generates a foreground image is 60 fps and the camera adapter 112 that generates a background image is 30 fps, the generated trigger signals will be 60 Hz and 30 Hz, respectively. The camera 111-2 captures (takes) images in synchronization with the received Genlock signal, and outputs the captured image data to the image processing unit 206 together with the received timecode.

[0046] The image processing unit 206 performs image processing (such as cutting out background and foreground parts) required to generate a virtual viewpoint image from the imaging data received from the camera 111-2, and transfers the image data resulting from the processing to the storage unit 202. Furthermore, the image processing unit 206 transmits a reference signal 210 including attribute information of the image data (information related to the generated image) to the transmission unit 207 at the timing when a trigger signal is received from the imaging control unit 205. That is, the image processing unit 206 transmits the reference signal 210 to the transmission unit 207 at intervals identical to the imaging frame rate (at periodic timing).

[0047] The configuration of the reference signal 210 is shown in FIG. 8(a). The reference signal 210 includes address information 211 stored in the storage unit 202 for each image type, such as a foreground image or a background image, data size information 212 for the image data, and a frame number 213 corresponding to the image data. The imaging process of the camera 111-2 and the image processing by the image processing unit 206 each involve overhead. The reference signal 210 notified based on the Nth received trigger signal does not necessarily notify the transmitter 207 of image data obtained by processing the Nth imaging data received from the camera 111-2 (N is an arbitrary number). Taking into account the delay due to the overhead, the reference signal 210 notifies the transmitter 207 of image data obtained by image processing the N-1th or N-2th imaging data. If the reference signal 210 notified based on the Nth received trigger signal notifies image data obtained by processing the N-1th imaging data, the relationship (the shift of one) will not change thereafter.

[0048] The transmitter 207 controls the transmission and reception of image packets. For example, the transmitter 207 packetizes image data acquired from the storage unit 202 based on the reference signal 210 received from the image processing unit 206 (image packets), and issues a transmission instruction (transfer instruction) to the first communication I / F unit 203a. The configuration of the transmitter 207 will be described in detail later with reference to FIG. 3. The transmitter 207 receives the reference signal 210, which is generated based on the time synchronized with the time server 150, at the same interval as the imaging frame rate, and therefore, an instruction to transmit image packets is also issued periodically. Note that the camera adapter 112-2 has a function of relaying image packets received from the camera adapter 112-3, and this is realized by the transmitter 207. The relay function will be described in detail later.

[0049] [Transmission section configuration] Next, the configuration of transmission unit 207 will be described using Fig. 3. Fig. 3 is a block diagram showing an example of the configuration of transmission unit 207. Note that although transmission unit 207 of camera adapter 112-2 in Fig. 2 will be described here, transmission units 207 of other camera adapters 112 may have a similar configuration.

[0050] The transmission unit 207 includes a setting unit 301, a receiving buffer unit 302, a forwarding unit 303, an arbitration unit 304, a transmitting buffer unit 305, a transmission control unit 306, and a communication I / F control unit 307. Note that in Fig. 3, arrows point to the destination of signals / information inside the transmission unit 207.

[0051] The setting unit 301 has a function of making various settings for each functional block of the transmission unit 207 in response to the CPU 201 enabling (activating) the transmission unit 207. For this setting, the setting unit 301 holds setting information, and in response to the CPU 201 enabling the transmission unit 207, each functional block can acquire and set the setting information. For example, the setting information includes thresholds (threshold X, threshold Y, threshold Z) used by the transmission control unit 306. The setting information also includes information on timeout periods (first to third timeout periods) used to determine whether the conditions for generating an image packet are met (S405 in FIG. 4, S503 and S509 in FIG. 5). The setting information may also include information on whether the camera adapter 112-2 generates a foreground image or a background image. Whether the camera adapter 112 generates a foreground image or a background image may be determined (confirmed) by the system administrator when the synchronized photography system 100 is assembled. The system administrator may notify all camera adapters 112 in advance, via the control device 140, of the camera adapter IDs of one or more camera adapters 112 that generate background images. The setting information may also include information on the data size read by the transmission control unit 306 when generating packets.

[0052] Receiving buffer unit 302 has a function (receiving buffer) of temporarily holding image packets received by communication I / F unit 203b, and a function of outputting the held image packets to forwarding unit 303. DMA transfer of image packets from second communication I / F unit 203b to receiving buffer unit 302 is instructed by communication I / F control unit 307. When receiving an image packet from second communication I / F unit 203b, receiving buffer unit 302 analyzes the header information of the image packet to determine whether the image packet contains background image information or foreground image information.

[0053] Furthermore, the receiving buffer unit 302 outputs three status signals indicating specific states to the transmission control unit 306. The first is a first status signal 380 indicating that background image packets are stored in the receiving buffer unit 302. The second is a second status signal 381 indicating that the total amount of image packets stored in the receiving buffer 302 (total amount of received packets) exceeds a threshold Y. The third is a third status signal 382 indicating that the total amount of image packets stored in the receiving buffer unit 302 exceeds a threshold Z. In this embodiment, the threshold Z is set to a value greater than the threshold Y. When the receiving buffer unit 302 stores one or more image packets, it starts outputting them to the forwarding unit 303. At this time, the receiving buffer unit 302 starts outputting the oldest image packets among the accumulated image packets.

[0054] The forward unit 303 has a function of receiving image packets from the receive buffer unit 302 and outputting them to the arbitration unit 304. The forward unit 303 also has a function of analyzing the header information of the image packets and notifying the transmission control unit 306 of a packet generation start instruction 360, and a function of rewriting the header information of the image packets received from the receive buffer unit 302. The forward unit 303 also generates relay packet information 361 based on the header information of the received image packets and notifies the transmission control unit 306 of this information. Referring to FIG. 8(b) described below, when the image type 83 in the header 80 is a background image and the last data information 86 is 1, the forward unit 303 generates relay packet information 361 including the camera adapter ID 82 and frame number 81 of the image packet.

[0055] The arbitration unit 304 has the function of arbitrating image packets received from the forward unit 303 and the transmission control unit 306, and outputting them to the transmission buffer unit 305. The arbitration unit 304 has a total of four input ports, and receives packets from the port with the highest priority. The transmission control unit 306 and the forward unit 303 each use two input ports.

[0056] Transmission buffer unit 305 has a function (transmission buffer) of temporarily holding image packets to be transmitted outside camera adapter 112-1, and when it receives image packets from arbitration unit 304, it instructs communication I / F control unit 307 to start transfer. Furthermore, transmission buffer unit 305 outputs fourth status signal 383 to transmission control unit 306 when the total amount of stored image packets (total amount of transmitted packets) exceeds threshold X.

[0057] The transmission control unit 306 has a function of controlling packet generation of background images and foreground images generated by the image processing unit 206. The transmission control unit 306 acquires image data to be the payload of an image packet via the system bus 208 using information included in the reference signal 210 (such as address information 211 and data size 212). The transmission control unit 306 also generates header information for the image packet. The transmission control unit 306 generates an image packet by linking the acquired image data with the generated header information. The transmission control unit 306 can determine the start of image packet generation based on a packet generation start instruction 360 from the forwarding unit 303, a timer inside the transmission control unit 306, and the timeout period of the timer. The transmission control unit 306 outputs the generated image packets to the arbitration unit 304, and controls the stopping and restart of image packet generation using a total of four status signals (the above-mentioned first status signal 380 to fourth status signal 383) received from the reception buffer unit 302 and the transmission buffer 305. Details will be described later.

[0058] The communication I / F control unit 307 has a function of issuing DMA transfer instructions to the first communication I / F unit 203a and the second communication I / F unit 203b for transmitting and receiving image packets. The communication I / F control unit 307 sets a DMA transfer instruction to the communication I / F unit 203 when a transmission packet is ready. On the other hand, when receiving an image packet, the CPU 201 enables the transmission unit 207 via the setting unit 301, and the communication I / F control unit 307 sets multiple DMA transfer instructions to the communication I / F unit 203b. Every time reception of an image packet is completed, the communication I / F control unit 307 sets a new DMA transfer instruction to the communication I / F unit 203b, thereby maintaining a state in which image packets can be received at any time.

[0059] Here, the header information of an image packet will be described with reference to Fig. 8(b). Fig. 8(b) shows the configuration of an exemplary image packet. In the image packet 8 shown in Fig. 8(b), a header 80 includes a frame number 81, a camera adapter ID 82, an image type 83, an image payload size 84, an image offset 85, last data information 86, last packet information 87, and camera adapter hop count information 88.

[0060] The frame number 81 indicates the number of frames in a video. For example, if the frame rate is 60 fps (frames per second), this information will be one of the values from 0 to 59. If one image packet cannot represent all the information for one frame, multiple image packets will have the same frame number.

[0061] The camera adapter ID 82 is information indicating which camera adapter 112 generated the image packet, and is a value unique to each camera adapter 112 . The image type 83 is information indicating the type of image, such as a foreground image or background image, extracted by performing image processing on the captured image data.

[0062] The image payload size 84 indicates the length of the image data included in the image packet (image payload length). The image offset 85 is offset information from the beginning of the image data. If the image offset 85 is zero, it indicates that it is the beginning data for that image type (image type 83). If one image packet cannot represent one frame of image data for that image type, the image data is divided, and offset information as position information for the divided parts is written in the image offset 85. The image processing device 160 uses the image offset 85 of the image packet to expand the image data in internal memory and perform reassembly.

[0063] The last data information 86 is information that is 1 when the frame number 81 indicates the last data of the image type (image type 83) of the frame number M (M is 0 or a positive integer). Last packet information 87 is information that is set to 1 if it is the last image packet of frame number M transmitted by that camera adapter 112. For example, if the image data notified by reference signal 210 includes two types of image data, a foreground image and a background image, that camera adapter 112 will transmit two packets with last data information 86 set to 1 and one packet with last packet information 87 set to 1. Furthermore, last packet information 87 is literally added when the last packet of frame number M is transmitted. Similarly, last data information 86 is also added to the last packet of that image type. When dividing image data to generate image packets, transmission control unit 306 divides the image data in order from the beginning. The camera adapter hop count 88 indicates how many camera adapters 112 the image packet has passed through. This value is decremented by the forwarding unit 303.

[0064] [Image packet transmission procedure using the camera adapter] Next, a transmission procedure for an image packet generated by the camera adapter 112 and a transmission procedure for an image packet received by the camera adapter 112 will be described with reference to FIG.

[0065] (1) Transmission procedure of image packets generated by the camera adapter First, a description will be given of a method for transmitting an image packet generated by camera adapter 112. Note that, although the description here takes an example in which an image packet generated by camera adapter 112-2 in Figures 1 to 3 is transmitted to camera adapter 112-1, the same description can be applied to other camera adapters 112.

[0066] The transmission control unit 306 starts generating image packets using information included in the reference signal 210, based on packet generation start instruction 360 received from the forwarding unit 303 or the timeout or timeout period setting value of an internal timer. The transmission control unit 306 outputs the generated image packets to the arbitration unit 304. The arbitration unit 304 outputs the image packets received from the transmission control unit 306 to the transmission buffer unit 305. When an image packet is also input from the forwarding unit 303, the arbitration unit 304 receives the image packet from the port with the highest priority. The transmission buffer unit 305 stores the received image packet internally and notifies the communication I / F unit 307 that DMA can start. Upon receiving the notification, the communication I / F control unit 307 issues a DMA transfer instruction to the communication I / F unit 203a, and the communication I / F unit 203a reads the image packet from the transmission buffer unit 305 and transmits it to the adjacent camera adapter 112-1 via the communication path 120-2.

[0067] (2) Transmission procedure of image packets received by the camera adapter Next, a description will be given of a method for transmitting image packets received by camera adapter 112. Here, an example will be described in which camera adapter 112-2 transmits image packets received from camera adapter 112-3 to camera adapter 112-1, but a similar description can also be applied to other camera adapters 112.

[0068] In response to a DMA transfer instruction generated by the communication I / F unit 307, the image packets received by the second communication I / F unit 203b are temporarily stored in the receiving buffer unit 302. When one or more image packets have accumulated in the receiving buffer unit 302, the receiving buffer unit 302 outputs the image packets to the forwarding unit 303.

[0069] The forwarding unit 303 outputs the received image packet to the arbitration unit 304, and also analyzes the header information of the image packet (header 80 in Figure 8(b)) and, if the last packet information 87 is 1, notifies the transmission control unit 306 of a packet generation start instruction 360. Furthermore, when forwarding unit 303 outputs an image packet to arbitration unit 304, it rewrites the header information of the image packet based on whether the image generated by camera adapter 112-2 is a foreground image or a background image. Forwarding unit 303 can determine whether the image generated by camera adapter 112-2 is a foreground image or a background image based on reference signal 210 (address information 211, etc.), setting information by setting unit 301, etc. When camera adapter 112-2 generates a foreground image, if last packet information 87 in header 80 of the received image packet is 1, forward unit 303 rewrites last packet information 87 to 0. On the other hand, when camera adapter 112-2 generates a background image, forward unit 303 does not rewrite it.

[0070] The arbitration unit 304 outputs the image packets received from the forwarding unit 303 to the transmission buffer unit 305. When an image packet is also input from the transmission control unit 306, the arbitration unit 304 receives the image packet from the port with the highest priority. The subsequent processing is the same as the procedure for transmitting the image packet generated by the camera adapter 112-2, and therefore a description thereof will be omitted.

[0071] [Operation flow of the transmission control unit] Next, an operational flow regarding the start of image packet generation by the transmission control unit 306 will be described with reference to Fig. 4. Fig. 4 is a flowchart of image packet transmission processing executed by the transmission control unit 306 of the camera adapter 112 according to this embodiment. The processing of this flow can be started when CPU 201 starts up transmission unit 207 via setting unit 301. The processing of this flow may also be started when reset is released after power is applied to camera adapter 112. Priority processing of image packets is performed according to this flow and the flows shown in Figures 5 to 7, which will be described later.

[0072] In S401, the transmission control unit 306 determines whether or not the reference signal 210 has been received from the image processing unit 206. If the reference signal 210 has been received (Yes in S401), the process proceeds to S402, and if the reference signal 210 has not been received (No in S401), the process of S401 continues. If the process proceeds to S402, the transmission control unit 306 holds information about the reference signal 210.

[0073] In S402, the transmission control unit 306 determines whether the first timeout time, which is the timeout time included in the setting information by the setting unit 301, is set to 0. If the first timeout time is set to 0 (Yes in S402), the process proceeds to S407; if not (No in S402), the process proceeds to S403.

[0074] In S403, the transmission control unit 306 starts a first timer, which is a timer inside the transmission control unit 306. Here, the first timer is assumed to be an increment timer configured to be incremented. Note that, when a decrement timer configured to be decremented is used, the first timer is started by setting a first timeout period included in the setting information by the setting unit 301 to the first timer.

[0075] In S404, the transmission control unit 306 determines whether or not the packet generation start instruction 360 has been received from the forwarding unit 303. If the packet generation start instruction 360 has been received (Yes in S404), the process proceeds to S407, and if not (No in S404), the process proceeds to S405.

[0076] In S405, the transmission control unit 306 determines whether the timer value indicated by the first timer started in S403 has reached the first timeout period (whether the first timer has timed out). If the timer value has reached the first timeout period (Yes in S405), the process proceeds to S407; if not (No in S405), the process proceeds to S406. Note that when a decrement timer is used, it is determined whether the first timer has reached 0; if it has reached 0, the process proceeds to S407; if not, the process proceeds to S406.

[0077] In S406, similar to S401, the transmission control unit 306 determines whether or not the reference signal 210 has been received from the image processing unit 206. If the reference signal 210 has been received (Yes in S406), the process proceeds to S402; if not (No in S406), the process proceeds to S404. Note that if the process proceeds to S402, the transmission control unit 306 retains the information of the most recently received reference signal 210 and discards the information of the older (previously received) reference signal 210.

[0078] In S407, the transmission control unit 306 starts image packet generation processing using information contained in the reference signal 210 received in S401 or S406. After starting, the processing proceeds to S408. The image packet generation processing will be described later with reference to FIG. 5. In S408, the transmission control unit 306 determines whether the total amount of packets stored in the transmission buffer unit 305 (total amount of transmitted packets) has exceeded the threshold X included in the setting information by the setting unit 301. When the transmission control unit 306 receives a fourth status signal 383 from the transmission buffer unit 305, it can determine that the total amount of transmitted packets has exceeded the threshold X. If the total amount of transmitted packets exceeds the threshold X (Yes in S408), the process proceeds to S409, and if the total amount of transmitted packets is equal to or less than the threshold X (No in S408), the process proceeds to S414.

[0079] In S409, the transmission control unit 306 stops packet generation of the foreground image in the image packet generation process started in S407. After stopping, the process proceeds to S410. Note that if the process has already stopped, the process of S409 is skipped. In S410, the transmission control unit 306 determines whether the total amount of packets stored in the reception buffer unit 302 (total amount of received packets) has exceeded the threshold Y set in the setting unit 301. When the transmission control unit 306 receives the second status signal 381 from the forwarding unit 303, it can determine that the total amount of received packets has exceeded the threshold Y. If the total amount of received packets exceeds the threshold Y (Yes in S410), the process proceeds to S411, and if the total amount of received packets is equal to or less than the threshold Y (No in S410), the process proceeds to S415.

[0080] In S411, the transmission control unit 306 determines whether the total amount of packets stored in the reception buffer unit 302 (total amount of received packets) has exceeded the threshold Z set in the setting unit 301. When the transmission control unit 306 receives the third status signal 382 from the forwarding unit 303, it can determine that the total amount of received packets has exceeded the threshold Z. If the total amount of received packets exceeds the threshold Z (Yes in S411), the process proceeds to S412, and if the total amount of received packets is equal to or less than the threshold Z (No in S411), the process proceeds to S413.

[0081] In S412, the transmission control unit 306 determines whether relaying of all background image packets for the current frame number has finished. If relaying of all background image packets has finished (Yes in S412), the process proceeds to S415; if not (No in S412), the process proceeds to S416. Here, a specific example of the processing of S412 will be described. As described above, the camera adapter 112 that generates the background image can be determined when the synchronized photography system 100 is assembled. In this case, the system administrator notifies all camera adapters 112 that generate background images via the control device 140 of the camera adapter IDs of all camera adapters 112 that generate background images. If the relay packet is a packet containing the last background image generated by the camera adapter 112 that generates the background image, the forwarding unit 303 notifies this in the relay packet information 361. Because the transmission control unit 306 knows the camera adapter ID of the camera adapter 112 that generates the background image, it can determine whether any subsequent relay packets contain a packet containing a background image. The reference frame number is included as frame number 213 in the reference signal 210 received in S401 or S406, and the frame number of the image packet to be relayed is included as frame number 81 in the relay packet information 361, making the determination in S412 possible. If the background is 30 fps and the foreground is 60 fps, and the frame number of the camera adapter 112 that generates the foreground image packets is 2M or 2M+1, S412 branches depending on whether all background image packets up to frame number M have been relayed. This means that if the background is 30 fps and the foreground is 60 fps, two foreground images are processed for every background image that is processed. Therefore, by the time the frame number of the camera adapter 112 that generates the foreground image packets processes 2M or 2M+1, all background image packets up to frame number M should have been relayed, and this is confirmed in S412.

[0082] In S413, the transmission control unit 306 determines whether or not there are background image packets in the receiving buffer unit 302. When the transmission control unit 306 receives the first status signal 380 from the forwarding unit 303, it can determine that there are background image packets in the receiving buffer unit 302. If there are background image packets in the receiving buffer unit 302 (Yes in S413), the process proceeds to S416; if there are not (No in S413), the process proceeds to S415.

[0083] In S414, transmission control unit 306 resumes packet generation for the S foreground image / background image. Note that if packet generation for the foreground image / background image is not stopped, the process of S414 is skipped.

[0084] In S415, the transmission control unit 306 resumes packet generation of the background image in the image packet generation process started in S407. Note that if packet generation of the background image is not stopped, the process of S415 is skipped. In S416, the transmission control unit 306 stops the packet generation of the background image in the image packet generation process started in S407. If the packet generation of the background image is already stopped, the process of S416 is skipped.

[0085] In S417, the transmission control unit 306 determines whether all images indicated by the received reference signal 210 have been packetized and output in the image packet generation process started in S407. If all images have been output (Yes in S417), the process proceeds to S418; if not (No in S417), the process proceeds to S408. Note that if the reference signal 210 received in S401 or S406 indicates that there are no images (image packets) to be generated, the process also proceeds to S418. If the process proceeds to S418, the transmission control unit 306 discards the information of the reference signal 210 received in S401 or S406 and prepares to receive the next reference signal 210.

[0086] In S418, the transmission control unit 306 determines whether an end instruction has been detected. If an end instruction has been detected (Yes in S418), this flow ends; if not (No in S418), the process proceeds to S401. The end instruction is issued, for example, by the CPU 201 via the setting unit 301.

[0087] In this way, after receiving the reference signal 210, the transmission control unit 306 starts packetizing image data when the first timeout period is set to 0 (S402), when the packet generation start instruction 360 is received (S404), or when the first timer reaches the first timeout period (S405). In the camera adapter 112, if the transmission control unit 306 continuously generates image packets when there are many relay packets, the received image packets and the generated image packets are alternately stored in the transmission buffer unit 305. This gradually depletes the receive buffer unit 302. To prevent this situation, the camera adapter 112 checks the buffer status by using a predetermined threshold to determine whether the total amount of data stored in the buffer exceeds or is below the threshold. Depending on the buffer status, the camera adapter 112 temporarily stops generating image packets and operates to preferentially store received packets in the transmission buffer unit 305, thereby making adjustments to reduce the likelihood of buffer overflow in the receive buffer unit 302. It should be noted that even if the transmission control unit 306 receives the reference signal 210 while the process is looping from S408 to S417, the information is internally discarded.

[0088] Next, the process of S407 in Fig. 4 will be described in detail with reference to Fig. 5. Fig. 5 is a flowchart of the image packet generation process. For the explanation of Fig. 5, Fig. 4 will be referred to. In S501, the transmission control unit 306 determines whether the reference signal 210 received in 401 or S406 in Fig. 4 indicates that there is a background image to transmit. If there is a background image to transmit (Yes in S501), the process proceeds to S502; if there is not (No in S501), the process proceeds to S507.

[0089] In S502, the transmission control unit 306 starts a second timer, which is a timer for generating packets of the background image. This second timer is a timer separate from the first timer used in the process of S403 in Fig. 4. The timeout period for the second timer is set to a second timeout period. In S503, the transmission control unit 306 determines whether the timer value indicated by the second timer started in S502 has reached the second timeout period (whether the second timer has timed out). If the timer value has reached the second timeout period (Yes in S503), the process proceeds to S504; if not (No in S503), the transmission control unit 306 waits until the timer value reaches the second timeout period. The second timeout period used in the process of S503 is the timeout period for generating and transmitting packets of the background image. Note that the second timeout period is set to a value greater than the third timeout period for transmitting the foreground image, which is used in the process of S509 (described later). This is to further equalize the bandwidth used for transmitting the background image.

[0090] In S504, the transmission control unit 306 determines whether packet generation for the background image is stopped. That is, the transmission control unit 306 determines whether packet generation for the background image is stopped by the process of S416 in FIG. 4. If packet generation for the background image is stopped (Yes in S504), the transmission control unit 306 waits until the stoppage of packet generation for the background image is released; if not (No in S504), the process proceeds to S505. The stoppage of packet generation for the background image is released and resumed by the process of S414 or S415 in FIG. 4.

[0091] In S505, the transmission control unit 306 uses the address information 211 included in the reference signal 210 to read a predetermined data size of the background image from the storage unit 202, packetize it, and output it to port 2 of the arbitration unit 304. As described above, when packetizing, the predetermined data size is read sequentially from the beginning of the background image indicated by the reference signal 210. The data size to be read is specified by the setting unit 301.

[0092] In S506, the transmission control unit 306 determines whether all of the background images indicated by the reference signal 210 have been packetized and output to the arbitration unit 304. If all have been output (Yes in S506), this flow ends; if not (No in S506), the process proceeds to S502.

[0093] In S507, transmission control unit 306 determines whether reference signal 210 received in S401 or S406 of Fig. 4 indicates that there is a foreground image to transmit. If there is a foreground image to transmit (Yes in S507), the process proceeds to S508, and if there is not (No in S507), this flow ends.

[0094] In S508, transmission control unit 306 starts a third timer, which is a timer for generating packets of the foreground image. This timer is a separate timer from the first and second timers. The timeout period for the third timer is set to a third timeout period. In S509, the transmission control unit 306 determines whether the timer value indicated by the third timer started in S508 has reached the third timeout period (whether the third timer has timed out). If the timer value has reached the third timeout period (Yes in S509), the process proceeds to S510; if not (No in S509), the transmission control unit 306 waits until the timer value reaches the third timeout period.

[0095] In S510, transmission control unit 306 determines whether packet generation for the foreground image is stopped. That is, transmission control unit 306 determines whether packet generation for the foreground image is stopped by the processing of S409 in FIG. 4. If packet generation for the foreground image is stopped (Yes in S510), transmission control unit 306 waits until the stoppage of packet generation for the foreground image is released; if not (No in S510), the processing proceeds to S511. Packet generation for the foreground image is released and resumed by the processing of S414 in FIG. 4.

[0096] In S511, transmission control unit 306 uses address information 211 included in reference signal 210 to read a predetermined data size of the foreground image from storage unit 202, packetize it, and output it to port 4 of arbitration unit 304. As with packetizing the background image, when packetizing, transmission control unit 306 reads the predetermined data size in order from the beginning of the foreground image indicated by reference signal 210. The data size to be read is specified by setting unit 301.

[0097] In S512, transmission control unit 306 determines whether all foreground images indicated by reference signal 210 have been packetized and output to arbitration unit 304. If all have been output (Yes in S512), this flow ends; otherwise (No in S512), the process proceeds to S508.

[0098] As described above, the transmission control unit 306 generates packets of background and foreground images based on the information of the reference signal 210 received from the image processing unit 206, and outputs the packets by switching the input port of the arbitration unit 304 depending on the type of image. The transmission control unit 306 can also adjust the output interval of the image packets by using an internal timer. The second and third timers may be configured as the same timer.

[0099] [Forward section operation] Next, the operational flow of the forward unit 303 will be described using Fig. 6. Fig. 6 is a flowchart of the image packet distribution process executed by the forward unit 303 of the camera adapter 112 according to this embodiment. The process of this flow can be started when the camera adapter 112 is powered on and reset is released.

[0100] In S601, the forward unit 303 determines whether or not an image packet has been input from the receiving buffer unit 302. If an input has been received (Yes in S601), the process proceeds to S602; if not (No in S601), the process of S601 continues. In S602 , the forwarding unit 303 receives one image packet from the receiving buffer unit 302 .

[0101] In S603, the forward unit 303 analyzes the header information (header 80 in FIG. 8) of the received image packet and determines whether the image type 83 indicates a background image (whether the image type is a background image). If the image type is a background image (Yes in S603), the process proceeds to S604; if not (No in S603), the process proceeds to S605. Note that if the image type 83 indicates a background image and the last data information 86 is 1 in the header 80 of the received image packet, the forward unit 303 generates relay packet information 361 including the camera adapter ID 82 and frame number 81 of the image packet and notifies the transmission control unit 306 of this.

[0102] In S604, forward unit 303 outputs the received image packet to port 1 of arbitration unit 304. After output, processing returns to S601. Here, if last packet information 87 in header 80 of the received image packet is 1, forward unit 303 notifies transmission control unit 306 of packet generation start instruction 360. Furthermore, forward unit 303 rewrites (decrements) camera adapter hop count 88. Furthermore, forward unit 303 rewrites last packet information to 0 only when a foreground image packet is generated by camera adapter 112 in which forward unit 303 is configured. Thereafter, forward unit 303 outputs the image packet to arbitration unit 304.

[0103] In S605, the forward unit 303 outputs the received image packet to port 3 of the arbitration unit 304. After output, the process returns to S601. Furthermore, the forward unit 303 notifies the generation start instruction 360 and rewrites the header information of the image packet, similar to the process of S604. As a result, the received image packets are distributed to the two ports of the arbitration unit 304.

[0104] [Operation of the arbitration unit] Next, the operational flow of the arbitration unit 304 will be described using Fig. 7. Fig. 7 is a flowchart of packet adjustment processing executed by the arbitration unit 304 of the camera adapter according to this embodiment. The processing of this flow can be started when the camera adapter 112 is powered on and reset is released.

[0105] In S701, the arbitration unit 304 determines whether or not an image packet has been input to port 1. If an image packet has been input to port 1 (Yes in S701), the process proceeds to S702; if not (No in S701), the process proceeds to S703. In S 702 , the arbitration unit 304 receives an image packet from port 1 and outputs the received image packet to the transmission buffer unit 305 .

[0106] In S703, the arbitration unit 304 determines whether an image packet has been input to port 2. If an image packet has been input to port 2 (Yes in S703), the process proceeds to S704; if not (No in S703), the process proceeds to S705. In S704, the arbitration unit 304 receives the image packet from port 2 and outputs the received image packet to the transmission buffer unit 305.

[0107] In S705, the arbitration unit 304 determines whether or not an image packet has been input to port 3. If an image packet has been input to port 3 (Yes in S705), the process proceeds to S706; if not (No in S705), the process proceeds to S707. In S706, the arbitration unit 304 receives the image packet from port 3 and outputs the received image packet to the transmission buffer unit 305.

[0108] In S707, the arbitration unit 304 determines whether an image packet has been input to port 4. If an image packet has been input to port 4 (Yes in S707), the process proceeds to S708; if not (No in S707), the process proceeds to S701. In S708, the arbitration unit 304 receives the image packet from port 4 and outputs the received image packet to the transmission buffer unit 305.

[0109] As described above, arbitration unit 304 determines whether or not a packet has been input in order from port 1, and outputs the received image packet to transmission buffer unit 305. The output priority order, from highest to lowest, is background image received from another camera adapter 112, background image generated by camera adapter 112 itself, foreground image received from another camera adapter 112, and foreground image generated by camera adapter 112 itself.

[0110] In this manner, according to this embodiment, even if a large number of relay packets containing foreground and background images are received, the background image generating device can transmit the image packets with higher priority according to the priority order. This allows the foreground and background images to be transmitted to the image processing device efficiently and without delay, enabling the image processing device to generate high-quality virtual viewpoint content.

[0111] The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program.The present invention can also be realized by a circuit (e.g., ASIC) that realizes one or more functions. [Explanation of symbols]

[0112] 100: Synchronous photography system, 110: Sensor system, 111: Camera, 112: Camera adapter, 113: Connection, 120: Transmission path, 130: Hub, 140: Control device, 150: Time server, 160: Image processing device, 170: User device

Claims

1. A communication device, generating means for generating one or more image packets including a foreground image or a background image from the captured image; receiving means for receiving one or more image packets containing a foreground image or a background image from a first other communication device; a transmitting means for transmitting the image packets generated by the generating means and the image packets received by the receiving means to a second other communication device in accordance with a predetermined priority; the predetermined priority is a priority based on whether the image packet is an image packet generated by the generating means or an image packet received by the receiving means, and whether the image packet includes a foreground image or a background image; the receiving means has a receiving buffer for temporarily storing received image packets; A communication device characterized in that when the total amount of image packets stored in the receiving buffer exceeds a second threshold and is equal to or less than a third threshold that is greater than the second threshold, and when image packets containing background images are stored in the receiving buffer, the generation means stops generating packets containing background images.

2. 2. The communication device according to claim 1, wherein the predetermined priority is configured so that transmission of an image packet received by the receiving means is given priority over transmission of an image packet generated by the generating means.

3. 3. The communication device according to claim 1, wherein the predetermined priority is configured so that transmission of an image packet containing a background image is given priority over transmission of an image packet containing a foreground image.

4. the transmitting means has a transmission buffer for temporarily storing image packets before being transmitted to the second other communication device; 4. The communication device according to claim 1, wherein the generating means stops generating packets containing foreground images when the total amount of image packets stored in the transmission buffer exceeds a first threshold.

5. the image packet generated by the generating means is transmitted to the second other communication device by the transmitting means; The communication device described in claim 1, characterized in that when the total amount of image packets stored in the receiving buffer exceeds the third threshold and the determination means determines that all of the background image packets have not been transmitted by the transmission means to the second other communication device, the generation means stops generating packets including the background image.

6. The communication device according to claim 5, characterized in that the determination means determines whether all of the background image packets have been transmitted by the transmission means to the second other communication device by analyzing header information of the image packets received by the reception means.

7. A method for controlling a communication device, comprising: a generating step of generating one or more image packets from the captured image, each image packet including a foreground image or a background image; receiving one or more image packets including a foreground image or a background image from a first other communication device; a transmitting step of transmitting the image packets generated in the generating step and the image packets received in the receiving step to a second other communication device in accordance with a predetermined priority order; the predetermined priority is a priority based on whether the image packet is an image packet generated in the generating step or an image packet received in the receiving step, and whether the image packet includes a foreground image or a background image; A control method characterized in that, in the receiving process, if the total amount of image packets stored in a receiving buffer that temporarily stores received image packets exceeds a second threshold and is equal to or less than a third threshold that is greater than the second threshold, and if image packets containing background images are stored in the receiving buffer, the generating process stops generating packets containing background images.

8. A program for causing a computer to function as each of the means of the communication device according to any one of claims 1 to 6.

Images