Terminal device, control method, and program

Abstract

To provide a technology of suppressing output of unnecessary image data in an image processing system that performs a plurality of image processes.SOLUTION: A terminal device includes: processing means of performing a process of accepting input of an image dataset including a plurality of image data, generating processed data by applying image processing to the image data, and outputting the processed data to an output buffer, the process requiring a prescribed time from the input to the output; transfer means of reading the processed data from the output buffer, and outputting the processed data to a communication channel; and control means of performing control so that the processed data stored in the output buffer should not be output to the communication channel during a predetermined period from inputting the image data of the image dataset to outputting the first processed data of the image dataset to the output buffer.SELECTED DRAWING: Figure 7

Description

Technical Field

[0001] The present invention provides a technique for suppressing the output of unnecessary image data in an image processing system that executes a plurality of image processes.

Background Art

[0002] Recently, techniques for performing control while maintaining time synchronization among a plurality of devices connected via a network have been widely used. For example, in the field of image processing and image transmission, a shooting timing is generated based on the time synchronized among devices, and shooting (synchronized shooting) is performed while maintaining synchronization among a plurality of terminals (such as cameras) by using this shooting timing. Patent Document 1 describes a virtual viewpoint image generation system that performs synchronized shooting from a plurality of viewpoints using a plurality of cameras installed at different positions, and generates virtual viewpoint content using the images from the plurality of viewpoints obtained by this synchronized shooting. In order to perform such generation of virtual viewpoint images with high quality, it is required to perform synchronization of imaging timing with high precision. As a protocol for synchronizing time with high precision among a plurality of terminals, Precision Time Protocol (PTP) is widely used. On the other hand, Patent Document 2 describes, as a technique for transmitting image data, a technique of using an arbitrary frame as a reference frame at the time of encoding, accumulating the encoded bit stream obtained by encoding based on this reference frame in a medium, and transmitting it to a network.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Patent Document 2

Summary of the Invention

Problems to be Solved by the Invention

[0004] Generally, when an image captured by a camera or the like is transmitted over a network, multiple image processes are performed on the image data. The multiple image processes can be executed using, for example, pipeline processing. FIG. 19 shows an example of the operation of pipeline processing composed of multiple image processes. In FIG. 19, an example is shown in which image processing is executed in each of the image processing units A to C for the image data 1903 to 1911 and output to the network via the transmission unit. For example, in pipeline processing, a control signal that is a pulse signal of a predetermined period is input to each functional unit, and each functional unit can be controlled to perform processing at the timing of the pulse of the control signal. For example, at timing t1, the image processing unit A processes the image 1903 and outputs it to the image processing unit B. At timing t2, the image processing unit A and the image processing unit B process the images 1904 and 1903, respectively, and output them to the image processing unit B and the image processing unit C, respectively. The image data processed in this pipeline is finally packetized in the transmission unit and output to the network. Here, in FIG. 19, the output of the control signal is stopped after timing t7 and resumed at timing t1'. At this time, when images of a series different from the images processed so far (images 1903 to 1909), such as images 1910 and 1911, are to be processed, the images 1907 to 1909 may be output from the transmission unit before the images 1910 and 1911 reach the transmission unit. That is, the images 1907 to 1909 that were being processed before the stop can be output to the network even though they are different from the images of the series specified at the time of resumption. As a result, network resources are used by images that do not need to be transmitted originally, and images that the user does not desire may be displayed.

[0005] The present invention provides a technique for suppressing the output of unnecessary image data in an image processing system that executes multiple image processes.

Means for Solving the Problems

[0006] A terminal device according to an aspect of the present invention receives an input of an image data set composed of a plurality of pieces of image data, generates processed data by performing image processing on the image data, and performs a process of outputting the processed data to an output buffer. The process includes a processing unit that performs the process that requires a predetermined time from the input to the output, a transfer unit that reads the processed data from the output buffer and outputs it to a communication line, and a control unit that controls so that the processed data stored in the output buffer is not output to the communication line during a predetermined period from when the image data of the image data set is input to the processing unit until the first processed data of the image data set is output to the output buffer.

Advantages of the Invention

[0007] According to the present invention, it is possible to suppress the output of unnecessary image data in an image processing system that executes a plurality of image processes.

Brief Description of the Drawings

[0008]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Figure 16

Figure 17

Figure 18

Figure 19

Figure 20

Embodiments for Carrying Out the Invention

[0009] Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments do not limit the invention according to the claims. Although a plurality of features are described in the embodiments, not all of these plurality of features are essential to the invention, and the plurality of features may be arbitrarily combined. Further, in the accompanying drawings, the same or similar configurations are given the same reference numerals, and duplicate explanations are omitted.

[0010] (Embodiment 1) (System Configuration) Fig. 1 shows a configuration example of the image processing system 100 according to this embodiment. The image processing system 100 includes, for example, sensor systems 110a to 110z, an image processing apparatus 120, a time server 130, a hub 140, a control terminal 150, and a user terminal 160. The control terminal 150 executes control of each functional unit constituting the image processing system 100. For example, the control terminal 150 remotely manages the operating states of the sensor systems 110a to 110z, the image processing apparatus 120, etc. and performs control such as parameter setting via a network. Note that the control terminal 150 may be connected to the hub 140 in order to communicate signals for setting and controlling each functional unit of the image processing system 100. Communication between the control terminal 150 and other functional units via the hub 140 may use GbE (Gigabit Ethernet), 10 GbE, 100 GbE, etc. that conform to the IEEE802.3 standard established for Ethernet (registered trademark, hereinafter omitted). Further, it is not limited to these, and it may be an interconnect Infiniband, industrial Ethernet, or other types of networks, or these may be combined. The hub 140 transfers packets including images, control signals, etc. exchanged between each functional unit constituting the image system 100.

[0011] Sensor systems 110a to 110z generate images by taking pictures using cameras. In this embodiment, unless otherwise specified, the term "image" may include both videos and still images. That is, the image processing system 100 of this embodiment can process both still images and videos. In this embodiment, the sensor systems 110a to 110z may be referred to as sensor system 110 without distinction. Similarly, the camera adapters 111a to 111z and the cameras 112a to 112z included in the sensor systems 110a to 110z may be referred to as camera adapter 111 and camera 112, respectively, without distinction. FIG. 1 shows an example in which the image processing system 100 includes 26 sets of sensor systems 110a to 110z. The image processing system 100 may include fewer than 26 sets of sensor systems 110, or may include more than 26 sets of sensor systems 110. Also, the configurations of each sensor system 110 may be the same or may be different from each other. For example, components included in the configuration of some sensor systems 110 may not be included in other sensor systems 110. The sensor system 110 may include a plurality of cameras 112. The camera adapter 111 and the camera adapter 112 may be connected using one or more signal lines. For example, the signal line for controlling the camera 112 and the signal line for transmitting the data captured by the camera 112 may be different. Each of the cameras 112 in the sensor system 110 can be used to take pictures of the same subject from multiple directions. The performance and models of each camera 112 may be different. Also, the sensor systems 110a to 110z may be connected in series by a daisy chain. With this connection form, the number of connection cables can be reduced and the wiring work can be streamlined, so that the construction of the system can be simplified in the case of a large amount of image data accompanying high resolution and high frame rate (such as 4K and 8K) of images. Note that not all of the sensor systems 110 included in the image processing system 100 need to be daisy chain-connected.For example, the sensor system 110 included in the image processing system 100 may be divided into a plurality of groups, and daisy chain connection may be performed for each group. For example, in a building composed of a plurality of floors such as a stadium, the sensor system 110 may be grouped by floor and daisy chain connection may be performed. Within a floor, for example, the group may be subdivided by a half circumference unit of the stadium. For example, when a plurality of image processing devices 120 are arranged, grouping may be performed in units of the sensor system 110 connected to each image processing device 120. By grouping the sensor systems 110, system construction can be simplified even in places where wiring is difficult, and system construction can be performed flexibly.

[0012] The sensor system 110 is configured to include a camera adapter 111 and a camera 112. Note that the sensor system 110 may include other components, for example, a voice device such as a microphone or a pan-tilt unit that controls the orientation of the camera. Also, the sensor system 110 may include one or more camera adapters 111 and one or more cameras 112. The camera adapter 111 and the camera 112 may be integrally configured or may be configured separately. Also, the image processing device 120 may have a part of the functions of the camera adapter 111. An image captured by the sensor system 110 upstream in the daisy chain is transferred to the image processing device 120 via the sensor system 110 downstream in the daisy chain. The sensor system 110 upstream in the daisy chain is a sensor system 110 disposed at a relatively distant position as seen from the image processing device 120, for example, the sensor system 110z. The sensor system 110 downstream in the daisy chain is a sensor system 110 disposed at a relatively close position as seen from the image processing device 120, for example, the sensor system 110a. For example, an image captured by the camera 112z is processed in the camera adapter 111z, then packetized, and transferred via the daisy chain to the camera adapter 111y. The camera adapter 111y transfers the packet received from the sensor system 111z and the packet obtained by performing image processing on the image captured by the camera 112y and packetizing it to the downstream sensor system 110. Finally, the images captured by the cameras 112a to 112z are transferred to the image processing device 120 via the sensor system 110a and the hub 140. In the present embodiment, the daisy chain is one of the communication lines connecting the camera adapters 111.

[0013] The image processing device 120 reconstructs an image using the packets acquired from each of the sensor systems 110. For example, the image processing device 120 extracts the identifier of the camera adapter 111, the type of the image, the frame number, etc. from the packet header, and uses these to reconstruct the image. Note that the image processing device 120 has a storage device for storing the acquired images, and can associate and store the image data extracted from the packet and the information obtained from the header. Then, the image processing device 120 can read out the corresponding image from the stored images based on the information specifying the viewpoint designated from the user terminal 160 or the control terminal 150, perform rendering processing, and generate a virtual viewpoint image. Note that a part of the functions of the image processing device 120 may be possessed by the control terminal 150, the user terminal 160, or the camera adapter 111. The virtual viewpoint image generated by the rendering processing is transmitted to the user terminal 160 and can be displayed on the user terminal 160. Thereby, the image processing system 100 can provide an image of the viewpoint based on the designation of the user operating the user terminal 160. That is, the image processing device 120 can select an image related to the viewpoint designated by the user terminal 160 or the like from the images captured by the plurality of cameras 112, and generate a virtual viewpoint image using these images. Note that the virtual viewpoint image may be generated by a functional unit other than the image processing device 120 (for example, the control terminal 150 or the user terminal 160). In this case, the image processing device 120 can provide the image related to the designated viewpoint to that functional unit.

[0014] The time server 130 provides time information to each of the sensor systems 110. For example, the time server 130 may distribute communication packets (synchronization packets) containing time information for performing time synchronization with the sensor system 110. The sensor system 110 can perform synchronization among the plurality of cameras 112 based on the time information included in the synchronization packets. Performing synchronization among the plurality of cameras 112 is sometimes referred to as Genlock. By synchronizing each camera 112, the image data output by each camera 112 can be used in cooperation. For example, it becomes possible to select images taken at the same time from among the images taken by different cameras 112 stored in the image processing apparatus 120. Synchronization among the cameras 112 can be executed by the Transparent Clock function and the Ordinary Clock function of PTP (Precision Time Protocol) that the camera adapter 111 has. The Transparent Clock function is a function that measures the time taken for the transfer of PTP packets (synchronization packets) passing through the device and writes it into the header of the PTP packet for transfer. The Ordinary Clock function is a function for a device having a single network connection. That is, in order to synchronize among the plurality of cameras 112 in the plurality of sensor systems 110 connected by daisy chain, the camera adapter 111 calculates the residence time of the synchronization packet in its own device and writes it into the header. Then, the camera adapter 111 transfers the synchronization packet to the next camera adapter 111. As a result, each camera adapter 111 can synchronize with the time server 130 with high accuracy based on the received synchronization packet. By each camera adapter 111 synchronizing with the time server 130, the shooting timings of the respective cameras 112 can be synchronized. For example, the camera adapter 111 can control the shooting timing of the camera 112 based on the time synchronized with the time server 130. Also, the camera adapter 111 can generate the time information to be attached to the image taken by the camera 112 based on the time synchronized with the time server 130. Note that a redundant configuration using a plurality of time servers 120 may be performed to enhance the reliability of the time server 130.If a problem occurs in the time server 130, the imaging timings of the respective cameras 112 do not match, and thus a virtual viewpoint image cannot be generated. In this case, the times among a plurality of time servers 130 may be aligned using, for example, the Global Positioning System (GPS).

[0015] The camera adapter 111 generates image data to be used for creating virtual viewpoint content and the like using the images captured by the camera 112. FIG. 2 shows an example of the configuration of the camera adapter 111. In the present embodiment, the camera adapter 111b will be described as an example, but other camera adapters 111 may be configured similarly. The camera adapter 111 may include a CPU 201, an internal storage unit 202, a DMA unit 203, a DMA unit 204, a signal generation unit 205, a camera control unit 206, a communication interface (IF) unit 207, a communication IF unit 208, and an external storage unit 209. Further, the camera adapter 111 includes an image notification unit 210, a transmission instruction unit 211, a transmission unit 212, an image processing unit A 231, an image processing unit B 232, and an image processing unit C 233. In the present embodiment, the image processing units A 231 to C 233 may sometimes be collectively referred to as the image processing unit 230 without distinction. The internal storage unit 202 may include frame buffers A 221 to D 224. In the present embodiment, the frame buffers A 221 to D 224 may sometimes be collectively referred to as the frame buffer 220 without distinction. FIG. 2 shows an example using three image processing units and four frame buffers, but the respective numbers of the image processing units and the frame buffers may be less than or more than these. Each functional unit constituting the camera adapter 111 is connected to a system bus and can communicate with each other. Further, each functional unit has registers, executes various processes according to the set values of the registers, and can display its own processing status and processing results in the registers. Each functional unit can know the state of that functional unit by displaying the registers of other functional units.

[0016] The CPU 201 controls the entire camera adapter 111. Also, the CPU 201 performs time synchronization based on the exchange of synchronization packets with the time server 130. Further, the CPU 201 exchanges control packets with the control terminal 150. Additionally, the CPU 201 can receive a designation of image data to be processed from the user terminal 160 or the image processing apparatus 120, etc. Moreover, the CPU 201 can control the pipeline processing constituted by the image processing units A 231 to the image processing units C 233, etc.

[0017] The internal storage unit 202 is a memory that holds programs executed by the CPU 201 for controlling the camera adapter 111, etc., and synchronization packets and control packets transmitted and received by the CPU 201 with other devices, etc. Also, the internal storage unit 202 stores the imaging data acquired by the camera control unit 206 from the camera 112b. Further, the image data input to the image processing unit 230 and the image data output by the image processing unit 230 are stored in the frame buffers A 221 to the frame buffers D 224 of the internal storage unit 202. In FIG. 2, for each of the frame buffers 220, one storage area is shown, but it may have areas capable of storing a plurality of image data independently. For example, the frame buffer 220 may divide one storage area for use according to the application, or may have a plurality of memories for storing a plurality of image data. The types of each memory may be different. Note that since the frame buffer A 221 is a buffer into which the image data to be subjected to a series of image processing is first input, it can also be called an input buffer. Also, since the frame buffer D 224 is a buffer from which the image data subjected to a series of image processing is output, it can also be called an output buffer.

[0018] The DMA unit 203 and the DMA unit 204 execute Direct Memory Access (DMA) for transmitting and receiving packets to and from other camera adapters. For example, the DMA unit 203 and the DMA unit 204 transmit and receive packets based on instructions from the CPU 201 or the transmission unit 212. The transmission unit 212 can issue a transfer instruction for a packet of image data. The CPU 201 can issue a transfer instruction for a synchronization packet. As an example, the DMA unit 203 transmits and receives packets to and from the camera adapter 111c via the communication IF 207. The DMA unit 203 reads a packet from a specified area (for example, the internal storage unit 202) and transfers it to the communication IF 207. Also, the DMA unit 204 transfers a packet received from the camera adapter 111a to a specified area (for example, the internal storage unit 202).

[0019] The communication IF units 207 and 208 transmit and receive communication packets (synchronization packets, TCP / IP packets, image data packets, etc.) exchanged between the camera adapter 111 and other functional units. For example, the communication IF units 207 and 208 can execute the processing of the first layer and the second layer of the OSI (Open Systems Interconnection) reference model. The communication IF unit 207 inputs the PTP packet (synchronization packet) received from the time server 130 via the camera adapter 111a to the DMA unit 204. The PTP packet input to the DMA unit 204 is transferred to the internal storage unit 202 based on the transfer instruction by the CPU 201, protocol processing is performed by the CPU 201, and it is input to the DMA unit 203. The communication IF unit 207 transfers the PTP packet input to the DMA unit 203 to the camera adapter 111c. On the other hand, the communication IF unit 208 inputs the PTP packets received from the camera adapters 111c to 111z to the DMA unit 203. The PTP packet input to the DMA unit 203 is transferred to the internal storage unit 202 based on the transfer instruction according to the instruction of the CPU 201, protocol processing is performed by the CPU 201, and it is input to the DMA unit 204. The communication IF 208 transfers the PTP packet input to the DMA unit 204 to the time server 130 via the camera adapter 111a. Note that the PTP packet generated by the CPU 201 is also input to the DMA unit 204 in the same way, and the communication IF unit 208 transfers this PTP packet to the time server 130 via the camera adapter 111a. Similarly, for the packets of the image data generated by the transmission unit 212, the communication IF unit 208 acquires them from the DMA unit 204 and transfers them to the camera adapter 111a. Note that when the communication IF units 207 and 208 transmit and receive PTP packets, they can hold the time of the clock inside their own functions (timestamp function). This timestamp function can be used in the CPU 201 for the calculation of time synchronization. With the timestamp function, the CPU 201 can accurately determine the transmission and reception times of the PTP packets and the residence time in the camera adapter 111.The time difference between the time server 130 and the camera adapter 111 determined by the protocol processing of PTP by the CPU 201 can be used to correct the clocks of both the communication IF unit 207 and the communication IF unit 208. Note that before starting time synchronization, time synchronization may be performed between the respective clocks of the communication IF unit 207 and the communication IF unit 208.

[0020] The signal generation unit 205 generates a control signal for the respective functional units to operate synchronously. For example, the signal generation unit 205 can generate a control signal based on the clock maintained by the communication IF unit 208. Further, the signal generation unit 205 can cause the CPU 201 to acquire the number of generated control signals (pulse generation times) by displaying the number of generated control signals (pulse generation times) in a register. Also, the signal generation unit 205 can be made to control the initialization of the pulse generation times via the register. The operation flow of the signal generation unit 205 will be described later.

[0021] The camera control unit 206 controls the camera 112b. For example, the camera control unit 206 outputs a Genlock signal for the camera 112b to perform Genlock and a Timecode for adding time information to the image captured by the camera 112b to the camera 112b. The Genlock signal and the Timecode can be generated based on the reference signal generated by the signal generation unit 205. Note that the camera control unit 206 can be controlled by the CPU 201 that has received an instruction from the control terminal 150. The camera control unit 206 can receive the imaging data (RAW format image data) in which the camera 112b has captured an image in synchronization with the Genlock signal and has been given a Timecode, and transfer it to the internal storage unit 202 or the external storage unit 209. For example, when the imaging data is transferred to the frame buffer A 221 of the internal storage unit 202, it may be the subject of pipeline processing. In the present embodiment, the description will be made using the imaging data stored in the external storage unit 209, and the method of performing pipeline processing and transmission using the imaging data received from the camera 112b will be omitted from the detailed description.

[0022] The image processing unit A231 reads, for example, the RAW format image data stored in the frame buffer A221, converts it into a BAYER format image, and writes it to the frame buffer B232. When the reset of the image processing unit A232 is released, the pointer of the address for reading the image data from the frame buffer A221 and the pointer of the address for writing the image data to the frame buffer B222 are initialized. Also, these address pointers can be recognized by other functional blocks through registers.

[0023] The image processing unit B232 reads the BAYER format image data stored in the frame buffer B222, performs anti-shake processing, and writes it to the frame buffer C233. Note that the anti-shake processing is a process of correcting image shake such as hand shake during shooting. The pointer of the address for reading the image data from the frame buffer B222 and the pointer of the address for writing the image data to the frame buffer C223 can be automatically calculated based on the address pointer in the processing of the image processing unit A231. For example, the pointer of the address for reading the image data from the frame buffer B222 can correspond to the address written by the image processing unit A231. Also, these address pointers can be recognized by other functional blocks through registers.

[0024] The image processing unit C233 reads the image data after the vibration control processing from the frame buffer C222 and separates it into a foreground area and a background area. The foreground area is a moving object such as a person, and the background area can be an area other than the foreground area. Then, the image processing unit C233 can write only the image of the foreground area (foreground image) to the frame buffer D224. Note that the image processing unit C233 may write the image of the background area (background image) to the frame buffer D224. The pointer of the address for reading the image data from the frame buffer C223 and the pointer of the address for writing the image data to the frame buffer D224 can be automatically calculated based on the pointer of the address in the processing of the image processing unit B232. For example, the pointer of the address for reading the image data from the frame buffer C223 can correspond to the address written by the image processing unit B232. Also, these pointers of the addresses can be recognized by other functional blocks through registers.

[0025] Each image process executed by the above image processing unit 230 is an example, and each image processing unit 230 can execute a process different from the above. The image data subjected to each image process can be called processed data. Also, in the above, an example in which the processing result of each image processing unit 230 is passed to the next image processing unit 230 using the frame buffer 220 has been described, but this technology can be applied to any process in which a plurality of image processes are performed on the image data.

[0026] The image notification unit 210 notifies the transmission instruction unit 211 of information specifying the image data to be transmitted. For example, when the image notification unit 210 detects that the image processing unit C233 has stored a foreground image (or background image) in the frame buffer D224, the image notification unit 210 can notify the transmission instruction unit 211 of the size (data length) of the image and the address where the image is stored. The pointer of the address where the image data detected by the image notification unit 210 is stored can be automatically calculated based on the pointer of the address in the processing of the image processing unit C233. For example, the pointer of the address for reading the image data from the frame buffer D224 can correspond to the address written by the image processing unit C233.

[0027] Based on the information notified by the image notification unit 210, the transmission instruction unit 211 instructs the transmission unit 212 to transmit the image data. For example, the transmission instruction unit 211 can identify the image data based on the pointer of the address where the image data notified by the image notification unit 210 is stored and the image size. In addition, the transmission instruction unit 211 can notify the transmission unit 212 based on the control signal input from the signal generation unit 205.

[0028] Based on the instruction from the transmission instruction unit 211, the transmission unit 212 packetizes the image data and outputs it to the communication line. For example, the transmission unit 212 acquires the image data instructed by the transmission instruction unit 211, performs packetization, and stores it in an area accessible by the DMA unit 203 or the DMA unit 204. The packetized image data can be called an image packet. In addition, the image packet can be stored in the internal storage unit 202 or the external storage unit 209. Note that the image packet may be stored in the transmission unit 212. For example, the transmission unit 212 can receive an image packet from another upstream camera adapter 111 via the DMA unit 203 and transfer the received image packet to a downstream camera adapter 111 by instructing the DMA unit 204.

[0029] The external memory unit 209 is a storage device that stores a large amount of image data captured by the camera 112b. For example, the external memory unit 209 can be a Solid State Drive (SSD), a Hard Disk Drive (HDD), or the like. The external memory unit 209 can manage the image data as imaging data grouped for each capture by the camera. FIG. 3 shows an example of the data configuration of the image data captured by the camera 112b. For example, image data 311 to image data 313 are image data generated when shooting for one minute with a 60 Frame Per Sec (FPS) setting. The imaging data 301 can be composed of 3600 pieces of image data 311 to 313 at 60 FPS × 60 seconds and Timecodes 321 to 323 associated with each piece of image data. The Timecodes 321 to 323 indicate, for example, the times at which the associated image data 311 to 313 were captured. The image data 311 to 313 may be referred to as image data 310 without distinction. Also, the Timecodes 321 to 323 may be referred to as Timecode 320 without distinction. Also, the imaging data 301 to 303 may be referred to as imaging data 300 without distinction. A sequence number for distinguishing each piece of imaging data may be assigned to each piece of imaging data. The CPU 201 can specify the sequence number and Timecode of the imaging data corresponding to each image in order to specify the first image and the last image to be subject to pipeline processing. For example, the CPU 201 can specify, as the first image to be subject to pipeline processing, Timecode = 10:29:30.0 with sequence number #3, and as the last image, Timecode = 10:30:29.59 with sequence number #3. For example, Timecode = 10:29:30.0 may be the first frame at 10:29:30, and Timecode = 10:30:29.59 may be the 59th frame at 10:30:29. In this way, a plurality of image data between the first Timecode and the last Timecode specified by the CPU 201 can be called an image data set.The CPU 201 controls the operations of each functional block so that pipeline processing of the image data is executed while incrementing the Timecode one by one from the specified first image data to the last image data. Note that the CPU 201 can specify only the first image data as the target of the pipeline processing. In this case, the pipeline processing can execute image processing by sequentially calling the image data after this first image data. Then, if another image data set is specified by the user or the like during the pipeline processing, the pipeline processing can target the image data included in the specified other image data set. In this case, a series of image data that was the target of the pipeline processing before the other image data was specified can be called the first image data set, and a series of image data starting from the other image data can be called the second image data set.

[0030] A pipeline process including a plurality of image processes in this embodiment will be described. FIG. 4 shows an example of functional units constituting the pipeline process. In FIG. 4, an example will be described in which image data sets stored in the external storage unit 209 are read by the CPU 201, the pipeline process is executed, and output by the transmission unit 212. The pipeline process is executed based on a control signal A401 generated by the signal generation unit 205. The control signal A401 is notified to the image processing unit A231 to the image processing unit C233 and the image notification unit 210. Note that the CPU 201 determines whether a pulse of the control signal A401 has occurred by polling the number of pulse generations of the control signal A401 displayed in the register of the signal generation unit 205, and can start processing. Note that the control signal A401 may be notified to the CPU 201, and the CPU 201 may start processing based on this notification. These functional units detect the rising edge timing of the input control signal A401 and start their respective processes. The CPU 201 identifies an image data set to be the target of the pipeline process based on a designation from the user terminal 160 or the like, and reads the first image data 310 of the image data set from the external storage unit 209. The CPU 201 stores the read image data 310 in the frame buffer A221. Further, the CPU 201 sequentially stores the image data 310 in the frame buffer A221 in accordance with the cycle of the pipeline process. The image processing unit A231 reads the image data stored by the CPU 201 from the frame buffer A221, performs image processing, and stores it in the frame buffer B222. The image processing unit B232 reads the image data stored by the image processing unit A231 from the frame buffer B222, performs image processing, and stores it in the frame buffer C223. The image processing unit C233 reads the image data stored by the image processing unit B232 from the frame buffer C223, performs image processing, and stores it in the frame buffer D224. The image notification unit 210 reads the address of the image data and the data length of the image data stored by the image processing unit C233 from the frame buffer D224, and notifies the transmission instruction unit 211. The processing up to this point is executed by a pipeline operation based on the control signal A401.Based on the control signal B402 generated by the signal generation unit 205, the transmission instruction unit 211 notifies the transmission unit 212 of the address and data length of the image data notified by the image notification unit 210. Based on the address and data length notified by the transmission instruction unit 211, the transmission unit 212 acquires the image data from the frame buffer D224. The transmission unit 212 packetizes the acquired image data to generate an image packet, controls the DMA unit 203 or the DMA unit 204, and outputs it to the communication line. Note that FIG. 4 is an example of a configuration diagram for executing pipeline processing, and the configuration for performing image processing is not limited to this. For example, a plurality of functional units may be integrated and configured as one functional unit, or some functions may be omitted.

[0031] The relationship between the control signal A401 and the control signal 402 will be described with reference to FIGS. 5 and 6. FIG. 5 shows an example of the functional configuration of the signal generation unit 205. The signal generation unit 205 includes a clock unit 501, a reference signal generation unit 502, and a control signal generation unit 503. The clock unit 501 maintains time information. For example, the clock unit 501 acquires the time information A504 from the communication IF unit 208 and adjusts the internal clock. For example, the clock unit 501 adjusts the internal clock to match the time information A504. Based on the internal clock, the clock unit 501 generates time information B505 and outputs it to the reference signal generation unit 502. The reference signal generation unit 502 uses the time information B505 to generate a reference signal 506 with a certain period and outputs it to the control signal generation unit 503. The certain period can be, for example, 1 Hz, 30 Hz, etc. The control signal generation unit 503 generates the control signal A401 and the control signal B402 using the received reference signal 506.

[0032] FIG. 6 is an example of a diagram showing the relationship between the reference signal 506, the control signal A401, and the control signal B402. The reference signal 506 is a rectangular signal with a constant period. The control signal A401 is a pulse signal generated at the timing of the rising edge and the falling edge of the reference signal 506. The control signal B402 is a pulse signal obtained by delaying the control signal A401 by a predetermined time. That is, the control signal B402 is output to the transmission instruction unit 211 with a delay of a predetermined time from the control signal A401 serving as a reference for pipeline processing. As a result, after the image-processed image data in the pipeline processing is stored in the frame buffer D224, the transmission unit 212 acquires the image data from the frame buffer D224. For example, the output of the control signal B402 can be set to be output after the image notification unit 210 determines the presence or absence of the image data in the frame buffer D224 and notifies the transmission instruction unit 211 of the address and data length of the image data. For this purpose, the predetermined time from the output of the control signal A401 to the output of the control signal B402 can be set to a value larger than the measurement result of the time required for the image notification unit 210 to determine the presence or absence of the image data and notify the transmission instruction unit 211 at the time of system construction. The control signal A401 and the control signal B402 are pulse signals generated at a period of 1 / 2 of the period of the reference signal 506. For example, when the frequency of the reference signal 506 is 30 Hz, the frequencies of the control signal A401 and the control signal B402 are 60 Hz. As a result, pipeline processing with a processing speed of 60 FPS can be executed using the 30 Hz reference signal 506. The processing speed of the pipeline can be arbitrarily changed by changing the period of the reference signal 506 or using a divided signal of the reference signal 506. In the present embodiment, it is assumed that the control signals A401 and B402 are output at periods such as 60 Hz, 59.94 Hz, 50 Hz, 30 Hz, 29.97 Hz, 25 Hz, which are typical frequencies as the image frame rate of the camera, and at periods of 1 Hz and 2 Hz. These periods can be notified from the CPU 201 to the signal generation unit 205 at the timing of the start of generation of the control signal A401 and the control signal B402.

[0033] In this way, the CPU 201 sequentially stores the image data 310 specified by the user in the frame buffer A 221, and controls the pipeline processing based on the control signal A 401 generated by the signal generation unit 205 and the output of the image packet 311 based on the control signal B 402. On the other hand, for example, when the pipeline processing is interrupted midway due to a stop instruction from the user, the processed image data 310 remains in the frame buffers A 221 to D 224. After that, even if a different image data set is specified at the time of a start instruction from the user, these image data 310 will be sequentially sent to the communication line when the next pipeline processing starts. For this reason, the image data 310 that does not need to be transmitted originally will use the resources of the communication line, and an image that the user does not desire may be displayed. Note that by flushing the image data stored in the frame buffers A 221 to D 224 at the timing when the CPU 201 stops the control signal, the influence of the remaining image data 310 can be avoided. However, when it is impossible to distinguish whether the interruption of the processing is due to a pause or a complete stop, the image data 310 that should be output originally will also be deleted when restarting after the pause. Also, when a system reset is required to erase the image data 310, it will take time to recover from the system reset each time.

[0034] In view of such circumstances, in this embodiment, by executing a process for preventing the transmission of unnecessary images, the past processed image data 310 stored in the frame buffer is prevented from being output to the communication line. For example, while performing pipeline processing on a certain first image data set, if a stop instruction is received from the user and then a start instruction is received from the user, assume that the target of the pipeline processing becomes a new second image data set. That is, assume that a switching instruction is given to switch the target of the pipeline processing from the first image data set to the second image data set. In this case, the CPU 201 instructs to mask the control signal B402 a predetermined number of times so that the processed first image data set is not output from the transmission unit 212. Since the output instruction is not notified to the transmission instruction unit 211 because the control signal B402 is not output, the transmission unit 212 does not read the image data. Thereby, it is possible to avoid outputting unnecessary image data. Here, when the image processing is executed by pipeline processing based on a common timing signal, the CPU 201 can identify a predetermined period (latency) required for the pipeline processing, and based on this, can identify the number of times (masking times) the control signal B402 should be masked. After the CPU 201 stores the first image data 310 of the image data set in the frame buffer A221, when a time corresponding to the latency required for the pipeline processing has elapsed, the image data subjected to each image processing is stored in the frame buffer D224. For example, the CPU 201 identifies the masking times based on the number of functional units included in the pipeline processing and notifies the control signal generation unit 503. In this case, the number of image processes included in the pipeline processing can be used as the masking times. Note that the identification of the predetermined period required for the pipeline processing by the CPU 201 may include the identification of the number of events occurring at a predetermined cycle such as the identification of the number of control signals output for executing the pipeline processing. Hereinafter, a hardware configuration capable of realizing this embodiment and an example of the operation flow of the functional units of the camera adapter 111 shown in FIG. 2 will be described.

[0035] (Hardware Configuration) Fig. 20 shows an example of the hardware configuration of the camera adapter 111 according to this embodiment. As an example of its hardware configuration, the camera adapter 111 has, for example, a storage unit 2001, a control unit 2002, a functional unit 2003, an input unit 2004, an output unit 2005, and a communication unit 2006. The storage unit 2001 is composed of one or more memories including ROM, RAM, etc., and may store control programs for various operations of each functional unit constituting the camera adapter 111 and various information such as parameters for the operation of the camera adapter 111. ROM and RAM are abbreviations for Read Only Memory and Random Access Memory respectively. As the storage unit 2001, in addition to memories such as ROM and RAM, it may be configured to include storage media such as flexible disks, hard disks, optical disks, magneto-optical disks, CD-ROMs, CD-Rs, magnetic tapes, non-volatile memory cards, DVDs, etc. For example, the external storage unit 209 and the internal storage unit 202 are realized by the storage unit 2001. The control unit 2002 is composed of one or more processors including, for example, a CPU, an MPU, etc., and controls the entire device of the camera adapter 111 by executing the control program stored in the storage unit 2001. Note that the control unit 2002 may control the entire device of the camera adapter 111 in cooperation with the control program stored in the storage unit 2001 and the OS (Operating System). Note that CPU and MPU are abbreviations for Central Processing Unit and Micro Processing Unit respectively. When the control unit 2002 has a plurality of processors that can be implemented by a multi-core or the like, the entire device of the camera adapter 111 may be configured to be controlled by the plurality of processors. For example, the CPU 201 may be realized by the control unit 2002. The functional unit 2003 is hardware for the camera adapter 111 to execute processing necessary for the operation of each functional unit such as image processing.

[0036] The input unit 2004 receives various operations from the user. The output unit 2005 performs various outputs to the user via a monitor screen, a speaker, etc. Here, the output by the output unit 2005 may be a display on the monitor screen, an audio output by the speaker, a vibration output, etc. Note that the input unit 2004 and the output unit 2005 may both be realized by one module, such as a touch panel display. Also, the input unit 2004 and the output unit 2005 may each be a device integrated with the camera adapter 111, or may be separate devices. The communication unit 2006 is, for example, GbE (Gigabit Ethernet), 10GbE, 100GbE, etc. that comply with the IEEE802.3 standard established for Ethernet (registered trademark, hereinafter omitted). Also, it is not limited to these, and may be an interconnect Infiniband, industrial Ethernet, or other types of networks, or these may be combined. For example, the communication IF unit 207 and the communication IF unit 208 may be realized by the communication unit 2006.

[0037] (Operation Flow) The operation flow of each functional unit will be described. FIG. 7 shows an example of the operation flow when the CPU 201 controls pipeline processing. This flow starts when the power of the camera adapter 111 is turned on and the CPU 201 is activated. The CPU 201 first releases the reset of each functional unit and executes initialization such as initial settings of each functional unit and initialization of registers and the like. For example, in the initialization by the CPU 201, setting of image processing parameters of the image processing unit A 231, the image processing unit B 232, and the image processing unit C 233, designation of the storage destination of the imaging data of the camera 112 in the camera control unit 206, and the like can be performed. As an example, the camera control unit 206 can operate to store the imaging data in the frame buffer A 221 in the first state after the reset is released. In the present embodiment, the camera control unit 206 performs settings so as to write the imaging data in an area other than the frame buffer 231A and an area not used by other frame buffers. Note that these settings may be executed by the CPU 201 before the image data is stored in the frame buffer A 221. Further, the CPU 201 performs time synchronization with the time server 130. For example, the CPU 201 performs time synchronization using PTP and maintains synchronization with the time server 130 using PTP thereafter. Then, the CPU 201 waits for an input from the user.

[0038] First, the CPU 201 determines whether it has received a designation of an image to be processed from the user (S701). For example, a predetermined image can be designated from among the image data 310 stored in the external storage unit 209 by the user. The predetermined image can be designated by the Timecode 320 associated with each of the image data 310 for starting the pipeline process and the image data 310 for ending the pipeline process, and the sequence number of the imaging data 300 in which they are included. When the CPU 201 receives a designation of an image from the user (YES in S701), it stores the designated image data 310 in the frame buffer A 221, initializes the image processing unit A 231, and waits for an instruction to start the pipeline process from the user (S702). Note that the CPU 201 can store a plurality of image data 310 in the frame buffer A 221. For example, when the number of buffer planes of the frame buffer A 221 is N, the CPU 201 can store up to N image data 310 in the frame buffer A 221. When the image processing unit A 221 is initialized, the image processing unit A 221 can read the image data 310 in order from the initial address. Therefore, the CPU 201 can store a plurality of image data 310 in order from the head buffer plane of the frame buffer A 221 according to the order in which they should be read. For example, assume that the image data 310 designated by the user has a sequence number of 5 for the imaging data, a Timecode from 9:50:30.0 to 9:51:30.0, and the number of buffer planes of the frame buffer A 221 is 16. In this case, the CPU 201 first stores the image data 310 with a sequence number of 5 for the imaging data and a Timecode of 9:50:30.0 in the head buffer plane of the frame buffer A 221. Then, the CPU 201 stores the image data 310 with a Timecode of 9:50:30.1 in the next buffer plane, and by repeating this, it can store up to 16 image data in the frame buffer A 221. Also, when the CPU 201 does not receive a designation of an image from the user and does not receive either an instruction to start or end the pipeline process from the user (NO in any of S701, S703, and S704), it waits for a reception of an instruction from the user.

[0039] When there is a start instruction from the user (YES in S703), the CPU 201 performs a process to prevent the transmission of unnecessary images (S706). The start instruction may include information on the time to start the pipeline process and information to specify the period of the pipeline process. In the present embodiment, as a process to prevent the transmission of unnecessary images, the CPU 201 instructs the signal generation unit 205 to mask the control signal B402 a predetermined number of times. Masking of the control signal means, for example, not outputting the control signal, or outputting a signal with a value of zero, etc., so as not to output a pulse signal or the like that should originally be output. By performing the process to prevent the transmission of unnecessary images, it is possible to avoid the image data left in each frame buffer from being output to the communication circuit due to interruption of the pipeline process or the like. On the other hand, when an end instruction is received from the user without receiving a start instruction from the user (NO in S703 and YES in S704), the CPU 201 notifies each functional unit of the end instruction and ends the process (S705).

[0040] The CPU 201 instructs the signal generation unit 205 to start generating the control signal A 401 and the control signal B 402 (S707). For example, this instruction to start generation may include information on the time when the pipeline process is to be started input by the user and information specifying the period of the pipeline process. The register of the signal generation unit 205 resets the display of the number of pulse generations of the control signal A 401 and the control signal B 402 and starts counting from 0. The reset of the register can be executed by the CPU 201. The CPU 201 sequentially stores the image data 310 to be input to the pipeline process in the frame buffer A 221 until the last image data 310 specified by the user is input to the frame buffer A 221. For this purpose, the CPU 201 monitors whether the pulse signal of the control signal A 401 has been output. When the pulse signal is detected (YES in S708), the CPU 201 determines whether the last image data 310 has been stored in the frame buffer A 221. If the last image data 310 has not been stored in the frame buffer A 221 (NO in S709), the CPU 201 stores the next image data 310 in the frame buffer A 221 (S711). For example, the CPU 201 can determine based on whether the image data 310 with a sequence number of 5 and a Timecode of 9:51:30.0 in the imaging data has been stored in the frame buffer A 221. If the CPU 201 first arranges 16 pieces of image data (9:50:30.0 to 9:50:30.15) in the frame buffer A 221, the Timecode of the image data to be stored in the frame buffer A 221 next is 9:50:30.16. That is, the CPU 201 stores in the frame buffer A 221 the image data 310 corresponding to the value obtained by adding 1 to the value of the Timecode 320 of the last image data 310 stored in the frame buffer A 221. On the other hand, if the last image data 310 has been stored in the frame buffer A (YES in S709), the CPU 201 waits until the last image data 310 is output by the transmission unit 212. When the last image data 310 is notified to the transmission unit 212 and output to the communication line, the CPU 201 instructs the signal generation unit 205 to stop generating the control signal A 401 and the control signal A 402 (S713).Then, the CPU 201 returns to S701 and waits for an instruction input from the user. Note that the CPU 201 can recognize that the last image data 310 has been notified to the transmission unit 212 by storing the last image data 310 in the frame buffer A221 and then waiting for a time corresponding to the latency of the pipeline processing. That is, since the pipeline processing of the image data is always performed by a certain number of functional units, the time (latency) required for the processing is always constant. For example, the CPU 201 can calculate the latency by multiplying the number of functional units included in the pipeline processing by the processing cycle of the pipeline processing (the output cycle of the control signal A401). On the other hand, if a stop instruction is received from the user before the last image data is stored in the frame buffer A221 (NO in S708 and YES in S710), the CPU 201 instructs the signal generation unit 205 to stop generating the control signal (S713). Then, the CPU 201 returns to S701 and waits for an instruction input from the user.

[0041] In this way, for example, when the pipeline processing of the first image data set is being executed and a stop instruction from the user is received (YES in S710), the CPU 201 waits for an instruction input from the user. At this time, it is possible to receive a designation of the second image data set from the user at S701 and a start instruction from the user at S703. In this case, the CPU 201 executes the masking process of the control signal B402 a predetermined number of times so that the processed data of the first image data set is not output to the communication line at S706. In this way, control can be performed so that the processed data is not output until the second image data set is input to the frame buffer A221 and the processed data is output to the frame buffer D224.

[0042] An example of the operation flow when the signal generation unit 205 generates the control signal A401 and the control signal B402 will be described with reference to FIGS. 8 and 9. First, FIG. 8 shows an example of the operation flow when the reference signal generation unit 502 generates a reference signal 506 that serves as a reference when generating the control signal A401 and the control signal B402. This flow starts when the reset of the signal generation unit 205 is released and the initial settings are completed. Note that at the start of the operation flow, the value of the reference signal 506 can be LOW (value is 0). When the reference signal generation unit 502 receives an instruction to start generating the control signal A401 and the control signal B402 from the CPU 201 (YES in S801), it waits until the time information B505 output from the clock unit 501 reaches the generation start time of the reference signal 506. The generation start time of the control signal A401 and the control signal B402 is, that is, the start time of the pipeline process. Note that the reference signal generation unit 502 may have an internal clock that is maintained in synchronization with the input time information B505. In this case, the reference signal generation unit 502 can execute the next step when this clock reaches the generation start time of the reference signal 506. If the reference signal generation unit 502 receives an end instruction before receiving the generation start instruction from the CPU 201 (NO in S801 and YES in S803), it ends the operation. When the time information B505 reaches the generation start time (YES in S802), the reference signal generation unit 502 reverses the output reference signal from LOW to HIGH (value is 0), and then calculates and holds the time (inversion time) when the value of the reference signal is changed. For example, the cycle information of the pipeline process input from the CPU 201 can be used for the calculation of the inversion time. As an example, when the cycle 2 Hz is input as the cycle information, since the reference signal 506 outputs a signal of 1 Hz (cycle is 1 second), it holds the time 50 milliseconds ahead as the inversion time. Note that if there are design differences such as the control signal generation unit 503 dividing the reference signal 506 using one or both of the rising edge and the falling edge, a reference signal 506 that conforms to that design can be generated. If the reference signal generation unit 502 receives a stop instruction from the CPU 201 before the time information B505 reaches the generation start time (YES in S804), it returns to S801 and waits for the generation start instruction.Subsequently, each time the signal generation unit 205 reaches the inversion time held by the time information B505 (YES in S806), the operation of inverting the reference signal and holding the next inversion time is repeated until a stop instruction is received from the CPU 201 (S807, S808). When the reference signal generation unit 502 receives a stop instruction from the CPU 201 (YES in S808), it sets the reference signal to LOW (S809), returns to S801, and waits for an instruction from the CPU 201. In this way, the reference signal generation unit 502 generates the reference signal 506 and outputs it to the control signal generation unit 503.

[0043] FIG. 9 shows an example of an operation flow when the control signal generation unit 503 of the signal generation unit 205 generates the control signal A401 and the control signal B402. This flow starts when the reset of the signal generation unit 205 is released and the initial setting is completed. First, when the control signal generation unit 503 receives an instruction from the CPU 201 to mask the control signal B402 (YES in S901), it holds the number of mask times (S902). The number of mask times is the number of times to mask the control signal B402. This instruction from the CPU 201 is notified based on S706 to prevent the transmission of unnecessary images. That is, while the first image data 310 specified by the user has not reached the image notification unit 210, if the control signal B402 is output, the image data 310 processed in the past may be output from the transmission unit 212. To avoid this, the output of the control signal B402 is masked until the control signal A401 is output a predetermined number of times. The number of mask times may be instructed by the CPU 201 or may be held by the signal generation unit 205 as a preset fixed value. Note that since the predetermined number of times is a fixed value based on the number of image processes included in the pipeline process, the latency from when the first image data 310 specified by the user is input to the frame buffer A221 until it reaches the image notification unit 210 is a fixed value. When the control signal generation unit 503 receives an end instruction from the CPU 201, it ends the operation (YES in S903). Also, when the control signal generation unit 503 receives a generation start instruction from the CPU 201, it starts detecting the rising edge or falling edge of the reference signal 506 (YES in S904). When the control signal generation unit 503 detects the rising edge or falling edge of the reference signal 506, it outputs the control signal A401 (S906). The control signal A401 is notified to the image processing unit A231 to the image processing unit C233 and the image notification unit 210. When a predetermined time has elapsed after the control signal generation unit 503 outputs the control signal A401 (YES in S907), it determines whether the held number of mask times is 1 or more. If the number of mask times is 0 (NO in S908), it outputs the control signal B402 (S909). The predetermined period between when the control signal A401 is output and when the control signal B402 is output is the predetermined period shown in FIG. 6.The scheduled period may be specified in a register as an initial setting, or may be specified from the CPU 201 together with an instruction to start generation. On the other hand, when the number of mask times held is 1 or more (YES in S908), the control signal generation unit 503 does not output the control signal B402 and decreases the number of mask times by one (S910). The control signal generation unit 503 repeats the operations of S905 to S910 until it receives a stop instruction from the CPU 201 (NO in S911), and when it receives a stop instruction from the CPU 201 (YES in S911), it returns to S901 and waits for an instruction from the CPU 201.

[0044] Subsequently, the operation flow of each functional unit constituting the pipeline processing will be described. FIG. 10 shows an example of the operation flow of the image processing unit A231. This flow starts when the reset of the image processing unit A231 is released and the initial setting is completed. When the image processing unit A231 receives the control signal A401 (YES in S1001), it reads out the image data 310 stored in the frame buffer A221, performs image processing (S1003), and stores it in the frame buffer B222 (S1004). The image processing unit A231 repeats the operations of S1001, S1003, and S1004 until it receives an end instruction from the CPU 201. When the image processing unit A231 receives an end instruction from the CPU 201, it ends the operation (YES in S1002). Note that the image processing unit B232 and the image processing unit C233 operate in the same manner except that the frame buffers used and the image processing to be executed are different.

[0045] FIG. 11 shows an example of the operation flow of the image notification unit 210. This flow starts when the reset of the image notification unit 210 is released and the initial settings are completed. When the image notification unit 210 receives the control signal A401 (YES in S1101), it determines the presence or absence of the image data 310 stored in the frame buffer D224. If the image data 310 is stored (YES in S1103), the image notification unit 210 notifies the transmission instruction unit 211 of the address and data length of the image data 310 (S1104). For example, the image notification unit 210 can determine the presence or absence of the image data by accessing the address of the automatically calculated pointer to determine whether the header information of the image data 310 is configured in a predetermined format. If the header information of the image data 310 includes information on the data length (total data size) of this image data 310, the image notification unit 210 can obtain the address and data length of the image data 310 stored in the frame buffer D224. Here, the address of the image data 310 is the starting address where the image data 310 is stored. On the other hand, if the image notification unit 210 determines that there is no image data 310 in the frame buffer D224 (NO in S1103), it can notify the transmission instruction unit 211 with the data length of the image data 310 set to 0 (S1105). The image notification unit 210 repeats the operations of S1101, S1103, S1104, and S1105 until it receives an end instruction from the CPU 201. When the image notification unit 210 receives an end instruction from the CPU 201 (YES in S1102), it ends this operation.

[0046] FIG. 12 shows an example of the operation flow of the transmission instruction unit 211. This flow starts when the reset of the transmission instruction unit 211 is released and the initial settings are completed. When the transmission instruction unit 211 receives the control signal B402 (YES in S1201), it notifies the transmission unit 212 of the address and data length notified by the image notification unit 210 (S1203). The image notification unit 210 repeats the operations of S1201 and S1203 until it receives an end instruction from the CPU 201, and when it receives an end instruction from the CPU 201 (YES in S1202), it ends the operation.

[0047] FIG. 13 shows an example of the operation flow of the transmission unit 212. This flow starts when the reset of the transmission unit 212 is released and the initial settings are completed. When the transmission unit 212 receives a notification of the image data 310 from the transmission instruction unit 211 (YES in S1301), it determines whether the notified data length is 0. If the data length is a value other than 0 (NO in S1302), the transmission unit 212 packetizes and outputs the image data 310 (S1303). The transmission unit 212 accesses the notified address to acquire the image data 310, and outputs the packetized image packet 310 to the communication line via the DMA unit 204. Note that when the size of the image data 310 is large, the transmission unit 212 can fragment and packetize the image data 310. On the other hand, when the notified data length is 0 (YES in S1302), the transmission unit 212 does not acquire the image data 310, so the image data is not output. Also, when the transmission unit 212 receives a packet from the DMA unit 203 (YES in S1304), it outputs the received packet via the DMA unit 204 (S1305). The transmission unit 212 repeats the operations of S1301 to S1305 until it receives an end instruction from the CPU 201. When it receives an end instruction from the CPU 201 (YES in S1306), it ends the operation.

[0048] As described above, in the present embodiment, the CPU 201 controls the control signal generation unit 205 so that the control signal B402 is not output to the transmission instruction unit 211 until the first image data 310 of the image data set specified by the user or the like is stored in the frame buffer D224. When image processing by pipeline processing is performed on the image data 310, the CPU 201 specifies the number of mask times based on the latency of the pipeline processing. Then, the control signal generation unit 205 does not output the control signal B402 for the number of mask times specified by the CPU 201. Thereby, it becomes possible to avoid the image data 310 stored in the frame buffer 220 before processing the image data set specified by the user or the like from being output to the communication line. By simply adding a process of masking the control signal B402 a predetermined number of times in this way, it becomes possible to control the image data to be output.

[0049] (Embodiment 2) In Embodiment 1, an example was described in which, in the signal generation unit 205, by masking the output of the control signal B402, control was performed so that unnecessary image data stored in the frame buffers A221 to D224 was not output. In this embodiment, an example will be described in which, for a predetermined period or a predetermined number of times, control is performed so that the image data 310 is not output to the communication line by setting the data length of the image data 310 notified by the image notification unit 210 to the transmission instruction unit 211 to 0. In this embodiment, the differences from Embodiment 1 will be described, and the operations of the other functional units are the same as those in Embodiment 1, and the description will be omitted. For example, the operation of the CPU 201 is the same as the operation flow in FIG. 7, and the content of the unnecessary image transmission prevention process in S706 is different. In Embodiment 1, the CPU 201 instructed the signal generation unit 205 to mask the output of the control signal B402. In this embodiment, instead of or in addition to this, the CPU 201 instructs the image notification unit 210 to set the data length of the image data 310 to 0.

[0050] FIG. 14 is an example of an operation flow when the control signal generation unit 503 of the signal generation unit 205 generates the control signal A401 and the control signal B402. The same reference numerals are given to the same components as in FIG. 9, and the description thereof is omitted. That is, in FIG. 9, the control signal generation unit 503 holds the number of mask times for masking the control signal B402 based on the instruction of the CPU 201, and determines whether to output the control signal B402 based on the held number of mask times. In FIG. 14, since the control signal generation unit 503 does not receive an instruction to mask the control signal B402, the components related to this process are deleted. That is, the control signal B402 is output after a predetermined period from the output of the control signal A401 based on the reference signal 506.

[0051] FIG. 15 shows an example of the operation flow of the image notification unit 210. For components similar to those in FIG. 11, the same reference numerals are assigned and the description is omitted. That is, based on an instruction from the CPU 201, the image notification unit 210 switches the ON and OFF states of a flag and controls the value to be notified as the data length of the image data 310. When starting the operation, the image notification unit 210 sets the flag to OFF (S1501). This flag is used to determine whether the image notification unit 210 should set the data length of the image data 310 to be notified to 0. When the image notification unit 210 receives an instruction from the CPU 201 to set the data length of the image data to be notified to the transmission instruction unit 211 to 0 (YES in S1502), it sets the flag to ON and sets the value of the counter to 0. This counter is used to count the number of times the data length of the image data has been notified as 0. When the image notification unit 210 receives the control signal A401 (S1101), it determines the presence or absence of the image data 310 in the frame buffer D224. When there is image data in the frame buffer D224 (YES in S1103) and the flag is OFF (NO in S1504), the image notification unit 210 notifies the transmission instruction unit 211 of the address and data length of the image data 310 (S1104). On the other hand, when there is image data 310 in the frame buffer D224 (YES in S1103) and the flag is ON (YES in S1504), the image notification unit 210 notifies the transmission instruction unit 211 that the data length of the image data 310 is 0 (S1506). Also, in this case, the image notification unit 210 increments the counter by one. When there is no image data 310 in the frame buffer D224 (NO in S1103) and the flag is ON (YES in S1505), the image notification unit 210 also notifies the transmission instruction unit 211 that the data length of the image data 310 is 0. Also in this case, the image notification unit 210 increments the counter by one (S1506). That is, when the flag is ON, regardless of the presence or absence of the image data 310 in the frame buffer D224, the image notification unit 210 issues a notification with the data length set to 0. As a result, no image data 310 is output from the transmission unit 212. Then, when the counter reaches a predetermined value (YES in S1507), the image notification unit 210 sets the flag to OFF and returns to S1101 to continue the operation (S1508).Also, when the counter has not reached the predetermined value (NO in S1507), the image notification unit 210 returns to S1101 and continues the operation. Similar to the number of mask times in the first embodiment, the predetermined value can be set based on the latency from when the first image data specified by the user is stored in the frame buffer A221 until it is stored in the frame buffer D224. For example, the predetermined value may be set in the initial setting after the reset of the image notification unit 210 is released, or may be a value fixedly set in the image notification unit 210. On the other hand, when there is no image data in the frame buffer D224 (NO in S1103) and the flag is OFF (NO in S1505), the image notification unit 210 notifies the transmission instruction unit 211 with the data length of the image data being 0.

[0052] As described above, in this embodiment, the CPU 201 controls the image notification unit 210 to notify the transmission instruction unit 211 that the data length of the image data is 0 until the first image data 310 of the specified image data set is stored in the frame buffer D224. Regardless of the presence or absence of data in the frame buffer D224, by being notified that the data length is 0 by the transmission instruction unit 211, the transmission instruction unit 211 does not give an output notification to the transmission unit 212. Or, the transmission instruction unit 211 also notifies the transmission unit 212 that the size of the image data 311 is 0. Thereby, it becomes possible to avoid the image data 310 stored in the frame buffer 220 before processing the image data set specified by the user or the like from being output to the communication line. Also, by configuring as in this embodiment, it becomes possible to control the image data to be output by adding only a process of correcting the output of the image notification unit 201 without modifying the operation of the conventional control signal generation unit 503.

[0053] (Embodiment 3) In Embodiment 1, an example was described in which the signal generation unit 205 controls so that unnecessary image data stored in the frame buffers A221 to D224 is not output by masking the output of the control signal B402. In this embodiment, a control signal C1601 different from the control signal A401 input to the image processing units A231 to C233 is input to the image notification unit 210. Then, by masking the output of this control signal C1601, control is performed so that the image data 310 is not output. Note that in this embodiment as well, the differences from Embodiment 1 will be described, and the operations of the other functional units are the same as those in Embodiment 1, so the description will be omitted. For example, the operation of the CPU 201 is the same as the operation flow in FIG. 7, and the content of the unnecessary image transmission prevention process in S706 is different. In Embodiment 1, the CPU 201 instructed the signal generation unit 205 to mask the output of the control signal B402, but in this embodiment, the CPU 201 instructs the image notification unit 210 to mask the output of the control signal C1601 to be notified. Note that the operation of the image notification unit 210 is the same as that in FIG. 11, but the difference from Embodiment 1 is that the control signal input to the image notification unit 210 changes from the control signal A401 to the control signal C1601.

[0054] FIG. 16 shows an example of the functional units constituting the pipeline process in this embodiment. The same components as those in FIG. 4 are given the same reference numerals and the description thereof is omitted. That is, the signal generation unit 205 generates three control signals, namely, the control signal A401, the control signal B402, and the control signal C1601, and outputs them to the image processing units A231 to C233, the transmission instruction unit 211, and the image notification unit 210, respectively. The control signal C1601 is a control signal having the same period and the same phase as the control signal A401, but has a different output destination. Also, the control signal C1601 can be a target of masking processing in the signal generation unit 205.

[0055] FIG. 17 shows an example of the operation flow of the control signal generation unit 503 of the signal generation unit 205. For components similar to those in FIG. 9, the same reference numerals are assigned and the description is omitted. That is, in FIG. 9, the control signal generation unit 503 holds the number of masking times for masking the control signal B402 based on the instruction from the CPU 201, and controls whether to output the control signal B402 based on the number of held masking times. In FIG. 17, the control signal generation unit 503 holds the number of masking times for masking the control signal C1601 based on the instruction from the CPU 201, and controls whether to output the control signal C1601 based on the number of held masking times. When the control signal generation unit 503 receives an instruction from the CPU 201 to mask the control signal C1601 (YES in S1701), it holds the number of masking times of the control signal C1601 (1702). This number of masking times is the number of times to mask the control signal C1601. The control signal generation unit 503 outputs the control signal A401 (S906) by detecting the rising or falling edge of the reference signal (YES in S905), and determines the number of masking times held. When the number of masking times is 0 (NO in S908), the control signal generation unit 503 outputs the control signal C1601 (S1703). On the other hand, when the number of masking times is 1 or more (YES in S908), the control signal generation unit 503 decrements the number of masking times by 1 without outputting the control signal C1601 (S1704). Then, when the control signal generation unit 503 outputs the control signal A401 and a predetermined time has elapsed (YES in S907), it outputs the control signal B402 (S1705).

[0056] As described above, in the present embodiment, the CPU 201 controls the control signal generation unit 205 so that the control signal C1601 is not output to the image notification unit 210 until the first image data 310 of the specified image data set is stored in the frame buffer D224. The image notification unit 210 does not notify the transmission instruction unit 211 while there is no input of the control signal C1601. Thereby, it becomes possible to avoid the image data 310 stored in the frame buffer 220 before processing the image data set specified by the user or the like from being output to the communication line.

[0057] (Embodiment 4) In Embodiments 1 to 3, an example has been described on the premise that the latency of the pipeline process is fixed by each functional unit constituting the pipeline process operating according to the period of the control signal A401 output by the control signal generation unit 503. That is, in these examples, the CPU 201 specifies the latency due to the pipeline process and instructs the signal generation unit 205 and the image notification unit 210 not to output a notification to the transmission unit 212 during a period corresponding to the latency. In the present embodiment, control is performed so that unnecessary image data stored in the frame buffers A221 to D224 is not output using identification information that can identify the image data set to which the image data 310 belongs. For example, the identification information can be the sequence number of the imaging data 300 or the Timecode 320. Therefore, the present embodiment is premised on the sequence number of the imaging data and the Timecode 320 associated with the image data 310 stored in the external storage unit 209 and each frame buffer 220 being maintained (not changed). Then, the CPU 201 notifies the image notification unit 210 of the sequence number of the imaging data and the Timecode 320 associated with the first image data 310 of the image data set specified by the user. The image notification unit 210 compares, for example, the values of the sequence number of the imaging data and the Timecode 320 associated with the image data 310 stored in the frame buffer D224 with the values notified from the CPU 201 based on the input of the control signal A401. The image notification unit 210 notifies the transmission instruction unit 211 that the data length of the image data 311 is zero until they match. Also, after they match, the image notification unit 210 notifies the transmission instruction unit 211 of the data length of the image data obtained from the header information of the image data 310 in the frame buffer D224. As a result, it becomes possible for the CPU 201 to control so that unnecessary image data stored in the frame buffers A221 to D224 is not output without specifying the latency of the pipeline process. For example, even when the latency of the pipeline process fluctuates, it is possible to apply this technology. The operations of each functional unit in the present embodiment will be described below.In addition, in this embodiment as well, the differences from Embodiment 1 will be described, and the operations of each other functional unit are the same as those in Embodiment 1, so the description thereof will be omitted. For example, the operation of the CPU 201 is the same as the operation flow in FIG. 7, and the content of the unnecessary image transmission prevention process in S706 is different. In this embodiment, the CPU 201 notifies the image notification unit 210 of the sequence number of the imaging data and the Timecode 320 associated with the first image data 310 of the image data set specified by the user.

[0058] FIG. 18 shows an example of the operation flow of the image notification unit 210. For components similar to those in FIG. 11, the same reference numerals are assigned and the description is omitted. That is, based on the sequence number of the imaging data and the reception of the Timecode 320 from the CPU 201, the image notification unit 210 switches the ON and OFF states of a flag and controls the value to be notified as the data length of the image data. When starting the operation, the image notification unit 210 sets the flag to OFF (S1801). This flag is used to determine whether the image notification unit 210 sets the data length of the image data to be notified to 0. When the image notification unit 210 receives the sequence number of the imaging data and the Timecode 320 from the CPU 201 (YES in S1802), it sets the flag to ON (S1803). When the image notification unit 210 receives the control signal A401 (S1101), it determines the presence or absence of the image data 310 in the frame buffer D224. When there is image data 310 in the frame buffer D224 (YES in S1103) and the flag is OFF (NO in S1804), the image notification unit 210 notifies the transmission instruction unit 211 of the address and data length of the image data 310 (S1104). On the other hand, when there is image data 310 in the frame buffer D224 (YES in S1103) and the flag is ON (YES in S1804), the image notification unit 210 determines whether the image data 310 matches the one notified from the CPU 201. That is, the image notification unit 210 determines whether the sequence number of the imaging data and the Timecode 320 associated with the image data 310 match the sequence number of the imaging data and the Timecode 320 notified from the CPU 201. If they match (YES in S1805), the image notification unit 210 turns off the flag (S1806) and notifies the transmission instruction unit 211 of the address and data length of the image data 310 (S1104). On the other hand, if they do not match (NO in S1805), the image notification unit 210 notifies the transmission instruction unit 211 that the data length of the image data is 0 (S1105). Also, when there is no image data in the frame buffer D224 (NO in S1103), the image notification unit 210 also notifies the transmission instruction unit 211 that the data length of the image data is 0.That is, when the flag is ON, the image notification unit 210 issues a notification with a data length of 0 regardless of the presence or absence of image data in the frame buffer D224. As a result, no image data is output from the transmission unit 212. For example, when executing the pipeline processing of the first image data set, a stop instruction from the user may be received, and a designation of the second image data set may be received at the next start instruction from the user. In this case, the sequence number and Timecode 320 of the imaging data corresponding to the second image data set are notified to the image notification unit 210. Then, the image notification unit 210 notifies the transmission instruction unit 211 with the data length of the image data set to 0 until the sequence number and Timecode 320 of the imaging data corresponding to the second image data set are detected from the image data in the frame buffer D224. In other words, while the image notification unit 210 is detecting the sequence number and Timecode 320 of the imaging data corresponding to the first image data set from the image data in the frame buffer D224, it notifies the transmission instruction unit 211 with the data length of the image data set to 0. In this way, it is possible to control so that the processed data of the first image data set is not output until the processed data of the second image data set is detected in the frame buffer D224 after the second image data set is input to the frame buffer A221.

[0059] As described above, in this embodiment, the CPU 201 controls such that the image notification unit 210 does not notify the transmission instruction unit 211 until the first image data 310 of the specified image data set is stored in the frame buffer D224. The image notification unit 210 detects that the image data 310 to be output to the frame buffer D224 has arrived based on the sequence number of the imaging data and the Timecode 320 notified from the CPU 201, and does not notify the transmission instruction unit 211 until then. Thereby, it is possible to avoid the image data 310 stored in the frame buffer 220 before processing the image data set specified by the user or the like from being output to the communication line. Further, according to the configuration of this embodiment, since it does not depend on the configuration of image processing in the camera adapter 111, this technology can be applied to pipeline processing in which latency varies and configurations other than pipeline processing.

[0060] As a modification of this embodiment, when the image notification unit 210 detects that the image data 310 to be output to the frame buffer D224 has arrived, it may notify the CPU 201 and the signal generation unit 205. For example, the CPU 201 notifies the image notification unit 210 of the sequence number of the imaging data and the Timecode 320, and causes the user or the like to determine whether or not the first image data 310 of the specified image data set is stored in the frame buffer D224. Further, the CPU 201 notifies the control signal generation unit 205 to mask the control signal B402. Then, when the image notification unit 210 detects that the first image data 310 of the image data set is stored in the frame buffer D224, it notifies the CPU 201. The CPU 201 notifies the signal generation unit 205 to release the mask based on this notification. Note that the image notification unit 210 may notify the signal generation unit 205 to release the mask. Thereby, it is possible to avoid the image data 310 stored in the frame buffer 220 before processing the image data set specified by the user or the like from being output to the communication line.

[0061] In the above description, an example has been described in which the number of times the CPU 201 should mask the control signal B402 and the control signal C1601, and the number of times the image notification unit 210 should notify with the data length of the image data 310 being 0 are indicated. However, the CPU 201 may indicate a predetermined period. For example, when image processing is executed by pipeline processing, based on the number of image processes included in the pipeline processing and the period of the pulse signal of the control signal A401, the time until the image data 310 reaches the frame buffer D224 can be calculated. By notifying each functional unit of the predetermined period thus calculated, the CPU 201 can control so that the image data 310 in the frame buffer D224 is not output by the transmission unit 212 over that period. Also, in the above embodiment, an example in which image processing is executed by pipeline processing has been described, but the present technology is not limited to application to pipeline processing. For example, the present technology can be applied to any image processing system that includes a plurality of image processes and requires a predetermined period from the input of the image data 310 to the output of the image processing result. Note that the target of the image processing may be the imaging data 300 or the image data 310. That is, the image data 310 and the imaging data 300 can be mutually interchanged. Furthermore, the functional diagrams described in each embodiment are examples, and for example, some functional units may not be included. For example, the transmission instruction unit 211 and the transmission unit 212 may be integrally configured. In this case, for example, the control signal B402 can be directly output to the transmission unit 212. Also, the image notification unit 210 can directly notify the transmission unit 212 of the address and data length of the image data 310. Furthermore, in the above embodiment, an example in which the image notification unit 210 notifies the transmission instruction unit 211 with the data length of the image data 310 being 0 has been described, but the image notification unit 210 may not notify the transmission instruction unit 211. Alternatively, the image notification unit 210 may notify that there is no image data 310 in the frame buffer D224 by a method other than setting the data length of the image data 310 to 0. For example, the image notification unit 210 may directly notify the transmission instruction unit 221 of the notification indicating that there is no data in the frame buffer D224.

[0062] (Summary of Embodiment) Summarizing at least a part of the above-described embodiments, it is as follows. (Item 1) A process of receiving an input of an image data set composed of a plurality of pieces of image data, generating processed data by performing image processing on the image data, and outputting the processed data to an output buffer, the process being performed by processing means that requires a predetermined time from the input to the output, transfer means for reading the processed data from the output buffer and outputting it to a communication line, control means for controlling so that the processed data stored in the output buffer is not output to the communication line during a predetermined period from when the image data of the image data set is input to the processing means until the first processed data of the image data set is output to the output buffer, A terminal device characterized by the above. (Item 2) The processing means is configured to include at least a plurality of image processing means for executing a series of image processing by pipeline processing and a plurality of buffers associated with each of the image processing means, The first image processing means included in the plurality of image processing means reads the image data from an input buffer included in the plurality of buffers, to which the image data is input, performs the first image processing included in the series of image processing on the image data to generate first processed data, and outputs the first processed data to a first buffer associated with the first image processing means, The second image processing means included in the plurality of image processing means reads the first processed data from the first buffer, performs a second image processing different from the first image processing included in the series of image processing on the first processed data to generate second processed data, and outputs the second processed data to a second buffer associated with the second image processing, The buffer associated with the image processing means that executes the last image processing in the series of image processing is the output buffer The terminal device according to item 1, characterized in that... (Item 3) When the terminal device receives a switching instruction to switch the target of the pipeline processing to a second image data set while performing the pipeline processing on the first image data set, the control means, at least after the image data of the second image data set is input to the processing means and until the first processed data of the second image data set is output to the output buffer, controls so that the processed data stored in the output buffer is not output to the communication line. The terminal device according to item 2, characterized in that... (Item 4) When each of the plurality of image processes included in the series of image processes in the pipeline process is executed based on a common timing signal, the control means specifies the predetermined period based on the number of image processes constituting the pipeline process. The terminal device according to item 2 or 3, characterized in that... (Item 5) For each of the plurality of image data belonging to the image data set, identification information that enables identification of each of the image data belonging to the image data set is associated, and the identification information is configured to be maintained also in the processed data after the processing by the processing means. The terminal device further has detection means for detecting that the first processed data of the image data set has been output to the output buffer based on the identification information associated with the processed data output to the output buffer. The control means controls so that the processed data output to the output buffer is not output to the communication line at least after the image data of the image data set is input to the processing means and until it is detected that the first processed data of the image data set has been output to the output buffer. The terminal device according to item 1, characterized in that... (Item 6) When the terminal device receives a switching instruction to switch the target of the image processing to a second image data set while performing the image processing on the first image data set, the control means controls such that, at least from when the image data of at least the second image data set is input to the processing means, until the identification information of the processed data detected by the detection means in the output buffer is the identification information corresponding to the first image data set, the processed data stored in the output buffer is not output to the communication line. The terminal device according to item 5, characterized in that. (Item 7) The control means controls such that, within the predetermined period, an output instruction to output the processed data stored in the output buffer to the communication line is not input to the transfer means, and after the predetermined period, the output instruction is input to the transfer means. The terminal device according to any one of items 1 to 6, characterized in that. (Item 8) When the transfer means receives a notification indicating that there is no processed data to be transferred to the output buffer, the transfer means operates so as not to output the processed data stored in the output buffer to the communication line. The control means controls to notify the transfer means that there is no image to be output within the predetermined period. The terminal device according to any one of items 1 to 6, characterized in that. (Item 9) A control method executed by a terminal device, comprising: A processing step of receiving an input of an image data set composed of a plurality of image data, generating processed data by performing image processing on the image data, and outputting the processed data to an output buffer, the processing taking a predetermined time from the input to the output; A transfer step of reading the processed data from the output buffer and outputting it to a communication line; In a predetermined period from when the image data of the image data set is input to the processing step until the first processed data of the image data set is output to the output buffer, control is performed so that the processed data stored in the output buffer is not output to the communication line. A control method characterized by the above. (Item 10) A program for causing a computer to function as each means included in the terminal device according to any one of Items 1 to 8.

[0063] The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiment to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program. Further, it can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

Explanation of Reference Numerals

[0064] 111: Camera adapter, 112: Camera, 120: Image processing device, 130: Time server, 140: Hub, 150: Control terminal, 160: User terminal, 201: CPU, 202: Internal storage unit, 203: DMA unit, 204: DMA unit, 205: Signal generation unit, 206: Camera control unit, 207: Communication IF unit, 208: Communication IF unit, 209: External storage unit, 210: Image notification unit, 211: Transmission instruction unit, 212: Transmission unit, 221: Frame buffer A, 222: Frame buffer B, 223: Frame buffer C, 224: Frame buffer D, 231: Image processing unit A, 232: Image processing unit B, 233: Image processing unit C

Claims

1. A process that receives an input of an image data set composed of a plurality of image data, generates processed data by performing image processing on the image data, and outputs the processed data to an output buffer, the process including: processing means for performing the process that requires a predetermined time from the input to the output; transfer means for reading the processed data from the output buffer and outputting it to a communication line; control means for controlling so that the processed data stored in the output buffer is not output to the communication line during a predetermined period from when the image data of the image data set is input to the processing means until the first processed data of the image data set is output to the output buffer; and a terminal device characterized by the above.

2. The processing means is configured to include at least a plurality of image processing means for executing a series of image processing by pipeline processing and a plurality of buffers associated with each of the image processing means, The first image processing means included in the plurality of image processing means reads the image data from an input buffer included in the plurality of buffers, where the image data is input, performs the first image processing included in the series of image processing on the image data to generate first processed data, and outputs the first processed data to a first buffer associated with the first image processing means, The second image processing means included in the plurality of image processing means reads the first processed data from the first buffer, performs a second image processing different from the first image processing included in the series of image processing on the first processed data to generate second processed data, and outputs the second processed data to a second buffer associated with the second image processing, The buffer associated with the image processing means that executes the last image processing in the series of image processing is the output buffer The terminal device according to claim 1, characterized by the above.

3. When the terminal device receives a switching instruction to switch the target of the pipeline process to a second image data set while performing the pipeline process on the first image data set, the control means controls such that the processed data stored in the output buffer is not output to the communication line at least from when the image data of at least the second image data set is input to the processing means until the first processed data of the second image data set is output to the output buffer. The terminal device according to claim 2, characterized in that.

4. When each of a plurality of image processes included in the series of image processes in the pipeline process is executed based on a common timing signal, the control means specifies the predetermined period based on the number of image processes constituting the pipeline process. The terminal device according to claim 2, characterized in that.

5. For each of the plurality of image data belonging to the image data set, identification information that enables identification of each of the image data belonging to the image data set is associated, and the identification information is configured to be maintained also in the processed data after the processing by the processing means. The terminal device further has a detection means for detecting that the first processed data of the image data set has been output to the output buffer based on the identification information associated with the processed data output to the output buffer. The control means controls such that the processed data output to the output buffer is not output to the communication line at least from when the image data of the image data set is input to the processing means until it is detected that the first processed data of the image data set has been output to the output buffer. The terminal device according to claim 1, characterized in that.

6. When the terminal device receives a switching instruction to switch the target of the image processing to a second image data set while performing the image processing on the first image data set, the control means controls such that, at least while the identification information of the processed data detected by the detection means in the output buffer is the identification information corresponding to the first image data set after the image data of at least the second image data set is input to the processing means, the processed data stored in the output buffer is not output to the communication line. The terminal device according to claim 5, characterized in that.

7. The control means controls such that, during the predetermined period, an output instruction to output the processed data stored in the output buffer to the communication line is not input to the transfer means, and the output instruction is input to the transfer means after the predetermined period. The terminal device according to claim 1, characterized in that.

8. When the transfer means receives a notification indicating that there is no processed data to be transferred to the output buffer, the transfer means operates so as not to output the processed data stored in the output buffer to the communication line. The control means controls to notify the transfer means that there is no image to be output during the predetermined period. The terminal device according to claim 1, characterized in that.

9. A control method executed by a terminal device, comprising: a process of receiving an input of an image data set composed of a plurality of image data, generating processed data by performing image processing on the image data, and outputting the processed data to an output buffer, the process taking a predetermined time from the input to the output; a transfer process of reading the processed data from the output buffer and outputting it to a communication line; a control process of controlling such that the processed data stored in the output buffer is not output to the communication line during a predetermined period from when the image data of the image data set is input to the processing process until the first processed data of the image data set is output to the output buffer. A control method characterized by the above.

10. A program for causing a computer to function as each means of the terminal device according to claim 1.

Images