COMMUNICATION DEVICE, CONTROL METHOD AND PROGRAM FOR COMMUNICATION DEVICE

Abstract

To be capable of appropriate time synchronization with another communication device on a communication device capable of processing each of a plurality of types of communication packets.SOLUTION: The communication device is configured to receive a communication packet, acquire a timestamp of when the communication packet was received, analyze a received communication packet type, transfer the received communication packet to a predetermined memory on the basis of the information indicating the analyzed communication packet type, and associate the analyzed communication packet type with the acquired timestamp.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present invention relates to a communication device capable of time synchronization with another communication device. [Background technology]

[0002] There is a technology that generates virtual viewpoint content using multiple viewpoint images obtained by installing multiple cameras at different positions and synchronously capturing images from multiple viewpoints. By using this virtual viewpoint content, for example, highlight scenes of soccer or basketball games can be viewed from various angles. This provides the user with a higher sense of realism than normal images.

[0003] One of the time synchronization technologies that uses a network to achieve this type of synchronized shooting is PTP (Precision Time Protocol: IEEE1588-2008). In PTP, a master device that has accurate time transmits the time information in a PTP packet. Slave devices perform time synchronization by receiving the PTP packet. In a communication network where time is synchronized by PTP, various communication packets are transferred. The types of communication packets transferred include PTP packets that perform time synchronization between communication devices or between a master device and a communication device, TCP / IP packets that control communication devices, and packets of images and videos captured by a camera.

[0004] Patent Document 1 discloses a communication device that distributes communication packets transferred on a communication network according to the type of the communication packets. Specifically, the communication device of Patent Document 1 analyzes the TOS (Type of Service) field of the IP header of a received communication packet, and distributes the communication packet according to the IP Precedence value. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] JP 2018-113623 A Summary of the Invention [Problem to be solved by the invention]

[0006] However, in a communication device that distributes a plurality of types of communication packets, there are cases where it is not possible to execute the PTP time synchronization process. Specifically, in a PTP slave device, when a received packet and a received timestamp are output separately and a different module is used for reception processing depending on the type of the received packet, the association between the received packet and the received timestamp is lost. As a result, it is not possible to execute the PTP time synchronization process using the appropriate received timestamp, and there is a risk of losing time synchronization with the master device.

[0007] The present invention has been made in consideration of the above-mentioned problems, and has an object to enable appropriate time synchronization with other communication devices in a communication device capable of processing multiple types of communication packets. [Means for solving the problem]

[0008] In order to solve the above problems, according to one aspect of the present invention, a communication device includes a receiving unit for receiving a communication packet, a transfer unit for transferring the communication packet received by the receiving unit to a predetermined memory, and a transfer unit for transferring the communication packet received by the receiving unit to a predetermined memory. Showing timing an acquisition means for acquiring a time stamp, an analysis means for analyzing a type of the communication packet received by the reception means, and a control means for notifying the transfer means of information indicating the type of the communication packet analyzed by the analysis means and controlling the transfer means to correspond to the time stamp acquired by the acquisition means; a plurality of storage means for storing the time stamps, respectively, and the control means allocates the time stamps to any of the plurality of storage means based on the type of the communication packet. A communications device is provided. Effect of the Invention

[0009] According to the present invention, in a communication device capable of processing a plurality of types of communication packets, appropriate time synchronization with other communication devices can be achieved. [Brief description of the drawings]

[0010] [Figure 1] FIG. 1 is a block diagram showing an example of a network configuration of a synchronized imaging system according to each embodiment. [Diagram 2] FIG. 1 is a block diagram showing an example of the hardware and functional configuration of a camera adapter 2 (communication device) according to each embodiment. [Diagram 3] A sequence diagram showing an example of PTP time synchronization processing [Figure 4] A flowchart showing an example of a processing procedure for receiving a communication packet executed by the camera adapter 2 according to each embodiment. [Diagram 5] A flowchart showing an example of a detailed processing procedure of the reception timestamp allocation process (S47) of FIG. 4 in the first embodiment. [Figure 6] 5 is a flowchart showing an example of a detailed procedure of the received packet sorting process (S48) of FIG. 4. [Figure 7] A flowchart showing an example of a processing procedure of PTP time synchronization processing according to each embodiment. [Figure 8] A flowchart showing an example of a detailed processing procedure of the reception timestamp distribution process (S47) of FIG. 4 in the second embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] Hereinafter, an embodiment for carrying out the present invention will be described in detail with reference to the attached drawings. Note that the embodiment described below is one example of a means for realizing the present invention, and should be appropriately modified or changed depending on the configuration of the device to which the present invention is applied and various conditions, and the present invention is not necessarily limited to the following embodiment. Also, not all of the combinations of features described in this embodiment are necessarily essential to the solution of the present invention. Note that the same components will be described with the same reference numerals.

[0012] (Embodiment 1) In this embodiment, the communication device assigns a reception timestamp according to the type of the received communication packet, and performs time synchronization by associating the received communication packet with the reception timestamp. An example will be described below in which the communication device performs time synchronization using PTP (Precision Time Protocol) conforming to the IEEE 1588 standard, but the time synchronization protocol of the present embodiment is not limited to this.

[0013] <Network Configuration of the Present Embodiment> FIG. 1 shows an example of a network configuration of a synchronized imaging system that generates virtual viewpoint images in a sports venue (stadium) or a concert hall, as an example of a system to which the time synchronization according to this embodiment can be applied.

[0014] 1, a synchronized imaging system 10 includes sensor systems 3a, 3b, ..., 3z, a time server 4, a hub 5, a control terminal 6, an image computing server 7, and a user terminal 8. The hub 5 and the sensor systems 3a, 3b, ..., 3z are connected by a daisy chain 9a, 9b, .... The daisy chain connects the hub 5 and each of the sensor systems 3a, 3b, ..., 3z in series. Each of the sensor systems 3a, 3b, ..., 3z includes a plurality of connectors, and can establish a daisy chain connection by connecting to separate devices via each connector.

[0015] Hereinafter, in this embodiment, unless otherwise specified, the 26 sets of systems from sensor system 3a to sensor system 3z will not be distinguished from one another and will be collectively referred to as sensor system 3. Similarly, unless otherwise specified, the devices in each sensor system 3 will not be distinguished from one another and will be collectively referred to as camera 1 and camera adapter 2. 1, 26 sets of sensor systems 3 are illustrated as an example, but the number of sensor systems is not limited to 26. Furthermore, the multiple sensor systems 3 do not need to have the same configuration, and may be composed of devices of different models, for example. In this embodiment, unless otherwise specified, the image will be described as including the concept of a moving image and a still image. That is, the synchronized imaging system 10 according to this embodiment is capable of processing both a still image and a moving image.

[0016] The time server 4 has a function of distributing time information, and distributes time information synchronized with a Global Navigation Satellite System (GNSS) 11 to the sensor system 3. Alternatively, the time server 4 may distribute the local time that the time server 4 has to the sensor system 3 as a master time. The hub 5 distributes PTP packets carrying time information distributed by the time server 4 and packets of image data transmitted by the sensor system 3 to the image computing server 7 to their respective destinations.

[0017] The control terminal 6 manages the operating state and controls parameter settings through a network for each component constituting the synchronized imaging system 10. Here, the network may be GbE (Gigabit Ethernet) or 10 GbE conforming to the IEEE standard, which is Ethernet (registered trademark), or may be configured by combining interconnect Infiniband, industrial Ethernet, etc. In addition, the network is not limited to these and may be any other type of network.

[0018] 1, the sensor system 3 includes a camera 1 and a camera adapter 2. However, the configuration of the sensor system is not limited to this. The sensor system 3 may further include, for example, an audio device such as a microphone, a storage device such as a hard disk drive (HDD) or a solid state drive (SSD), a camera platform for controlling the orientation of the camera, and the like.

[0019] Furthermore, the sensor system 3 may be composed of one camera adapter 2 and multiple cameras 1, or may be composed of one camera 1 and multiple camera adapters 2. The multiple sensor systems 3a, 3b, ..., 3z are connected to each other by a daisy chain 9b. The daisy chain connection makes it possible to reduce the number of connection cables and save wiring work when transferring image data that becomes larger in volume as captured images become higher resolution to 4K, 8K, etc. and higher frame rates. However, the connection form is not limited to a daisy chain. A star-type connection form may be adopted in which each sensor system 3 is connected to a hub 5, and data is transmitted and received between the multiple sensor systems 3 via the hub 5.

[0020] Also, in FIG. 1, a configuration in which all of the multiple sensor systems 3 are daisy-chained is illustrated, but this is not limiting. For example, the multiple sensor systems 3a, 3b, . . . , 3z may be divided into several groups, and the sensor systems 3 may be daisy-chained in units of the divided groups. Such a configuration is effective in a stadium. For example, a stadium may be configured with multiple floors, and a sensor system 3 may be installed on each floor. In this case, input to the image computing server 7 can be performed for each floor or for each half circumference of the stadium. In this way, even in a place where it is difficult to wire all of the sensor systems 3 in a single daisy chain, it is possible to simplify installation and make the system more flexible.

[0021] The cameras 1a, 1b, . . . , 1z capture images of a subject from multiple directions. In this embodiment, the multiple cameras 1 are described as being the same, but the performance and model of each camera 1 may be different. Each of camera adapters 2a, 2b, . . . , 2z is connected to camera 1 and controls camera 1, acquires captured images, provides synchronization signals, sets the time, and so on.

[0022] The camera adapter 2 performs image processing on the imaging data captured by the camera 1. Specifically, the camera adapter 2 outputs a foreground image and a background image from the imaging data. A virtual viewpoint image is generated based on the foreground image and the background image captured from multiple viewpoints by multiple camera adapters 2. Note that some camera adapters 2 output a foreground image separated from a captured image, but do not necessarily output a background image. The camera adapter 2 transmits the image-processed image data to the image computing server 7 via the hub 5 connected in a daisy chain.

[0023] Here, the camera adapter 2 operates as a PTP slave device and synchronizes time with a time server 4 which is a Grand Master Clock (GMC) device. The camera adapter 2 also operates as a Boundary Clock (BC) device which synchronizes time with other sensor systems 3 as a PTP master device.

[0024] Alternatively, the camera adapter 2 may operate as a TC (Transparent Clock) device that performs time correction on PTP packets from a GMC device and transfers them to another sensor system 3. The camera adapter 2 may also operate as an OC (Ordinary Clock) device that performs only time synchronization using PTP packets from a GMC device and PTP packets transferred from another sensor system 3.

[0025] Alternatively, the camera adapter 2 may operate as a TC device and an OC device. In this case, the camera adapter 2 performs time synchronization using PTP packets transferred from the GMC device and from other sensor systems 3, performs time correction on the PTP packets, and transfers the time-corrected PTP packets to the other sensor systems 3. Furthermore, the camera adapter 2 uses the synchronized time and reference signal to supply a control clock that defines the imaging timing to the camera 1. Details of the communication packet processing in the camera adapter 2 will be described later with reference to FIG.

[0026] The image computing server 7 processes the data acquired from the multiple sensor systems 3 . Specifically, the image computing server 7 first reconstructs the image data acquired from the sensor system 3, converts the data format, and stores the converted image data in a storage device according to the identifier of the camera 1, the data type, and the frame number. The image computing server 7 also accepts a designation of a viewpoint from the control terminal 6, reads out corresponding image data from the stored information based on the accepted viewpoint, and performs rendering processing on the read out image data to generate a virtual viewpoint image. That is, the image computing server 7 generates virtual viewpoint content based on captured images (multiple viewpoint images) captured by multiple sensor systems 3 and viewpoint information. At least a part of the functions of the image computing server 7 may be included in the control terminal 6, the sensor system 3, or the user terminal 8.

[0027] The user terminal 8 receives the rendered image from the image computing server 7. This allows the user operating the user terminal 8 to view the image from the viewpoint specified. Note that, although the above describes an example in which the virtual viewpoint content is generated by the image computing server 7, the present invention is not limited to this. In other words, the virtual viewpoint content may be generated by the control terminal 6 or the user terminal 8.

[0028] <Hardware and functional configuration of Camera Adapter 2> FIG. 2 is a block diagram showing an example of the hardware and functional configuration of the camera adapter 2 according to each embodiment, and mainly illustrates the network control portion of the camera adapter 2 which performs all or part of the functions of the communication device according to this embodiment.

[0029] Among the functional modules of the camera adapter 2 shown in FIG. 2, the functions realized by software are realized by storing a program for providing the function of each functional module in a memory such as a ROM, reading it into a RAM, and executing it by a CPU. For the functions realized by hardware, for example, a specific compiler may be used to automatically generate a dedicated circuit on an FPGA from a program for realizing the function of each functional module. FPGA is an abbreviation for Field Programmable Gate Array. Also, a gate array circuit may be formed in the same manner as an FPGA, and may be realized as hardware. Also, it may be realized by an ASIC (Application Specific Integrated Circuit). Note that the configuration of the functional blocks shown in FIG. 2 is an example, and a plurality of functional blocks may constitute one functional block, or any functional block may be divided into blocks performing a plurality of functions.

[0030] 2, the camera adapter 2 includes a CPU 21, a video packet processing unit 22, a first storage unit 23, a second storage unit 24, a DMA control unit 25, a PHY / MAC 26, and a packet analysis processing unit 27. The functional blocks (21 to 24, 27) in the camera adapter 2 are connected to each other via a system bus 28. The camera adapter 2 is also connected to the camera 1 via an image transmission cable 203.

[0031] A CPU (Central Processing Unit) 21 executes application software at the application layer defined by the OSI reference model, and also executes overall control of the camera adapter 2. The CPU 21 also stores data, programs, etc. required to execute the application software inside a first storage unit 23, and reads and writes them via a system bus 28 as necessary. Specifically, in this embodiment, the CPU 21 generates a descriptor for PTP stack processing and DMA transfer control using the DMA control unit 25.

[0032] Video packet processor 22 receives image data captured by camera 1 via image transmission cable 203 and packetizes the received image data. Hereinafter, the packetized image data is referred to as a video packet. Video packet processor 22 also generates a descriptor for DMA transfer control using DMA controller 25. In FIG. 2, video packet processor 22 is illustrated as dedicated hardware, but this is not limited to this, and video packet processing may be performed by another CPU different from CPU 21 but having similar functions.

[0033] The first storage unit 23 is a storage area used by the CPU 21. The first storage unit 23 may be mainly composed of a semiconductor memory such as a dynamic random access memory (DRAM) or a static random access memory (SRAM). The area inside the first storage unit 23 is divided into an area for storing programs, an application buffer area used by application software, etc., and these areas store descriptors for packet transfer, etc. Here, the descriptor is information used by the DMA control unit 25 described later for packet transfer, and describes information such as address information on the source side where data transfer is performed, address information on the destination side, and transfer size of data transfer.

[0034] The second storage unit 24 is a storage area used by the video packet processing unit 22. The second storage unit 24 is mainly composed of semiconductor memory such as DRAM or SRAM, similar to the first storage unit 23. The internal area of ​​the second storage unit 24 stores descriptors and the like for transmitting video packets. 2 shows an example in which the first storage unit 23 and the second storage unit 24 are respectively occupied by the CPU 21 and the video packet processing unit 22, but this is not limiting. For example, the first storage unit 23 and the second storage unit 24 may be configured as a single storage unit, and the CPU 21 and the video packet processing unit 22 may share and use this storage unit.

[0035] A DMA (Direct Memory Access) control unit 25 executes DMA transfer for directly transferring data between the PHY / MAC 26 and each storage unit based on an instruction from the CPU 21 or the video packet processing unit 22 . Specifically, the DMA control unit 25 executes a DMA transfer in accordance with the contents of the descriptors generated by the CPU 21 and the video packet processing unit 22, respectively. In this embodiment, the DMA control unit 25 selects a descriptor to be used for the DMA transfer based on an instruction from an analysis unit 272 in the packet analysis processing unit 27, which will be described later. After the DMA transfer is completed, the DMA control unit 25 notifies the CPU 21 or the video packet processing unit 22, which is the source of the instruction for the DMA transfer, of a completion interrupt 201 corresponding to the descriptor used for the DMA transfer.

[0036] The PHY / MAC 26 is an interface with a network 202 such as a LAN (Local Area Network), etc. The PHY / MAC 26 is responsible for communication control of a PHY layer (physical layer) and a MAC layer (data link layer) in communication. The PHY / MAC 26 also outputs a reception timestamp for every communication packet it receives. Here, the timestamp refers to the time information of when the PHY / MAC 26 transmits or receives a communication packet. This timestamp is used for calculations such as time correction for PTP protocol processing.

[0037] The packet analysis processing unit 27 includes a time stamp temporary storage unit 271 , an analysis unit 272 , an analysis result storage unit 273 , a time stamp allocation unit 274 , a time stamp storage unit 275 , and a register unit 276 . The time stamp temporary storage unit 271 is a storage unit that temporarily stores the reception time stamp output from the PHY / MAC 26, and may be a memory having a first input first output (FIFO) configuration.

[0038] The analysis unit 272 analyzes a MAC (Media Access Control) header of a communication packet received by the PHY / MAC 26. Specifically, the analysis unit 272 determines the EtherType of the received communication packet, and notifies the DMA control unit 25 and the analysis result storage unit 273 of the analysis result. The analysis result by the analysis unit 272 may be directly notified to the time stamp allocating unit 274 without being stored in the analysis result storage unit 273 . The analysis result storage unit 273 is a storage unit that stores the analysis results obtained by the analysis unit 272 analyzing and discriminating the received communication packet, and may be a FIFO memory.

[0039] The time stamp allocating unit 274 refers to the analysis results stored in the analysis result storage unit 273 and allocates the received time stamps stored in the time stamp temporary storage unit 271 to any one of a plurality of time stamp storage units 275 . Each of the multiple time stamp storage units 275 (#1 to #N) is configured with an IFO memory, and stores the reception time stamps assigned by the time stamp assignment unit 274. The reception time stamps are stored in a predetermined FIFO based on the packet type, which is the analysis result of the received communication packet.

[0040] The register unit 276 stores information such as the correspondence between the analysis result and the descriptor, the correspondence between the analysis result and deletion, and the correspondence between the analysis result and the timestamp storage unit 275 that stores the analysis result. These pieces of information stored in the register unit 276 are information for determining which descriptor to use for the communication packet received by the DMA control unit 25. These pieces of information stored in the register unit 276 are also information for determining to which timestamp storage unit 275 the timestamp distribution unit 274 distributes the received timestamp. These pieces of information are set in the register unit 276 by the CPU 21 before the CPU 21 starts the DMA transfer.

[0041] <PTP Time Synchronization Sequence> FIG. 3 is a sequence diagram showing an example of PTP time synchronization processing. In PTP time synchronization, synchronization packets for time synchronization are transmitted and received between the PTP's GMC (Grand Master Clock) master device and the PTP slave device. With reference to FIG. 3, an example in which the time server 4 in FIG. 1 operates as the GMC master device and the sensor system 3 in FIG. 1 operates as the PTP slave device will be described.

[0042] At S31, the time server 4, which is the GMC master device, transmits the synchronization packet Sync to the sensor system 3, which is the PTP slave device, at regular intervals. The sensor system 3 that has received the synchronization packet Sync holds the received timestamp information t2 of the synchronization packet Sync.

[0043] At S32, the time server 4, which is the GMC master device, places the transmission timestamp information t1 of the transmitted synchronization packet Sync on the synchronization packet Follow_Up and transmits it to the sensor system 3, which is the PTP slave device. When the sensor system 3 receives the synchronization packet Follow_Up, it acquires and holds the transmission timestamp information t1 carried by the Follow_up. 3 shows an example in which the transmission timestamp information t1 of the synchronization packet Sync is transmitted on Follow_Up, but is not limited to this. For example, the transmission timestamp information of the synchronization packet Sync may be transmitted on Sync itself, rather than on Follow_Up.

[0044] In S33, the sensor system 3, which is a PTP slave device, transmits a synchronization packet Delay_Request to the time server 4, which is a GMC master device. At this time, the sensor system 3 holds transmission timestamp information t3 when the Delay_Request is transmitted.

[0045] In S34, the time server 4, which is the GMC master device, transmits a synchronization packet Delay_Response to the sensor system 3, which is the PTP slave device. At this time, the time server 4 transmits the reception timestamp information t4, which indicates that the Delay_Request was received, to the sensor system 3 in the Delay_Response. By receiving the synchronization packet Delay_Response, the sensor system 3 acquires and holds reception timestamp information t4 indicating when the time server 4 received the Delay_Request.

[0046] The sensor system 3, which is a PTP slave device, calculates the average transmission delay time using the timestamps t1 to t4 obtained by transmitting and receiving the synchronization packets at S31 to S34, and corrects the time within the sensor system 3. Here, the average transmission delay time can be calculated using the following formula. {(t2-t1)+(t4-t3)} / 2 = (average transmission delay time)

[0047] In this embodiment, when the sensor system 3 receives the synchronization packet Sync, the packet analysis processing unit 27 stores the reception timestamp t2 in the PTP time synchronization processing sequence of Fig. 3 in a predetermined timestamp storage unit 275 corresponding to the packet type. The CPU 21 executes the PTP time synchronization processing using the reception timestamp t2 stored in the predetermined timestamp storage unit 275 corresponding to the predetermined packet type (PTP packet).

[0048] <Packet reception and analysis processing> FIG. 4 is a flowchart showing an example of a processing procedure for each process executed by the camera adapter 2 according to each embodiment to receive a communication packet and analyze the received communication packet. The process shown in FIG. 4 may be started, for example, when the sensor system 3 is started, but the timing of the start is not limited to this. For example, the process may be started when the communication function of the center system 3 is started, or the user may input a trigger via the GUI.

[0049] In S41, the CPU 21 of the camera adapter 2 sets setting information in the register unit 276. Specifically, the CPU 21 sets in the register unit 276 the correspondence between the analysis result of the received communication packet and the descriptor, the correspondence between the analysis result and packet discard, the correspondence between the analysis result and the time stamp storage unit 275 in which the analysis result is stored, and the like. 4 shows an example in which the CPU 21 sets the setting information in the register unit 276 after the sensor system 3 is started, but this is not limiting. For example, when the sensor system 3 is started, the setting information may be set in the register unit 276 as an initial setting.

[0050] In S42, the PHY / MAC 26 of the camera adapter 2 determines whether or not it has received a communication packet. If the PHY / MAC 26 has received a communication packet (S42: Yes), the process proceeds to S43 and S44. On the other hand, if the PHY / MAC 26 has not received a communication packet (S42: No), the process of S42 is repeated to wait for reception of a communication packet.

[0051] When the PHY / MAC 26 receives a communication packet in S42, in S43 the time stamp temporary storage unit 271 of the camera adapter 2 temporarily stores the reception time stamp supplied from the PHY / MAC 26. When the reception time stamp supplied from the PHY / MAC 26 is stored in the time stamp temporary storage unit 271, the process proceeds to S47.

[0052] When the PHY / MAC 26 receives a communication packet in S42, in parallel with the process of S43, the analysis unit 272 of the camera adapter 2 analyzes the MAC header of the received communication packet in S44. Specifically, the analysis unit 272 reads the type field of the MAC header (Ethernet header) of the received communication packet, and determines the EtherType indicating the packet type of the received communication packet. EtherType is a two-octet field in an Ethernet frame that indicates the higher-level protocol encapsulated in the payload of the frame.

[0053] For example, if the EtherType is 0x0800, it can be determined that the packet is an IPv4 packet, if it is 0x86DD, it can be determined that the packet is an IPv6 packet, and if it is 0x88F7, it can be determined that the packet is a PTP packet. When the analysis unit 272 analyzes the MAC header of the received communication packet and determines the EtherType, the process proceeds to S45 and S46.

[0054] In S45, the analysis unit 272 of the camera adapter 2 stores the packet type of the communication packet corresponding to the reception timestamp stored in the timestamp temporary storage unit 261 in S43 in accordance with the EtyerType determined in S44 in the analysis result storage unit 273. When the analysis unit 272 stores the packet type as the analysis result in the analysis result storage unit 273, the process proceeds to S47.

[0055] In parallel with the process of S45, in S46, the analysis unit 272 of the camera adapter 2 notifies the DMA control unit 25 of a descriptor to be used according to the packet type (EtherType) determined in S44. The correspondence between the EtherType determined in S44 and the descriptor to be used may be acquired by referring to the setting information set in advance in the register unit 276 in S41. Alternatively, the analysis unit 272 may notify the DMA control unit 24 of the determined packet type, and the DMA control unit 25 may acquire a descriptor corresponding to the notified packet type by referring to the register unit 276. That is, the analysis unit 272 notifies the DMA control unit 25 of information indicating the packet type, which is the analysis result. When the analysis unit 272 notifies the DMA control unit 25 of the descriptor or packet type to be used, the process proceeds to S48.

[0056] When the reception timestamp storage process in S43 and the packet type storage process in S45 are completed, in S47 the timestamp allocating unit 274 of the camera adapter 2 allocates the reception timestamp to one of the multiple timestamp storage units 275 (#1 to #N). Details of the reception timestamp allocating process in S47 will be described later with reference to FIG. When the time stamp allocating unit 274 completes the allocation process of the received time stamps, the process proceeds to S49.

[0057] When the analysis unit 272 notifies the DMA control unit 25 of the descriptor or packet type to be used for the received communication packet in S46, the DMA control unit 25 of the camera adapter 2 selects the notified descriptor in S48. The DMA control unit 25 transfers the received communication packet to a predetermined memory in accordance with the selected descriptor. The predetermined memory to which the received communication packet is transferred in S48 is either the first storage unit 23 or the second storage unit 24, with reference to FIG. 1. Details of the received packet transfer process in S48 will be described later with reference to FIG. 6.

[0058] When the reception timestamp allocation process in S47 and the reception packet transfer process in S48 are completed, in S49 the CPU 21 of the camera adapter 2 determines whether or not to continue the series of processes in S42 to S48. For example, if a factor that ends the communication state with the external device occurs, it is determined not to continue the process and the process in Fig. 4 is terminated, whereas if not, the process returns to S42 and repeats the processes in S42 to S48.

[0059] As described above, based on the analyzed packet type, the received timestamp is assigned to one of multiple timestamp memory units 275 in S47, and the received communication packet is transferred to either the first memory unit 23 or the second memory unit 24 in S48. Here, the received communication packet and the reception timestamp are associated with each other for each packet type. Therefore, each of the reception control modules that process the communication packet transferred to the first storage unit 23 or the second storage unit 24 can obtain the reception timestamp corresponding to the communication packet to be processed from the timestamp storage unit 275.

[0060] For example, the CPU 21 can read out a communication packet whose packet type is a PTP packet from the first storage unit 23, and can also read out a reception timestamp corresponding to the read PTP packet from the timestamp storage unit 275. This allows the CPU 21 to execute the PTP time synchronization process using an appropriate reception timestamp corresponding to the PTP packet. On the other hand, the video packet processing unit 22 reads out a communication packet whose packet type is a video packet from the second storage unit 24, and also reads out from the time stamp storage unit 275 a reception time stamp corresponding to the read video packet.

[0061] <Received timestamp sorting process> FIG. 5 is a flowchart showing an example of a detailed processing procedure of the reception time stamp allocation process (S47) of FIG. 4, the reception timestamp is stored in the timestamp temporary storage unit 271, and in S45, the packet type of the communication packet corresponding to the reception timestamp is stored in the analysis result storage unit 273, the process proceeds to S471 in FIG.

[0062] In S471, the time stamp allocating unit 274 of the camera adapter 2 determines whether or not the packet type of the communication packet corresponding to the reception time stamp stored in the analysis result storage unit 273 is a video packet. If the packet type is a video packet (S471: Yes), the process proceeds to S472, and if the packet type is not a video packet (S471: No), the process proceeds to S475.

[0063] If the packet type is a video packet, then in S472 the time stamp allocating section 274 determines whether or not to discard the reception time stamp of the video packet. Specifically, the time stamp allocating unit 274 can read out setting information that has been set in advance in the register unit 276 by the CPU 21, and determine whether the time stamp allocating unit 274 should discard the received time stamp or store it in the time stamp storage unit 275. If it is determined that the received timestamp is to be discarded (S472: Yes), the process proceeds to S473. On the other hand, if it is determined that the received timestamp is not to be discarded (S472: No), the process proceeds to S474.

[0064] If it is determined in S472 that the reception timestamp is to be discarded, then in S473 the timestamp allocating unit 274 discards the reception timestamp corresponding to the video packet, and the process proceeds to S49 in FIG. Specifically, to discard the reception timestamp, the timestamp allocating unit 274 may, for example, write the reception timestamp to be discarded in the timestamp storage unit 275 for discarding among the multiple timestamp storage units 275. Alternatively, the timestamp allocating unit 274 may write the reception timestamp to be discarded in an area prepared for discarding that is different from the timestamp storage unit 275, or may read and discard the temporarily stored reception timestamp. The method of discarding the reception timestamp may be the same in S477 and S480a below.

[0065] 4. On the other hand, if it is determined in S472 that the reception timestamp should not be discarded, in S474 the timestamp allocating unit 274 stores the reception timestamp corresponding to the video packet in the timestamp storage unit 265 for video packets, and proceeds to S49 in FIG. Specifically, when three timestamp storage units 265 are provided, the timestamp allocating unit 274 allocates the reception timestamp of a PTP packet to timestamp storage unit #1 (275). Also, the timestamp allocating unit 274 allocates the reception timestamp of a video packet to timestamp storage unit #2 (275). Similarly, the timestamp allocating unit 274 allocates the reception timestamp of a packet different from either a PTP packet or a video packet to timestamp storage unit #3 (275).

[0066] The association between the packet type and the storage destination of the received timestamp is not limited to this. The CPU 21 may set the association between the packet type and the timestamp storage unit #N (275) that stores the received timestamp of the packet type in advance as setting information in the register unit 276. The method of associating the packet type with the storage destination of the timestamp may be the same in S478 and S480b below.

[0067] Returning to S471, if the packet type is not a video packet, in S475, the time stamp allocating unit 274 determines whether or not the packet type of the communication packet corresponding to the reception time stamp stored in the analysis result storage unit 273 is a PTP packet. If the packet type is a PTP packet (S475: Yes), proceed to S476, whereas if the packet type is not a PTP packet (S475: No), proceed to S479.

[0068] If the packet type is a PTP packet, in S476, the time stamp allocating unit 274 determines whether or not to discard the reception time stamp of the PTP packet. Specifically, the time stamp allocating unit 274 can read out setting information that has been set in advance in the register unit 276 by the CPU 21, and determine whether the time stamp allocating unit 274 should discard the received time stamp or store it in the time stamp storage unit 275. If it is determined that the received timestamp is to be discarded (S476: Yes), the process proceeds to S477. On the other hand, if it is determined that the received timestamp is not to be discarded (S476: No), the process proceeds to S478.

[0069] If it is determined in S476 that the reception timestamp is to be discarded, in S477 the timestamp allocating unit 274 discards the reception timestamp corresponding to the PTP packet, and the process proceeds to S49 in FIG. On the other hand, if it is determined in S476 that the reception timestamp should not be discarded, in S478 the timestamp allocating unit 274 stores the reception timestamp corresponding to the video packet in the timestamp memory unit 275 for PTP packets, and proceeds to S49 in FIG.

[0070] Returning to S475, if the packet type is neither a video packet nor a PTP packet, in S479, the time stamp allocating unit 274 determines whether or not to discard the reception time stamp of a communication packet of another packet type. Specifically, the time stamp allocating unit 274 can read out setting information that has been set in advance in the register unit 276 by the CPU 21, and determine whether the time stamp allocating unit 274 should discard the received time stamp or store it in the time stamp storage unit 275. If it is determined that the received timestamp is to be discarded (S479: Yes), the process proceeds to S480a. On the other hand, if it is determined that the received timestamp is not to be discarded (S479: No), the process proceeds to S480b.

[0071] If it is determined in S479 that the reception timestamp is to be discarded, then in S480a the timestamp allocating section 274 discards the reception timestamp corresponding to the communication packet of the other packet type, and the process proceeds to S49 in FIG. On the other hand, if it is determined in S479 that the reception timestamp should not be discarded, in S480b, the timestamp allocation unit 274 stores the reception timestamp corresponding to the communication packet of the other packet type in a specified timestamp memory unit 275, and proceeds to S49 in Figure 4.

[0072] <Incoming packet sorting process> FIG. 6 is a flowchart showing an example of detailed processing steps of the received packet sorting process (S48) of FIG. 4. When the analysis unit 262 notifies the DMA control unit 25 of the descriptor or packet type to be used in accordance with the packet type (EtherType) determined by the analysis unit 272 in S46 of FIG. 4, the process proceeds to S481 of FIG.

[0073] In S481, the DMA control unit 25 of the camera adapter 2 determines whether or not the notification content notified from the analysis unit 272 indicates a video packet descriptor. If the notification content from the analysis unit 272 indicates a video packet descriptor (S481: Yes), the process proceeds to S482, and if the notification content from the analysis unit 272 does not indicate a video packet descriptor (S481: No), the process proceeds to S483.

[0074] If the notification content indicates a video packet descriptor in S481, the DMA control unit 25 selects the video packet descriptor in S482, transfers the video packet by DMA to a predetermined storage unit corresponding to the video packet, and proceeds to S49 in FIG. On the other hand, if the notification content does not indicate a video packet descriptor in S481, the DMA control unit determines in S483 whether or not the notification content notified from the analysis unit 272 indicates a PTP packet descriptor. If the notification content from the analysis unit 272 indicates a PTP packet descriptor (S483: Yes), the process proceeds to S484, and if the notification content from the analysis unit 272 does not indicate a PTP packet descriptor (S483: No), the process proceeds to S485.

[0075] If the notification content indicates a PTP packet descriptor in S483, the DMA control unit 25 selects the PTP packet descriptor in S484, DMA transfers the PTP packet to a predetermined storage unit corresponding to the PTP packet, and proceeds to S49 in FIG. On the other hand, if the notification content does not indicate a PTP packet descriptor in S483, the DMA control unit 25 selects a communication packet descriptor different from both the video packet and the PTP packet in S485. Then, the DMA control unit 25 DMA transfers the communication packet to a predetermined storage unit corresponding to the packet type of the communication packet, and proceeds to S49 in FIG.

[0076] <PTP time synchronization processing in sensor system 3> FIG. 7 is a flowchart showing an example of a processing procedure of PTP time synchronization processing that the sensor system 3 executes with a time server using the reception timestamp of a PTP packet (synchronization packet) assigned by the timestamp assignment unit 274. 7, an example will be described in which the time server 4 operates as a PTP master device (GMC master device) and the sensor system 3 operates as a PTP slave device in the synchronized imaging system 10. The transmission and reception process of the synchronization packet and the PTP time synchronization process in the sensor system 3 may be executed by, but is not limited to, the CPU 21 of the camera adapter 2. The transmission and reception process of the synchronization packet and the PTP time synchronization process may be executed by the CPU of the camera 1 or other device constituting the sensor system 3, and some or all of these processes may be offloaded to hardware.

[0077] In S71, the CPU 21 of the camera adapter 2 of the sensor system 3 determines whether or not the synchronization packets Sync and Follow_Up have been received from the time server 4. If the synchronization packets Sync and Follow_up have been received (S71: Yes), the process proceeds to S72 and S73, whereas if these synchronization packets have not been received (S71: No), the process returns to S71 and waits for reception of these synchronization packets.

[0078] When the synchronization packets Sync and Follow_up are received in S71, the CPU 21 acquires the time stamp information of the synchronization packets Sync and Follow_up in S72. Specifically, when the PHY / MAC 26 receives the synchronization packet Sync, the packet analysis processing unit 27 (time stamp allocating unit 274) stores the reception time stamp t2 of the synchronization packet Sync in a predetermined time stamp storage unit 275 for PTP packets. Then, when the DMA control unit 25 completes the DMA transfer of the synchronization packet Sync to a predetermined storage unit (first storage unit 23), the DMA control unit 25 notifies the CPU 21 of a completion interrupt 201. The CPU 21 that receives the completion interrupt 201 acquires the reception timestamp t2 stored in a predetermined timestamp storage unit 275 for PTP packets.

[0079] Meanwhile, the time server 4 transmits the transmission timestamp t1 at which the synchronization packet Sync was transmitted, loaded onto a synchronization packet Follow_up, to the sensor system 3. The PHY / MAC 26 of the camera adapter 2 of the sensor system 3 receives the synchronization packet Follow_up, and the DMA control unit 25 DMA-transfers the received synchronization packet Follow_up to a predetermined storage unit (first storage unit 23) and notifies the CPU 21 of a completion interrupt 201. The CPU 21, having received the completion interrupt 201, obtains the transmission timestamp t1 at which the time server 4 transmitted the synchronization packet Sync, from the synchronization packet Follow_up DMA-transferred to the predetermined storage unit.

[0080] Returning to S71, when the synchronization packets Sync and Follow_up are received, in parallel with the process of S72, the CPU 21 generates a synchronization packet Delay_request in S73. The CPU 21 transmits the generated synchronization packet Delay_request to the time server 4. Here, the CPU 21 acquires the transmission timestamp t3 at which the synchronization packet Delay_request is transmitted.

[0081] In S74, the CPU 21 determines whether or not the synchronization packet Delay_response has been received from the time server 4. If the synchronization packet Delay_response has been received (S74: Yes), the process proceeds to S76. On the other hand, if the synchronization packet Delay_response has not been received (S74: No), the process proceeds to S75. In S75, the CPU 21 returns to S74 and waits for reception of a synchronization packet Delay_response from the time server 4 until a certain time has elapsed (S75: No), and returns to S71 when the certain time has elapsed (S75: Yes).

[0082] Returning to S74, when the synchronization packet Delay_response is received from the time server 4, the CPU 21 obtains the time stamp information of the synchronization packet Delay_response. Specifically, the PHY / MAC 26 receives the synchronization packet Delay_response, and the DMA control unit 25 DMA-transfers the synchronization packet Delay_response to a predetermined storage unit (first storage unit 23). When the DMA transfer of the synchronization packet Delay_response is completed, the DMA control unit 25 notifies the CPU 21 of a completion interrupt 201. The CPU 21, which has received the completion interrupt 201, obtains a reception timestamp t4 at which the time server 4 received the synchronization packet Delay_request from the synchronization packet Delay_response DMA-transferred to the predetermined storage unit.

[0083] After the processing of S72 to S76 has been executed, in S77, the CPU 21 uses the time stamp information t1 to t4 acquired by the CPU 21 to calculate the time difference between the time of the time server 4 and the time of the sensor system 3, and determines the amount of correction by which to correct the time of the sensor system 3. In S78, the CPU 21 corrects the time of the sensor system 3 based on the correction amount determined in S77. When this time correction process is completed, the sensor system 3 is synchronized with the time server 4. Here, if the amount of correction made at one time is large, it may become unstable as a whole operation of the sensor system 3. For this reason, the time may be corrected stepwise over a certain period of time, or the amount of correction made at one time may be limited.

[0084] In S79, the CPU 21 determines whether or not the sensor system 3 continues to operate. For example, this is the case when the control terminal 6 stops the synchronous imaging process of all of the sensor systems 3a, 3b, ..., 3z, or when the control terminal 6 stops the synchronous imaging process of only a specified sensor system among the sensor systems 3a, 3b, ..., 3z. If the operation of the sensor system 3 is to be continued (S79: Yes), the process returns to S71 and repeats the processes of S71 to S78. On the other hand, if the operation of the sensor system 3 is not to be continued (S79: No), the process ends.

[0085] As described above, according to this embodiment, the communication device receives a communication packet, acquires a reception timestamp of the received communication packet, and analyzes the received communication packet to determine the packet type of the communication packet. Furthermore, the communication device transfers the communication packet to a predetermined storage unit based on the determined packet type, and stores the determined packet type in association with the acquired reception timestamp.

[0086] As a result, even if the communication device outputs the received packet and the received timestamp separately and executes the reception process in a different module depending on the type of the received packet, the received packet and the received timestamp can be associated with each other. Therefore, it is possible to execute time synchronization with other communication devices using an appropriate received timestamp. Therefore, this embodiment contributes to improving the accuracy of time synchronization processing between multiple communication devices.

[0087] (Embodiment 2) The second embodiment of the present invention will be described below focusing on the differences from the first embodiment. In the above embodiment 1, the timestamp allocation unit 274 allocated the received timestamp stored in the timestamp temporary storage unit 271 to one of multiple timestamp storage units 275 based on the analysis results of the packet type stored in the analysis result storage unit 273. In the second embodiment, an example in which an analysis result of a packet type is added to a reception timestamp and stored in one timestamp storage unit 275 will be described.

[0088] The hardware and functional configuration of the camera adapter 2 (communication device) according to the second embodiment is similar to that of the first embodiment shown in Fig. 2. However, in the second embodiment, since it is sufficient to use one timestamp storage unit 275, one or more timestamp storage units 275 may be provided, but an example in which multiple timestamp storage units 275 are provided will be described below.

[0089] FIG. 8 is a flowchart illustrating an example of a detailed processing procedure of the reception timestamp allocation process (S47) in FIG. 4 according to the second embodiment. In S43 of FIG. 4, the reception timestamp of the received communication packet is stored in the timestamp temporary storage unit 271, and in S45, the packet type corresponding to the reception timestamp is stored in the analysis result storage unit 273, and then the processing shown in FIG. 8 is started.

[0090] 8, the processes in S471 to S473 are similar to those in S471 to S473 in the first embodiment shown in FIG. In S472, when the time stamp allocating section 274 determines not to discard the reception time stamp of the video packet (S472: No), the process proceeds to S474b. In S474b, the time stamp allocation processing unit 274 adds the analysis result of the packet type of the video packet to the reception time stamp, and stores it in one of the time stamp storage units 275.

[0091] For example, the time stamp allocating unit 274 writes the packet type analysis result in the upper bits and the reception timestamp in the lower bits, and puts them together in one entry of time stamp information. When reading the reception timestamp from the time stamp storage unit 275, the CPU 21 can simultaneously read the packet type analysis result and the reception timestamp, and can determine which packet type the read reception timestamp corresponds to.

[0092] In the above, an example in which one timestamp storage unit 275 is used has been described, but the present invention is not limited to this. For example, the received timestamp may be further divided into higher bits and lower bits and stored in different timestamp storage units 275. For example, the analysis result + received timestamp (higher bits) may be stored in timestamp storage unit #1 (275), and the analysis result + received timestamp (lower bits) may be stored in timestamp storage unit #2 (275).

[0093] Thereafter, the processes of S475 to S477 and S479 to S480a are similar to the processes of S475 to S477 and S479 to S480a in the first embodiment shown in FIG. 8, in S478b, the time stamp allocation processing unit 274 adds the analysis result of the packet type of the PTP packet to the reception time stamp and stores it in one of the time stamp storage units 275, similarly to S474b. Also, similarly to S474b, in S480c, the time stamp allocation processor 274 adds the analysis result of a predetermined packet type other than the video packet and the PTP packet to the reception time stamp and stores it in one of the time stamp storage units 275.

[0094] As described above, according to this embodiment, the time stamp allocating unit 274 of the camera adapter 2 adds a packet type to a received time stamp and stores it in one time stamp storage unit 275. Therefore, the sensor system 3 according to this embodiment can execute the PTP time synchronization process using an appropriate received time stamp without allocating the received time stamp to one of the multiple time stamp storage units 275.

[0095] <Other embodiments> It should be noted that the above-described embodiments can be realized by combining a plurality of them. The present invention can also be realized by a program that realizes part or one or more functions of the above-mentioned embodiments. That is, the program can be supplied to a system or device via a network or a storage medium, and one or more processors in a computer (or a CPU, MPU, etc.) of the system or device read and execute the program. The program may also be provided by recording it on a computer-readable recording medium. In addition, the functions of the embodiments are not limited to those that are realized by a computer reading and executing a program. For example, an operating system (OS) running on a computer may perform a part or all of the actual processing based on instructions from the program, and the functions of the above-mentioned embodiments may be realized by the processing. [Explanation of symbols]

[0096] 1...camera, 2...camera adapter, 3...sensor system, 4...time server, 10...synchronous imaging system, 11...GNSS, 21...CPU, 22...video packet processing unit, 23...first memory unit, 24...second memory unit, 25...DMA control unit, 26...PHY / MAC, 27...packet analysis processing unit, 28...system bus, 271...timestamp temporary memory unit, 272...analysis unit, 273...analysis result memory unit, 274...timestamp allocation unit, 275timestamp memory unit, 276...register unit

Claims

1. A receiving means for receiving a communication packet; a transfer means for transferring the communication packet received by the receiving means to a predetermined memory; an acquisition means for acquiring a time stamp indicating the timing at which the communication packet was received by the receiving means; an analysis means for analyzing a type of the communication packet received by the receiving means; a control means for notifying the transfer means of information indicating the type of the communication packet analyzed by the analysis means and for controlling the transfer means to associate the information with the timestamp acquired by the acquisition means; a plurality of storage means for storing the time stamps, The control means allocates the time stamp to one of the plurality of storage means based on the type of the communication packet. A communication device comprising:

2. The transfer means selects a descriptor to be used for DMA (Direct Memory Access) transfer of the communication packet based on information indicating the type of the communication packet notified from the control means.

2. The communication device according to claim 1 .

3. The transfer means selects a memory corresponding to a module that is to process the communication packet based on information indicating the type of the communication packet notified from the control means, and transfers the communication packet to the selected memory.

3. The communication device according to claim 1 or 2.

4. The control means determines whether to discard the time stamp or to store the time stamp in a storage means in association with the type of the communication packet, based on the type of the communication packet.

4. The communication device according to claim 1, wherein the first and second inputs are connected to the first and second inputs.

5. a first temporary storage means for temporarily storing the type of the communication packet analyzed by the analysis means; A second temporary storage means for temporarily storing the time stamp acquired by the acquisition means, The control means associates the type of the communication packet read from the first temporary storage means with the time stamp read from the second temporary storage means.

5. The communication device according to claim 1, wherein the first and second inputs are connected to the first and second inputs.

6. The control means assigns the type of the communication packet to the time stamp, and stores the time stamp to which the type of the communication packet has been assigned in a storage means.

2. The communication device according to claim 1 .

7. The apparatus further includes a synchronization unit that executes a time synchronization process by reading out the synchronization packet transferred to the predetermined memory by the transfer unit and acquiring the time stamp associated with the read synchronization packet.

7. The communication device according to claim 1, wherein the first and second inputs are connected to the first and second inputs.

8. The synchronization means executes the time synchronization process using PTP (Precision Time Protocol) conforming to the IEEE 1588 standard.

8. The communication device according to claim 7,

9. The analyzing means analyzes the type of the communication packet by referring to an EtherType described in a MAC (Media Access Control) header of the communication packet received by the receiving means.

9. A communication device according to claim 1, wherein the communication device is a communication device having a plurality of communication ports.

10. A method for controlling a communication device that receives a communication packet, comprising the steps of: receiving the communication packet; obtaining a timestamp indicating when the communication packet was received; analyzing a type of the received communication packet; transferring the received communication packet to a predetermined memory based on information indicating the analyzed type of the communication packet; Correlating the analyzed type of the communication packet with the acquired timestamp; allocating the time stamp to one of a plurality of storage means based on the type of the communication packet; A control method comprising:

11. A program for causing a computer to function as each of the means of the communication device according to any one of claims 1 to 9.

Images