Data acquisition systems, methods, and programs

Abstract

To achieve data order guarantee and high scalability.SOLUTION: Data receiving units 103 and 113 perform mutually synchronized time management, add a timestamp representing a time based on the time management to data received from the data source, and transmit data and association information that associates data identification information and the timestamp to data transfer units 104 and 114, the data transfer units 104 and 114 receive data and association information from the data receiving units 103 and 113, temporarily save the association information and transfer the data and the association information to a data storage unit 105, and cancel the temporary storage of the association information after confirming that the data has been saved in the data storage unit 105.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] This disclosure relates to a technique for collecting data.

Background Art

[0002] Data hubs have drawn attention in order to achieve data linkage among multiple information systems. Data hubs are characterized by little data loss and high scalability. In recent years, the areas to which data hubs are applied have expanded, and accordingly, the data handled by data hubs has become diversified. For example, data hubs are required not only to collect numerical information output from devices but also to be used for file transfer. In file transfer, ensuring the order between data is often required. And a large amount of data is output to the data hub from an information system performing batch processing at the same time.

[0003] Patent Document 1 discloses a technique for ensuring the order of data. The data processing system disclosed in Patent Document 1 is composed of a router connected to a network, a load balancing server connected to the router, a plurality of reception servers connected to the load balancing server, and a processing server connected to the plurality of reception servers. The router assigns a timestamp to the input packets. The reception server assigns the latest time of the timestamps assigned to the plurality of packets as the reception time of the transaction for the transaction divided into a plurality of packets having the same identifier and transmitted. The processing server sorts and processes the plurality of transactions received from the plurality of reception servers according to the reception time of the transaction assigned to the transaction that has elapsed the preset maximum delay time.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

[0005] As described above, in the technology of Patent Document 1, the router adds a timestamp to the packet. The router becomes a path that packets must always take, and by adding a timestamp there, the order of packets is guaranteed.

[0006] However, because timestamps are added to packets at a single router to guarantee order, even with a configuration that distributes the packet processing load across multiple receiving servers, there are limitations to the overall system scalability.

[0007] One of the purposes of this disclosure is to provide a technology that ensures data order and high scalability. [Means for solving the problem]

[0008] A data collection system according to one aspect of the present disclosure comprises: a plurality of data receiving units that share the processing of receiving data from a data source; a plurality of data transfer units provided corresponding to each of the data receiving units and transferring the data received by the data receiving units; and a data storage unit that receives and stores the data transferred from the plurality of data transfer units. The plurality of data receiving units perform synchronized time management with each other, assign a timestamp representing the time based on the time management to the received data, transmit the data and correspondence information that associates the data's identification information with the timestamp to the data transfer unit, the data transfer unit receives the data and the correspondence information from the data receiving unit, temporarily stores the correspondence information, transfers the data and the correspondence information to the data storage unit, and releases the temporary storage of the correspondence information after confirming that the data has been stored in the data storage unit. [Effects of the Invention]

[0009] According to one aspect of this disclosure, it is possible to achieve data order guarantee and high scalability. [Brief explanation of the drawing]

[0010] [Figure 1] This is a block diagram of the information gathering system of this embodiment. [Figure 2] Figure 1 is a sequence diagram illustrating the process when no failures occur in the data receiving unit and data transfer unit of the information collection system shown in Figure 1. [Figure 3] Figure 2 is a flowchart illustrating the details of the processing in the data receiving unit. [Figure 4] Figure 2 is a flowchart illustrating the details of the processing in the data transfer section. [Figure 5] Figure 1 shows an example of the configuration of the work-in-progress status management database. [Figure 6] Figure 1 shows an example of the configuration of the data storage database. [Figure 7] Figure 1 is a sequence diagram illustrating the process that occurs when a failure occurs in the data receiving unit during data collection in the information collection system shown in Figure 1. [Figure 8] Figure 1 is a sequence diagram illustrating the process that occurs when a failure occurs in the data receiving unit after data has been transferred from the data receiving unit to the data transfer unit in the information collection system shown in Figure 1. [Figure 9] Figure 1 is a sequence diagram illustrating the process that occurs when a failure occurs in the data transfer unit before data is transferred from the data transfer unit to the data storage unit in the information collection system shown in Figure 1. [Figure 10] Figure 1 is a sequence diagram illustrating the process that occurs when a failure occurs in the data transfer unit while transferring data from the data transfer unit to the data storage unit in the information collection system shown in Figure 1. [Figure 11] Figures 9 and 10 are flowcharts illustrating the details of the processing in the data transfer section. [Figure 12] This is a sequence diagram for explaining another example of the processing when no failure occurs in the data reception unit and the data transfer unit in the information collection system shown in FIG. 1. [Figure 13] This is a flowchart for explaining the details of the processing in the data transfer unit shown in FIG. 12. [Figure 14] This is a diagram showing an example of the configuration of the in-process state management database shown in FIG. 1. [Figure 15] This is a sequence diagram for explaining another example of the processing when a failure occurs in the data transfer unit before transferring data from the data transfer unit to the data storage unit in the information collection system shown in FIG. 1. [Figure 16] This is a flowchart for explaining the details of the processing in the data transfer unit shown in FIG. 15.

Mode for Carrying Out the Invention

[0011] Hereinafter, embodiments of the present invention will be described with reference to the drawings.

[0012] FIG. 1 is a block diagram showing the information collection system of the present embodiment.

[0013] As shown in FIG. 1, the data collection system 100 of the present embodiment is a system that transfers data transmitted from data sources 101 and 111 for data utilization 106, and includes a load distribution unit 102, data reception units 103 and 113, data transfer units 104 and 114, and a data storage unit 105. Note that the number of data reception units 103 and 113 and data transfer units 104 and 114 is not limited to two each.

[0014] The load distribution unit 102 distributes and transmits the data transmitted from the data sources 101 and 111 to the data reception units 103 and 113 in a determined load distribution method such as round robin. Note that the method of distributing data is not limited to the load distribution method, and the distributed data may also be referred to as a chunk.

[0015] The data receiving units 103 and 113 are responsible for processing the reception of data transmitted from data sources 101 and 111 via the load balancing unit 102. Each unit performs synchronized time management with the others. The time synchronization generation units 131 and 132 generate and attach a timestamp representing the time based on the time management to the received data, and then transmit the received data, along with correspondence information that associates the ID (data identification information) and the timestamp, to the data transfer units 104 and 114.

[0016] The data transfer units 104 and 114 are provided in correspondence with the data receiving units 103 and 113, respectively, and transfer the data received by the data receiving units 103 and 113. They receive data and correspondence information from the data receiving units 103 and 113, temporarily store the correspondence information, transfer the data and correspondence information to the data storage unit 105, and release the temporary storage of the correspondence information after confirming that the data has been stored in the data storage unit 105. In addition, the data transfer units 104 and 114 use the data and correspondence information received from the data receiving units 103 and 113 to generate new records in the work-in-progress status management databases 141 and 142, store the data and correspondence information received from the data receiving units 103 and 113 in the work-in-progress status management databases 141 and 142, and set the work-in-progress status that corresponds to the transfer state. Furthermore, if the data transfer units 104 and 114 encounter a failure after receiving data and mapping information from the data receiving units 103 and 113, and then resume operations after recovering from the failure, they send a response prompting the data to the data sources 101 and 111 via the data receiving units 103 and 113, while also retaining the mapping information and using the timestamp within the retained mapping information for the retransmitted data.

[0017] The data storage unit 105 receives and stores data transferred from the data transfer units 104 and 114. The time indicated by the timestamp in the correspondence information transferred from the data transfer units 104 and 114 is used as the data creation time, and the data and its creation time are associated and stored in the data storage destination database 151.

[0018] The following describes the processing in the information gathering system configured as described above.

[0019] First, we will explain the process when no failures occur in the data receiving units 103, 113 and the data transfer units 104, 114.

[0020] Figure 2 is a sequence diagram illustrating the process when no failures occur in the data receiving units 103, 113 and data transfer units 104, 114 in the information collection system shown in Figure 1. In Figure 2, the example of transferring data to data source 101, data receiving unit 103, and data transfer unit 104 is used for the explanation.

[0021] In step 201, when data is transmitted from the data source 101, the load balancing unit 102 determines the data receiving unit to which the data will be transferred using a predetermined load balancing method in step 202, and then transfers the data to the determined data receiving unit 103 in step 222.

[0022] The data receiving unit 103, having received the data transferred from the load balancing unit 102, performs the processing described in step 203.

[0023] Figure 3 is a flowchart illustrating the details of the processing in the data receiving unit 103 shown in Figure 2.

[0024] In step 203, when the data receiving unit 103 receives data transferred from the load balancing unit 102, in step 231, it first obtains a timestamp generated by the time synchronization generation unit 131 using high-precision time synchronization (for example, GPS (Global Positioning System) or PTP (Precision Time Protocol)) in order to perform synchronized time management. In step 232, it generates a data ID to identify the data. The data ID consists of, for example, "file name" + "data source ID" (IP address or client ID, etc.). Files with the same name from the same data source are assigned the same ID, while files with the same name are assigned different IDs if they come from different data sources. High-precision time synchronization is possible by performing time management using GPS or PTP.

[0025] Next, in step 233, the data receiving unit 103 combines the timestamp obtained from the time synchronization generation unit 131, the data name, and the data ID to create a triplicate that will serve as the correspondence information.

[0026] Next, in step 234, the data receiving unit 103 transfers the data transferred from the load balancing unit 102 and the triplets created in step 233 to the data transfer unit 104, and in step 705, the processing in step 203 is completed.

[0027] The data transfer unit 104, having received the data and triplets transferred from the data receiving unit 103, performs the processing described in step 204.

[0028] Figure 4 is a flowchart illustrating the details of the processing in the data transfer unit 104 shown in Figure 2.

[0029] In the processing in step 204, when the data transfer unit 104 receives the data and triplets transferred from the data receiving unit 103, it first determines in step 801 whether the received triplets are stored in the work-in-progress status management database 141. This is done using the data ID within the triplets.

[0030] If the received triplets are not stored in the work-in-progress status management database 141, the data transfer unit 104 generates a new record in the work-in-progress status management database 141 in step 241, saves the data and triplets received from the data receiving unit 103 in the work-in-progress status management database 141, and sets the work-in-progress status of the transfer to "received".

[0031] Furthermore, if the received triplets are stored in the work-in-progress status management database 141, the data transfer unit 104 changes the work-in-progress status of the stored triplets to "transferring" in step 242.

[0032] Figure 5 shows an example of the configuration of the work-in-progress status management databases 141 and 142 shown in Figure 1.

[0033] As shown in Figure 5, the transfer status 1004 is associated with a triplicate consisting of a timestamp 1001, a data name 102, and a data ID 1003 in the work-in-progress status management databases 141 and 142 shown in Figure 1.

[0034] Next, in step 243, the data transfer unit 104 transfers the data received from the data receiving unit 103 and the triplets to the data storage unit 105, and in step 802, waits for a response from the data storage unit 105.

[0035] When the data storage unit 105 receives the data and its triplicate transferred from the data transfer unit 104, in the processing in step 205, it saves the received data to the data storage destination database 151 in step 251, and in step 252, it returns an ACK to the data transfer unit 104 indicating that the data received from the data transfer unit 104 has been saved to the data storage destination database 151 in step 251.

[0036] Furthermore, in step 253, the data storage unit 105 uses the time indicated by the timestamp within the triplet as the data creation time, and stores the data and its creation time in the data storage destination database 151 in association with each other.

[0037] Figure 6 shows an example of the configuration of the data storage database 151 shown in Figure 1.

[0038] As shown in Figure 6, the data storage destination database 151 shown in Figure 1 stores the data name 1101, data ID 1102, and timestamp 1103 within the triplicate received by the data storage unit 105 from the data transfer unit 104, in association with the data content 1104 that indicates the content of that data.

[0039] In this way, the timestamp at the data receiving unit 103 is used as the data creation time, so the creation time can be set using highly accurate timekeeping managed by multiple data receiving units.

[0040] When the data transfer unit 104 receives an ACK from the data storage unit 105, in step 244, it confirms that the data for the triplet corresponding to the ACK from the data storage unit 105 among the triplets stored in the work-in-progress status management database 141 has been stored in the data storage unit 105, and changes the work-in-progress status of that transfer to "transfer complete".

[0041] Subsequently, in step 245, the data transfer unit 104 returns an ACK to the data receiving unit 103 for the triplets corresponding to the ACK from the data storage unit 105. In step 246, it releases and deletes the triplets that were temporarily stored in the work-in-progress status management database 141. In step 807, it terminates the processing in step 204.

[0042] When the data receiving unit 103 receives an ACK from the data transfer unit 104, in step 206, it returns an ACK to the load balancing unit 102 as step 261. When the load balancing unit 102 receives an ACK from the data receiving unit 103, it returns an ACK to the data source 101 as step 223 in step 202.

[0043] Subsequently, in step 212, the data source 101 deletes the data it sent to the load balancing unit 102.

[0044] In this way, data order is guaranteed based on time management at multiple load-distributed data receiving units, thus achieving both data order guarantee and high scalability.

[0045] Next, we will explain how to handle the case of a failure in the data receiving units 103 and 113 during the data collection process.

[0046] Figure 7 is a sequence diagram illustrating the process that occurs when a failure occurs in the data receiving unit 103 during data collection processing in the information collection system shown in Figure 1.

[0047] If, after processing in step 201 in the data source 101 as shown in Figure 2, and processing in step 233 in the data receiving unit 103, a failure occurs in step 301 in the data receiving unit 103, and the load balancing unit 102 detects the failure in step 321, for example using a timeout or heartbeat, the load balancing unit 102 returns a negative ACK to the data source 101 in step 322, indicating that the data collection process failed.

[0048] Subsequently, even if the data receiving unit 103 recovers from the failure in step 302, it does not retain the data transmitted from the load balancing unit 102, so the same processing as shown in Figure 2 will be performed.

[0049] Next, we will explain how to handle the case where a failure occurs in the data receiving unit 103 after data has been transferred from the data receiving unit 103 to the data transfer unit 104.

[0050] Figure 8 is a sequence diagram illustrating the process that occurs when a failure occurs in the data receiving unit 103 after data has been transferred from the data receiving unit 103 to the data transfer unit 104 in the information collection system shown in Figure 1.

[0051] Even if a failure occurs in step 401 in the data receiving unit 103 after the processing in step 201 in the data source 101 and the processing in step 234 in the data receiving unit 103 shown in Figure 2, the data and its triplets have been transferred from the data receiving unit 103 to the data transfer unit 104. Therefore, the data transfer unit 104 and the data storage unit 105 will perform the same processing as shown in Figure 2.

[0052] Furthermore, when the data transfer unit 104 returns an ACK in step 245, if the data receiving unit 103 has recovered from the failure in step 402, the data receiving unit 103, load balancing unit 102, and data source 101 will then perform the same processing as shown in Figure 2.

[0053] Next, we will explain how to handle the case where a failure occurs in the data transfer unit 104 before transferring data from the data transfer unit 104 to the data storage unit 105.

[0054] Figure 9 is a sequence diagram illustrating the process that occurs when a failure occurs in the data transfer unit 104 before transferring data from the data transfer unit 104 to the data storage unit 105 in the information collection system shown in Figure 1.

[0055] If, after the processing in step 201 in the data source 101 shown in Figure 2 and the processing in step 241 in the data transfer unit 104, an error occurs in step 501 in the data transfer unit 104, and then the data transfer unit 104 recovers from the error in step 502, the data transfer unit 104 detects a triplet with the status "received" from the work-in-progress status management database 141 as step 541 in step 504.

[0056] Then, in step 542, the data transfer unit 104 returns a negative ACK to the data receiving unit 103 indicating that the data collection process failed. At this time, the data transfer unit 104 does not delete the triplets detected from the work-in-progress status management database 141.

[0057] When the data receiving unit 103 receives a negative ACK from the data transfer unit 104, in step 506 it returns the negative ACK to the load balancing unit 102 as step 561.

[0058] When the load balancing unit 102 receives a negative ACK from the data receiving unit 103, in step 202 it returns the negative ACK to the data source 101 as step 521.

[0059] Subsequently, the series of processes shown in Figure 2 will be performed again, but since the data transfer unit 104 has not deleted the triplets detected from the work-in-progress status management database 141, the stored timestamp will be used for the retransmitted data.

[0060] In this way, the previous timestamp is used when resending data, preventing the data from being rearranged due to retransmission.

[0061] Next, we will explain how to handle the situation when a failure occurs in the data transfer unit 104 while data is being transferred from the data transfer unit 104 to the data storage unit 105.

[0062] Figure 10 is a sequence diagram illustrating the process that occurs when a failure occurs in the data transfer unit 104 while transferring data from the data transfer unit 104 to the data storage unit 105 in the information collection system shown in Figure 1.

[0063] If, after processing in step 201 in the data source 101 as shown in Figure 2, and processing in step 243 in the data transfer unit 104, an error occurs in step 601 in the data transfer unit 104, then in step 205, the data storage unit 105 saves the data transferred from the data storage unit 105 to the data storage destination database 151, but it is unclear whether all the data has been transferred.

[0064] Therefore, if the data transfer unit 104 recovers from the failure in step 602, it will perform the processing in step 604.

[0065] Figure 11 is a flowchart illustrating the details of the processing in the data transfer unit 104 shown in Figures 9 and 10.

[0066] When the data transfer unit 104 starts processing in step 901, it first determines in step 902 whether there is a triplet of status "received" in the pending status management database 141.

[0067] Then, if there is a triplet of status "received" in the work-in-progress status management database 141, a negative ACK for the triplet of status "received" is returned to the data receiving unit 103 in step 542, as shown in Figure 9.

[0068] If there are no triplets with the status "received" in the pending status management database 141, or if there are triplets with the status "received" in the pending status management database 141, and after returning a negative ACK regarding the triplets with the status "received" to the data receiving unit 103, the data transfer unit 104 determines in step 903 whether there are any triplets with the status "transferring" in the pending status management database 141.

[0069] Then, if there is a set of three items with the status "Transferring" in the work-in-progress status management database 141, in step 641, it is determined that a set of three items with the status "Transferring" has been detected in the work-in-progress status management database 141, and in step 642, a request is made to the data storage unit 105 to delete the data indicated by the data ID of the detected set of three items, and in step 931, the system waits for an ACK from the data storage unit 105 in response.

[0070] In step 651, the data storage unit 105 deletes data from the data storage destination database 151 in accordance with a request from the data transfer unit 104, and in step 652, it returns an ACK to the data transfer unit 104 indicating that the data has been deleted.

[0071] When the data transfer unit 104 receives an ACK from the data storage unit 105, in step 604, it returns a negative ACK to the data receiving unit 103 in step 643, indicating that the collection process of the tripty data that the data storage unit 105 was requested to delete failed.

[0072] Furthermore, the data transfer unit 104 does not delete the triplets stored in the work-in-progress status management database 141, but changes their status from "transferring" to "receiving" in step 644.

[0073] Then, the data transfer unit 104 terminates its processing in step 904.

[0074] When the data receiving unit 103 receives a negative ACK from the data transfer unit 104, in step 661 it returns the negative ACK to the load balancing unit 102 as step 662.

[0075] When the load balancing unit 102 receives a negative ACK from the data receiving unit 103, in step 202 it returns the negative ACK to the data source 101 as step 621.

[0076] Subsequently, the series of processes shown in Figure 2 will be performed again, but since the data transfer unit 104 has not deleted the triplets detected from the work-in-progress status management database 141, the stored timestamp will be used for the retransmitted data.

[0077] (Other embodiments)

[0078] The following describes an embodiment in which, in the information collection system shown in Figure 1, the timing at which data can be deleted on the data sources 101 and 111 side is accelerated by using the resources on the information collection system 1 side.

[0079] First, we will explain the process when no failures occur in the data receiving units 103, 113 and the data transfer units 104, 114.

[0080] Figure 12 is a sequence diagram illustrating another example of processing in the information collection system shown in Figure 1 when no failures occur in the data receiving units 103, 113 and the data transfer units 104, 114.

[0081] After the processing in step 201 in the data source 101 shown in Figure 2, and the processing in step 243 in the data transfer unit 104, the data transfer unit 104 performs the processing in step 1204.

[0082] Figure 13 is a flowchart illustrating the details of the processing in the data transfer unit 104 shown in Figure 12.

[0083] In step 1204, when the data transfer unit 104 receives the data and triplets transferred from the data receiving unit 103, it first determines in step 1801 whether the received triplets and data are stored in the work-in-progress status management database 141. This is done using the data ID within the triplets.

[0084] If the received triplets and data are not stored in the work-in-progress status management database 141, the data transfer unit 104 generates a new record in the work-in-progress status management database 141 in step 1241, saves the data and triplets received from the data receiving unit 103 in the work-in-progress status management database 141, and sets the work-in-progress status of the transfer to "received".

[0085] The data transfer unit 104 saves the data and triplets received from the data receiving unit 103 to the working state management database 141 and sets the working state for the transfer to "received". Alternatively, if the received triplets and data are already stored in the working state management database 141, the data transfer unit 104 returns an ACK to the data receiving unit 103 regarding the received triplets in step 1201.

[0086] When the data receiving unit 103 receives an ACK from the data transfer unit 104, in step 206, it returns an ACK to the load balancing unit 102 as step 261. When the load balancing unit 102 receives an ACK from the data receiving unit 103, it returns an ACK to the data source 101 as step 223 in step 202.

[0087] Subsequently, in step 212, the data source 101 deletes the data it sent to the load balancing unit 102.

[0088] In step 1242, the data transfer unit 104 changes the status of the stored triplicate transfer state from "in progress" to "transferring".

[0089] Figure 14 shows an example of the configuration of the work-in-progress status management databases 141 and 142 shown in Figure 1.

[0090] As shown in Figure 14, the work-in-progress status management databases 141 and 142 shown in Figure 1 have a triplicate consisting of a timestamp 1601, a data name 1602, and a data ID 1603, and the data content 1604 that constitutes that triplicate and its transfer status 1605 are set in association with it.

[0091] Next, in step 243, the data transfer unit 104 transfers the data received from the data receiving unit 103 and the triplets to the data storage unit 105, and in step 802, waits for a response from the data storage unit 105.

[0092] When the data storage unit 105 receives the data and its triplicate transferred from the data transfer unit 104, in the processing in step 205, it saves the received data to the data storage destination database 151 in step 251, and in step 252, it returns an ACK to the data transfer unit 104 indicating that the data received from the data transfer unit 104 has been saved to the data storage destination database 151 in step 251.

[0093] Furthermore, in step 253, the data storage unit 105 uses the time indicated by the timestamp within the triplet as the data creation time, and stores the data and its creation time in the data storage destination database 151 in association with each other.

[0094] When the data transfer unit 104 receives an ACK from the data storage unit 105, in step 1244, it confirms that the triplets and data corresponding to the ACK from the data storage unit 105 have been saved to the data storage unit 105 from among the triplets and data stored in the work-in-progress status management database 141, and changes the work-in-progress status of that transfer to "transfer complete".

[0095] Subsequently, in step 1245, the data transfer unit 104 releases and deletes the temporarily stored triplets and data in the work-in-progress status management database 141, and in step 1405, it terminates the processing in step 1204.

[0096] In this way, the data transfer unit 104 temporarily stores the data along with the triplicate, sends an ACK to the data source 101 via the data receiving unit 103 to indicate that the data has been saved, transfers the triplicate and data to the data storage unit 105, and releases the temporary storage of the triplicate and data after confirming that the data has been saved in the data storage unit 105. This allows for the deletion of data in the data source at an early stage.

[0097] Next, we will explain how to handle the case where a failure occurs in the data transfer unit 104 before transferring data from the data transfer unit 104 to the data storage unit 105.

[0098] Figure 15 is a sequence diagram illustrating another example of processing that occurs when a failure occurs in the data transfer unit 104 before transferring data from the data transfer unit 104 to the data storage unit 105 in the information collection system shown in Figure 1.

[0099] If, after the processing in step 201 in the data source 101 shown in Figure 12 and the processing in step 1201 in the data transfer unit 104, an error occurs in step 1301 in the data transfer unit 104, and then the data transfer unit 104 recovers from the error in step 1302, the data transfer unit 104 performs the processing in step 1304.

[0100] In step 1341, the data transfer unit 104 detects a triplet and data whose status is "received" from the work-in-progress status management database 141, and in step 1342, changes its status to "transferring".

[0101] Next, in step 1343, the data transfer unit 104 transfers the data and triplets stored in the work-in-progress status management database 141 to the data storage unit 105.

[0102] When the data storage unit 105 receives the data and its triplicate transferred from the data transfer unit 104, in the processing in step 205, it saves the received data to the data storage destination database 151 in step 251, and in step 252, it returns an ACK to the data transfer unit 104 indicating that the data received from the data transfer unit 104 has been saved to the data storage destination database 151 in step 251.

[0103] Furthermore, in step 253, the data storage unit 105 uses the time indicated by the timestamp within the triplet as the data creation time, and stores the data and its creation time in the data storage destination database 151 in association with each other.

[0104] When the data transfer unit 104 receives an ACK from the data storage unit 105, in step 1244, it confirms that the triplets and data corresponding to the ACK from the data storage unit 105 have been saved to the data storage unit 105 from among the triplets and data stored in the work-in-progress status management database 141, and changes the work-in-progress status of that transfer to "transfer complete".

[0105] Subsequently, in step 1245, the data transfer unit 104 releases and deletes the temporarily stored triplets and data that were saved in the work-in-progress status management database 141.

[0106] Figure 16 is a flowchart illustrating the details of the processing in the data transfer unit 104 shown in Figure 15.

[0107] In the processing in step 1204, when the data transfer unit 104 starts processing in step 1501, first in step 1502, the data transfer unit 104 determines whether a triple set of status "received" and data are stored in the working status management database 141.

[0108] If the status database 141 contains a triplet of data with the status "received", the data transfer unit 104 performs the process shown in step 1201 in Figure 13 in step 1521.

[0109] Furthermore, if the "received" state trilogy and data are not stored in the "in progress" status management database 141, the data transfer unit 104 determines in step 1503 whether the "transferring" state trilogy and data are stored in the "in progress" status management database 141.

[0110] If the work-in-progress status management database 141 has a triplet of data with the status "transferring", the data transfer unit 104 requests the data storage unit 105 to delete the data indicated by the data ID of the triplet in step 1531, and waits for an ACK from the data storage unit 105 in step 1532.

[0111] Subsequently, when the data transfer unit 104 receives an ACK from the data storage unit 105 indicating that the data has been deleted, in step 1533 it performs the process shown in step 243 of Figure 13.

[0112] The data transfer unit 104 performs the series of processes described above and terminates in step 1504.

[0113] The method executed by the information gathering system of this embodiment can also be described as a software program. Furthermore, this software program can be stored on a storage medium and provided externally via a network. The information gathering system of this embodiment can be realized by a computer executing this software program.

[0114] The embodiments of the present invention described above are illustrative for the purpose of explaining the invention and are not intended to limit the scope of the invention to those embodiments only. Those skilled in the art can implement the present invention in various other forms without departing from the scope of the invention. [Explanation of symbols]

[0115] 100...Data acquisition system, 101,111...Data source, 102...Additional distribution unit, 103,113...Data receiving unit, 104,114...Data transfer unit, 105...Data storage unit, 131,132...Time synchronization generation unit, 141,142...Work in progress status management database, 151...Data storage destination database

Claims

1. Multiple data receiving units share the processing of receiving data from the data source, A plurality of data transfer units are provided corresponding to each of the data receiving units and transfer the data received by the data receiving units, A data storage unit that receives and stores the data transferred from the plurality of data transfer units, It has, The multiple data receiving units perform synchronized time management with each other, assign a timestamp representing the time based on the time management to the received data, and transmit the data and the correspondence information, which associates the data's identification information with the timestamp, to the data transfer unit. The data transfer unit receives the data and the correspondence information from the data receiving unit, temporarily stores the correspondence information, transfers the data and the correspondence information to the data storage unit, and after confirming that the data has been stored in the data storage unit, releases the temporary storage of the correspondence information. If the data transfer unit experiences an error after receiving the data and the correspondence information, and then resumes operation from the error, it sends a response to the data source via the data receiving unit prompting the retransmission of the data, and also retains the correspondence information and uses the timestamp within the correspondence information for the retransmitted data. Data collection system.

2. The data storage unit uses the time indicated by the timestamp in the correspondence information as the creation time of the data, and stores the data in association with the creation time. The data acquisition system according to claim 1.

3. The data transfer unit temporarily stores the data together with the correspondence information, sends a response to the data source via the data receiving unit indicating that the data has been stored, transfers the correspondence information and the data to the data storage unit, and after confirming that the data has been stored in the data storage unit, releases the temporary storage of the correspondence information and the data. The data acquisition system according to claim 1.

4. The aforementioned multiple data receiving units perform synchronized time management with each other using a Global Positioning System or Precision Time Protocol. The data acquisition system according to claim 1.

5. The data collection system according to Claim 1, wherein, upon resumption from the failure, if the mapping information was being transferred to the data storage unit, the data transfer unit requests the data storage unit to delete the data corresponding to the identification information in the mapping information, and after confirming the deletion of the data by the data storage unit, sends a response prompting the data to be retransmitted.

6. A data collection method in a data collection system comprising: a plurality of data receiving units that share the processing of receiving data from a data source; a plurality of data transfer units provided corresponding to each of the data receiving units and transferring the data received by the data receiving units; and a data storage unit that receives and stores the data transferred from the plurality of data transfer units, wherein The multiple data receiving units perform synchronized time management with each other, assign a timestamp representing the time based on the time management to the received data, and transmit the data and the correspondence information, which associates the data's identification information with the timestamp, to the data transfer unit. The data transfer unit receives the data and the correspondence information from the data receiving unit, temporarily stores the correspondence information, transfers the data and the correspondence information to the data storage unit, and after confirming that the data has been stored in the data storage unit, releases the temporary storage of the correspondence information. If the data transfer unit experiences a failure after receiving the data and the correspondence information, and then resumes operation from the failure, it sends a response to the data source via the data receiving unit prompting the retransmission of the data, and also retains the correspondence information and uses the timestamp within the correspondence information for the retransmitted data. Data collection methods.

7. The data storage unit takes the time indicated by the timestamp in the correspondence information as the creation time of the data, and stores the data in association with the creation time. The data collection method according to claim 6.

8. The data collection method according to claim 6, wherein when the data transfer unit resumes operation after the failure, and the mapping information is being transferred to the data storage unit, the data storage unit requests the data storage unit to delete the data corresponding to the identification information in the mapping information, and after confirming the deletion of the data by the data storage unit, sends a response prompting the data to be retransmitted.

9. A data collection program for a computer to operate a data collection system having: a plurality of data receiving units that share the processing of receiving data from a data source; a plurality of data transfer units provided corresponding to each of the data receiving units and transferring the data received by the data receiving units; and a data storage unit that receives and stores the data transferred from the plurality of data transfer units. The multiple data receiving units perform synchronized time management with each other, assign a timestamp representing the time based on the time management to the received data, and transmit the data and the correspondence information, which associates the data's identification information with the timestamp, to the data transfer unit. The data transfer unit receives the data and the correspondence information from the data receiving unit, temporarily stores the correspondence information, transfers the data and the correspondence information to the data storage unit, and after confirming that the data has been stored in the data storage unit, releases the temporary storage of the correspondence information. If the data transfer unit experiences a failure after receiving the data and the correspondence information, and then resumes operation from the failure, it sends a response to the data source via the data receiving unit prompting the retransmission of the data, and also retains the correspondence information and uses the timestamp within the correspondence information for the retransmitted data. A data collection program that causes the aforementioned computer to perform the following action.

10. The data storage unit causes the computer to perform the following actions: set the time indicated by the timestamp in the correspondence information as the creation time of the data, and store the data in association with the creation time. The data acquisition program according to claim 9.

11. When the data transfer unit resumes operation after the failure, if the mapping information was being transferred to the data storage unit, the computer will perform the following actions: request the data storage unit to delete the data corresponding to the identification information in the mapping information, and after confirming that the data storage unit has deleted the data, send a response prompting the computer to retransmit the data. The data acquisition program according to claim 9.

Images