DATA PROCESSING SYSTEM, DATA PROCESSING METHOD, AND DATA PROCESSING PROGRAM

Abstract

To easily and appropriately set an accumulation time of a data registered to a data registration destination.SOLUTION: In a data processing system 20 that receives a data and stores them to a buffer, and executes a processing for outputting the data stored into the buffer to a predetermined data accumulation device 12, the data processing system 20 contains one or more processors, and constructs a processor so that a reference period for referring the data of a data accumulation device 12 is received, an accumulation time for accumulating the data stored in the buffer until the output of the data storage destination is determined as a time within the reference period on the basis of the presence / absence of a data loss that cannot store the data that becomes a storage object to the buffer, and the data in the buffer is output to the data accumulation device 12 in accordance with the accumulation time determined.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a technique for registering data from a data source in a data storage destination. [Background technology]

[0002] Data generated at a site or the like is registered in a data storage destination such as a database, and the data stored in the data storage destination is used for visualization, analysis, and other processing.

[0003] For example, in a processing device that receives generated data and executes a process of registering the data in the data registration destination, if the process of registering the data in the data registration destination is performed sequentially every time the data arrives, the overhead of the process of registering the data in the data registration destination will be large. Therefore, depending on the performance of the processing device (e.g., the size of the buffer, etc.), there is a risk of data loss (data loss) occurring, where the arrived data cannot be stored in the buffer, due to the influence of the process of registering the data in the data registration destination.

[0004] In response to this, a technique is known that improves the efficiency of the registration process at the data registration destination and prevents data loss, for example by aggregating data that arrives within a specified time (accumulation time) and registering it at the data registration destination.

[0005] However, if the accumulation time is too long, it will take a long time to store the data at the data registration destination, and there is a risk that the real-time nature of processing at the data utilization destination that utilizes the data at the data registration destination will be compromised.

[0006] For example, Patent Document 1 discloses a technology for processing stream data that is subject to time constraints and requires strict management. In a device that processes stream data and non-stream data, the technology has two types of queues for requests with parameters, a process for planning the order in which requests are executed based on the parameters, and a process for scanning the queues and executing the optimal requests, thereby switching the order in which requests are executed in real time to optimize processing. [Prior art documents] [Patent documents]

[0007] [Patent Document 1] JP 2001-142649 A Summary of the Invention [Problem to be solved by the invention]

[0008] As mentioned above, if the storage time is too long, it will take a long time to store the data at the data registration destination, and there is a risk that real-time processing at the data utilization destination that utilizes the data from the data registration destination will be compromised; however, there is no known technology for how to set the storage time.

[0009] The present invention has been made in consideration of the above circumstances, and an object of the present invention is to provide a technique that enables easy and appropriate setting of the accumulation time of data to be registered in a data registration destination. [Means for solving the problem]

[0010] In order to achieve the above-mentioned object, a data processing system according to one aspect is a data processing system that receives data, stores it in a buffer, and executes a process of outputting the data stored in the buffer to a specified data storage destination, the data processing system including one or more processors, the processors receive a reference period for referencing data in the data storage destination, determine a storage time, which is the time for which data stored in the buffer is stored until it is output to the data storage destination, to a time within the reference period based on the presence or absence of data loss that prevents data to be stored in the buffer from being stored in the buffer, and output the data in the buffer to the data storage destination in accordance with the determined storage time. Effect of the Invention

[0011] According to the present invention, the accumulation time of data to be registered in a data registration destination can be easily and appropriately set. [Brief description of the drawings]

[0012] [Figure 1] FIG. 1 is a diagram showing the overall configuration of an information processing system according to the first embodiment. [Diagram 2] FIG. 2 is a diagram showing the configuration of a list table according to the first embodiment. [Diagram 3] FIG. 3 is a configuration diagram of a setting screen according to the first embodiment. [Figure 4] FIG. 4 is a flowchart of the accumulation time setting process according to the first embodiment. [Diagram 5] FIG. 5 is a diagram showing an example of a state of a list table according to the first embodiment. [Figure 6] FIG. 6 is a diagram showing another example of the state of the list table according to the first embodiment. [Figure 7] FIG. 7 is a hardware configuration diagram of the data processing system according to the first embodiment. [Figure 8] FIG. 8 is a functional configuration diagram of a data processing system according to the second embodiment. [Figure 9] FIG. 9 is a diagram showing the configuration of a list table according to the second embodiment. [Figure 10] FIG. 10 is a flowchart of the accumulation time setting process according to the second embodiment. [Figure 11] FIG. 11 is a configuration diagram of a data processing system according to a modified example. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0013] The following embodiments will be described with reference to the drawings. Note that the following embodiments do not limit the scope of the invention, and not all of the elements and combinations thereof described in the embodiments are necessarily essential to the solution of the invention.

[0014] FIG. 1 is a diagram showing the overall configuration of an information processing system according to the first embodiment.

[0015] The information processing system 10 includes a data source device 11, a data processing system 20, a data storage device 12, and a data utilization device 13. The data source device 11 and the data processing system 20 are connected via a network 15.

[0016] The data source device 11 is a device that serves as a source of data to be stored in the data storage device 12. The data source device 11 may include, for example, a sensor that detects various conditions. The data source device 11 sequentially transmits data to the data processing system 20.

[0017] Data storage device 12 is an example of a data storage destination, such as a database, that stores data from data source device 11 so that it can be used.

[0018] The data utilization device 13 executes various processes by referring to the data stored in the data storage device 12. For example, the data utilization device 13 executes processes by referring to data required for the processes at a predetermined reference period (for example, 5 seconds).

[0019] The data processing system 20 executes a process for storing the data transmitted from the data source device 11 in the data storage device 12 .

[0020] The data processing system 20 includes a data receiving unit 21, a data output unit 22, and a storage time determination processing unit 23. The data receiving unit 21 performs a process of receiving data (data to be stored) transmitted from the data source device 11 and storing the data in a buffer. The data output unit 22 transmits the data stored in the buffer to the data storage device 12 in accordance with a setting of the time (storage time) for storing the data before transmitting it.

[0021] The storage time determination processing unit 23 is a functional unit that executes a storage time determination process (see FIG. 4) for determining a data storage time in the data output unit 22, and includes a processing unit 30 and a setting unit 40.

[0022] The setting unit 40 includes a priority setting unit 41, a test data setting unit 42, and a resetting unit 43. The priority setting unit 41 sets priorities in the accumulation time determination process in the parameter adjustment unit 33 according to settings on a setting screen 100 (see FIG. 3) described later. The test data setting unit 42 sets test data to be generated by the test data generation unit 34 according to settings on the setting screen 100. The resetting unit 43 performs processing to execute the accumulation time determination process again when resetting of the accumulation time is set.

[0023] The processing unit 30 includes a loss detection unit 31, a throughput detection unit 32, a parameter adjustment unit 33, and a test data generation unit 34. The loss detection unit 31 detects the occurrence of data loss in which the data receiving unit 21 cannot store data from the data source device 11 in a buffer. The throughput detection unit 32 detects the throughput of data storage in the data storage device 12 by the data output unit 22. The parameter adjustment unit 33 executes a storage time determination process for adjusting a parameter (data storage time) in the data output unit 22. The test data generation unit 34 generates test data to be used in the storage time setting process in accordance with the setting of the test data, and performs a process of passing the test data to the data receiving unit 21.

[0024] Next, a list table generated in the accumulation time determination process will be described.

[0025] FIG. 2 is a diagram showing the configuration of a list table according to the first embodiment.

[0026] List table 50 includes an entry for each storage time (candidate storage time) used in the test in the storage time setting process. The entries in list table 50 include fields for storage time 51, throughput value 52, and data loss presence / absence 53. Storage time 51 stores the candidate storage time corresponding to the entry. Throughput value 52 stores the throughput by data output unit 22 when the candidate storage time corresponding to the entry is used. Data loss presence / absence 53 stores the presence or absence of data loss in data receiving unit 21 when the candidate storage time corresponding to the entry is used.

[0027] Next, the setting screen 100 will be described.

[0028] FIG. 3 is a configuration diagram of a setting screen according to the first embodiment.

[0029] The setting screen 100 is a screen for setting various information required for the accumulation time determination process and displaying the execution results of the accumulation time determination process, and is displayed by the processor 81 described later which executes the data processing program, for example, on a display device such as a display connected to the data processing system 20, or on a display device of an information processing device connected to the data processing system 20 via a network.

[0030] The setting screen 100 includes a priority setting area 110 , a test data setting area 120 , an automatic resetting area 130 , a test start button 140 , and an execution result area 150 .

[0031] The priority setting area 110 includes a priority button display area 111 and a reference cycle setting area 112. The priority button display area 111 is an area for setting a priority for setting a storage time, and displays a radio button for selecting whether to prioritize avoidance of loss of data transmitted from the data source device 11 (data loss avoidance) or to prioritize the performance of an application that uses data in the data utilization device (application side performance). The reference cycle setting area 112 is an area for setting a reference cycle for data in the data storage device in an application when application side performance is prioritized.

[0032] The test data setting area 120 includes a test data generation amount setting area 121 and a test data size setting area 122. The test data generation amount setting area 121 is an area for setting the number of pieces of test data per unit time (number of items) that simulates data arriving from the data source device 11. The test data size setting area 122 is an area for setting the size of the test data.

[0033] The automatic resetting area 130 is an area where a radio button for setting whether or not to automatically reset the accumulation time is displayed.

[0034] The test start button 140 is a button for issuing an instruction to start a storage time setting process using test data, and when the test start button 140 is pressed, the storage time setting process is started.

[0035] The execution result area 150 includes a storage time display area 151 and a throughput value display area 152. The storage time display area 151 is an area that displays the storage time set by the storage time setting process. The throughput value display area 152 is an area that displays the value of the throughput when storing data in the data storage device 12 when the storage time set by the storage time setting process is used.

[0036] Next, the processing operation of the data processing system 20 will be described.

[0037] FIG. 4 is a flowchart of the accumulation time setting process according to the first embodiment.

[0038] The accumulation time setting process is executed, for example, when the test start button 140 is pressed on the setting screen 100, or when automatic resetting is set in the parameter adjustment unit 33 and processing is started using the determined accumulation time, and then the loss detection unit 31 detects that data loss has occurred with a frequency equal to or greater than a predetermined frequency.

[0039] First, the setting unit 40 acquires various requirements set on the setting screen 100, and sets the processing unit 30 in accordance with the acquired requirements (S11). Specifically, the priority setting unit 41 sets the priority set on the setting screen 100 in the parameter adjustment unit 33, the test data setting unit 42 sets the requirements for the test data set on the setting screen in the test data generation unit 34, and the resetting unit 43 sets automatic resetting in the parameter adjustment unit 33 when automatic resetting is set on the setting screen 100.

[0040] Next, the parameter adjusting section 33 determines whether or not the application side performance is set as a priority (S12).

[0041] As a result, if the application side performance is set as a priority (S12: Yes), the parameter adjustment unit 33 sets the accumulation time in the data output unit 22 to the data reference period on the setting screen (S13), and the process proceeds to step S15.

[0042] On the other hand, if the application-side performance is not set as a priority (S12: No), the parameter adjusting section 33 sets the accumulation time in the data output section 22 to 0 seconds (S14), and the process proceeds to step S15.

[0043] In step S15, the parameter adjustment unit 33 causes the test data generation unit 34 to start generating test data in accordance with the settings of the test data.

[0044] Next, the parameter adjustment unit 33 causes the test data generation unit 34 to generate test data for a certain period of time and transmit it to the data reception unit 21 (S16). As a result, the data reception unit 21 executes a process of receiving the transmitted data and storing it in a buffer, and the data output unit 22 executes a process of outputting the buffer data to the data storage device 12 according to the set storage time.

[0045] In this manner, when the data receiving unit 21 and the data output unit 22 are performing processing, the loss detection unit 31 detects whether or not data loss has occurred in the data receiving unit 21 and notifies the parameter adjustment unit 33, and the throughput detection unit 32 detects the throughput by the data output unit 22 and notifies the parameter adjustment unit 33.

[0046] Next, the parameter adjustment unit 33 registers the test results using the test data, that is, information on the set storage time, throughput, and the presence or absence of data loss, in the list table 50 (S17).

[0047] Next, the parameter adjustment unit 33 determines whether or not the application-side performance is set as a priority (S18), and if the application-side performance is set as a priority (S18: Yes), the process proceeds to step S19, whereas if the application-side performance is not set as a priority (S18: No), the process proceeds to step S20.

[0048] In step S19, the parameter adjustment unit 33 determines whether the storage time is equal to or less than 0, and if it is satisfied (S19: Yes), it determines one of the storage times without data loss from the list table 50 as the storage time to be actually used, sets it in the data output unit 22 (S22), and ends the process. Here, the storage time to be actually used may be, for example, the shortest storage time when the application side performance is set as a priority, or may be the longest storage time (the same time as the reference period) when there is no storage time without data loss. In addition, the parameter adjustment unit 33 may display the determined storage time in the execution result area 150 of the setting screen 100.

[0049] On the other hand, if the accumulation time does not satisfy the condition of 0 or less (S19: No), the parameter-adjusting unit 33 advances the process to step S21.

[0050] In step S20, the parameter adjustment unit 33 determines whether or not data loss has occurred. If data loss has not occurred (S20: No), the process proceeds to step S22. If data loss has occurred (S20: Yes), the parameter adjustment unit 33 adds (for example, +1) to the accumulation time set in the data output unit 22 (S23), and proceeds to step S15.

[0051] In step S21, the parameter adjustment unit 33 determines whether or not data loss has occurred, and if data loss has not occurred (S21: No), it subtracts (for example, -1) the accumulation time set in the data output unit 22 and sets it (S24), and proceeds to step S15, whereas if data loss has occurred (S21: Yes), it proceeds to step S22.

[0052] Next, some examples of the list table when the accumulation time setting process is performed and the accumulation time determined in each case will be described with reference to Figures 2, 5, and 6. Figure 5 is a diagram showing an example of the state of the list table according to the first embodiment, and Figure 6 is a diagram showing another example of the state of the list table according to the first embodiment.

[0053] The list table in FIG. 2 is an example of a case where a setting is made to prioritize application performance and the reference time is set to 7 seconds. The list table in FIG. 5 is an example of a case where a setting is made to prioritize application performance and the reference time is set to 2 seconds. The list table in FIG. 6 is an example of a case where a setting is made to prioritize application performance, i.e., a setting is made to prioritize avoidance of data loss.

[0054] When the application side performance is set to have priority and the reference time is set to 7 seconds, entries with storage times ranging from the reference time of 7 seconds to 4 seconds, which is the time when data loss occurred, are stored in the list table 50 by the storage time setting process, as shown in Fig. 2. In this case, for example, 5 seconds, which is the shortest storage time in which no data loss occurs, is determined as the storage time to be used. This makes it possible to store data in the data storage device 12 with a storage time equal to or shorter than the reference time in the data utilization device 13 without causing data loss, and makes it possible to appropriately maintain real-time performance in the data utilization device 13.

[0055] Furthermore, when the application performance is set to have priority and the reference time is set to 2 seconds, the accumulation time setting process starts the process from the reference time of 2 seconds as shown in Fig. 5, but since data loss occurs, only entries with an accumulation time of 2 seconds are stored in the list table 50. In this case, 2 seconds, which is the same time as the reference time, is determined as the accumulation time to be used. As a result, although data loss occurs, data can be stored in the data storage device 12 with the same accumulation time as the reference time in the data utilization device 13, and loss of real-time performance in the data utilization device 13 can be suppressed.

[0056] Furthermore, if the application performance is not set as a priority, the storage time setting process starts the process from 0 seconds, as shown in Fig. 6, and entries up to 5 seconds, which is the time when no data loss occurs, are stored in list table 50. In this case, 5 seconds, which is the shortest time when no data loss occurs, is determined as the storage time to be used. This allows data to be stored in data storage device 12 without causing data loss.

[0057] Next, an example of the hardware configuration of the data processing system 20 will be described.

[0058] FIG. 7 is a hardware configuration diagram of the data processing system according to the first embodiment.

[0059] The data processing system 20 is, for example, a computer such as a PC (Personal Computer) or a general-purpose server, and includes a processor 81, a main memory device 82, a secondary memory device 83, and a network interface (NW IF) 84.

[0060] The NW IF 84 is, for example, an interface such as a wired LAN card or a wireless LAN card, and communicates with other devices (such as the data source device 11 and the data storage device 12) via the network 15 or the like.

[0061] The processor 81 is, for example, a CPU (Central Processing Unit), and executes various processes according to programs stored in the main storage device 82 and / or the secondary storage device 83.

[0062] The main memory device 82 is, for example, a RAM (RANDOM ACCESS MEMORY) and stores the programs executed by the processor 81 and necessary information (for example, the list table 50). The main memory device 82 may also be used as a buffer for data received from the data source device 11, for example.

[0063] The secondary storage device 83 is, for example, a hard disk or a flash memory, and stores programs executed by the processor 81 and data used by the processor 81. In this embodiment, the secondary storage device 83 stores a data processing program that executes a storage time determination process.

[0064] In this embodiment, for example, the processor 81 executes a data processing program to configure the data receiving unit 21, the data output unit 22, the processing unit 30, and the setting unit 40.

[0065] Next, a data processing system according to a second embodiment will be described.

[0066] Fig. 8 is a functional configuration diagram of a data processing system according to the second embodiment. In Fig. 8, the same components as those in the data processing system 20 according to the first embodiment are denoted by the same reference numerals.

[0067] The data processing system 20A according to the second embodiment includes a processing unit 30A instead of the processing unit 30. The processing unit 30A includes a shaping unit 35 in addition to the processing unit 30, and includes a parameter adjustment unit 33A instead of the parameter adjustment unit 33.

[0068] The shaping unit 35 performs a thinning process (shaping process) to thin out input data according to a predetermined transmittance. During normal operation, the shaping unit 35 inputs data (data to be stored) obtained by performing shaping process on data received from the data source device 11 to the data receiving unit 21, and during testing (during the accumulation time determination process), inputs data (data to be stored) obtained by performing shaping process on data generated by the test data generating unit 34 to the data receiving unit 21.

[0069] The parameter adjustment unit 33A executes a storage time setting process (see FIG. 10) in which a process of adjusting the transmittance to be set in the shaping unit 35 is added to the storage time setting process by the parameter adjustment unit 33.

[0070] Next, a list table generated in the accumulation time determination process will be described.

[0071] Fig. 9 is a diagram showing the structure of a list table according to the second embodiment. Note that the same parts as those in the list table 50 shown in Fig. 2 are denoted by the same reference numerals.

[0072] List table 60 includes an entry for each storage time (candidate storage time) used in the test. The entry of list table 60 includes fields for storage time 51, throughput value 52, data loss presence / absence 53, and data transmittance 54. Data transmittance 54 stores the ratio (data transmittance) of input data that is shaped by shaping unit 35 and input to data receiving unit 21.

[0073] Next, the processing operation of the data processing system 20A will be described.

[0074] Fig. 10 is a flowchart of the accumulation time setting process according to the second embodiment. Note that the same process steps as those in the accumulation time setting process according to the first embodiment shown in Fig. 4 are denoted by the same reference numerals, and duplicated explanations may be omitted.

[0075] After performing step S13, parameter adjustment unit 33A sets the search flag to OFF (S31), sets the data transmittance of shaping unit 35 to 100% (S32), sets the storage time fixed flag to OFF (S33), and proceeds to step S15. Here, the search flag indicates whether a storage time in which no data loss has occurred has been detected, and the storage time fixed flag indicates whether the storage time should be changed.

[0076] In step S21, if no data loss has occurred (S21: No), the process proceeds to step S34, whereas if data loss has occurred (S21: Yes), the process proceeds to step S35.

[0077] In step S34, the parameter adjustment unit 33A determines whether the accumulation time fixed flag is OFF or not. If the accumulation time fixed flag is not OFF (S34: No), the parameter adjustment unit 33A proceeds to step S22 since this means that the accumulation time is not changed.

[0078] On the other hand, if the accumulation time fixed flag is OFF (S34: Yes), the parameter adjustment unit 33A subtracts (e.g., -1) from the accumulation time set in the data output unit 22 and sets it (S36), sets the search flag to ON (S37), and proceeds to step S15.

[0079] In step S35, the parameter adjustment unit 33A determines whether the search flag is OFF or not. As a result, if the search flag is not OFF (S35: No), this means that a storage time during which no data loss occurred has already been searched for, so the parameter adjustment unit 33A advances the process to step S22.

[0080] On the other hand, if the search flag is OFF (S35: Yes), this means that data loss occurs when the storage time is the same as the reference period, and that data loss cannot be prevented unless the data transmittance is changed, so parameter adjustment unit 33A sets the storage time fixed flag to ON (S38), subtracts the data transmittance (for example, -10%) and sets it in shaping unit 35 (S39), and proceeds to step S15.

[0081] Next, an example of a list table when the storage time setting process is performed, and the storage time and data transparency rate determined in that case will be described with reference to FIG.

[0082] The list table in FIG. 9 is an example in which application performance is prioritized and the reference time is set to 2 seconds.

[0083] When the application performance is set to have priority and the reference time is set to 2 seconds, an entry with only the reference time of 2 seconds as the storage time is stored in the list table 60 by the storage time setting process, as shown in FIG. 9. Then, the data transmission rate 54 stores the data transmission rate subtracted until no data loss occurs (50% in the example of FIG. 9). In this case, 2 seconds, which is the same storage time as the reference time, is determined as the storage time to be used, and the shaping unit 35 sets the data transmission rate to 50%. This allows data to be stored in the data storage device 12 with the storage time as the reference time and with a data transmission rate that does not cause data loss. In this case, a part of the data from the data source device 11 is shaped, but the latest data after shaping can be stored in the data storage device 12 for each reference time, so that the real-time property in the data utilization device 13 can be appropriately maintained.

[0084] Next, a data processing system according to a modified example will be described.

[0085] Fig. 11 is a configuration diagram of a data processing system according to a modified example, in which the same functional units as those in the data processing system shown in Fig. 1 are denoted by the same reference numerals.

[0086] The data processing system 20B according to the modified example includes a gateway 71 and a server 70 connected via a network 75.

[0087] The gateway 71 includes a data receiving unit 72 , a data output unit 73 , and a test data generating unit 74 .

[0088] The data receiving unit 72 receives data transmitted from the data source device 11 and stores the data in a buffer (first buffer) in the gateway 1. The data output unit 73 transmits the data stored in the buffer to the server 70 in accordance with a setting of a time for storing the data before transmitting the data (storage time). The test data generating unit 74 generates test data in accordance with a setting of the test data and passes the test data to the data receiving unit 72.

[0089] The server 70 includes a data receiving unit 21, a data output unit 22, a processing unit 30B, and a setting unit 40. The data receiving unit 21 receives data transmitted from a gateway 71, not from the data source device 11, and stores the data in a buffer (second buffer).

[0090] The processing unit 30B includes a loss detection unit 31B, a throughput detection unit 32B, and a parameter adjustment unit 33B. The loss detection unit 31B detects the occurrence of data loss in which data input by the data reception unit 21 and the data reception unit 72 cannot be stored in a buffer. The throughput detection unit 32B detects the throughput in data output by the data output unit 22 and the data output unit 73. The parameter adjustment unit 33B executes a storage time determination process for adjusting parameters (data storage time) in each of the data output unit 22 and the data output unit 73. The storage time determination process is the same as the storage time determination process shown in FIG. 4, and is performed for each of the gateway 71 and the server 70.

[0091] According to the data processing system 20B, the accumulation time for data output from the gateway 71 to the server 70 can also be appropriately determined.

[0092] The present invention is not limited to the above-described embodiment, and can be modified as appropriate without departing from the spirit of the present invention.

[0093] For example, in the above embodiment, in the accumulation time setting process, test data is input to determine the accumulation time, but the present invention is not limited to this, and, for example, actual data from a data source device may be input to determine the accumulation time.

[0094] In the above-described embodiments, a part or all of the processing performed by the processor may be performed by a dedicated hardware circuit. The programs in the above-described embodiments may be installed from a program source. The program source may be a program distribution server or a recording medium (e.g., a portable recording medium). [Explanation of symbols]

[0095] 10...information processing system, 11...data source device, 12...data storage device, 13...data utilization device, 20, 20A, 20B...data processing system, 21...data reception unit, 22...data output unit, 23...storage time determination processing unit, 30...processing unit, 31...loss detection unit, 32...throughput detection unit, 33...parameter adjustment unit, 34...test data generation unit, 40...setting unit, 41...priority setting unit, 42...test data setting unit, 43...resetting unit

Claims

1. A data processing system that receives data, stores it in a buffer, and outputs the data stored in the buffer to a predetermined data storage destination, the data processing system includes one or more processors; The processor, receiving a reference cycle for referencing data in the data storage destination; determining a storage time, which is a time for storing the data stored in the buffer until it is output to the data storage destination, to a time within the reference period based on the presence or absence of a data loss that prevents the data to be stored in the buffer from being stored in the buffer; The data in the buffer is output to the data storage destination according to the determined storage time. Data processing system.

2. The processor, detecting whether or not a data loss occurs when one or more times within the reference period are set as a candidate storage time that is a candidate for the storage time, and The candidate storage time during which no data loss occurred is determined as the storage time.

2. The data processing system of claim 1.

3. The processor, Among the plurality of candidate storage times, the shortest candidate storage time during which no data loss occurs is determined as the storage time.

3. The data processing system of claim 2.

4. The processor, A plurality of candidate storage times are determined one by one by gradually subtracting from the same value as the reference period, and the shortest candidate storage time during which no data loss occurs is searched for by detecting whether or not a data loss occurs in which data cannot be stored in the buffer.

4. The data processing system of claim 3.

5. The processor, When the data loss occurs during a storage time that is the same as the reference period, the storage time is determined to be the same as the reference period.

2. The data processing system of claim 1.

6. The processor, executing a thinning process for thinning out the input data in accordance with a data transparency ratio indicating a ratio of data to be stored in the buffer among the input data, and storing the data after the thinning process in the buffer; detecting whether or not a data loss occurs when the data to be stored in the buffer is prevented from being stored in the buffer when the data transparency is changed; The data transparency in the case where the data loss does not occur is determined as the data transparency to be used thereafter.

6. The data processing system of claim 5.

7. The processor, Accept the setting of test data to be used as input data, Generate test data according to the settings and use it as the input data.

2. The data processing system of claim 1.

8. The data processing system includes: A gateway and a server connected to the gateway via a network, the buffers include a first buffer in the gateway and a second buffer in the server; The processor includes a processor of the server; the gateway executes a process of receiving input data, storing the data in the first buffer, and outputting the data stored in the first buffer to the server; a processor of the server executes a process of receiving data from the gateway, storing the data in the second buffer, and outputting the data stored in the second buffer to the data storage destination; The processor of the server determining a first storage time, which is a time for storing the data stored in the first buffer until it is output to the server, to a time within the reference period based on the presence or absence of a data loss that prevents the data to be stored in the first buffer from being stored in the first buffer; A second storage time, which is a time for storing the data stored in the second buffer until it is output to the data storage destination, is determined to be within the reference period based on the presence or absence of data loss that prevents the data to be stored in the second buffer from being stored in the second buffer.

2. The data processing system of claim 1.

9. The processor, Accepting instructions to prioritize avoiding data loss, When an instruction to prioritize data loss avoidance is received, a loss avoidance storage time, which is a time for storing data stored in the buffer until it is output to the data storage destination when data loss avoidance is set to be prioritized, is determined to be the longest candidate loss avoidance storage time among the candidate loss avoidance storage times during which no data loss has occurred that makes it impossible to store data in the buffer, when a plurality of times are set as candidate loss avoidance storage times that are candidates for the loss avoidance storage time, When a setting is made to give priority to avoiding data loss, the data in the buffer is output to the data storage destination in accordance with the determined storage time for avoiding data loss.

2. The data processing system of claim 1.

10. The processor, Using actual data from a predetermined data source, the presence or absence of data loss that prevents data to be stored in the buffer from being stored in the buffer is detected.

2. The data processing system of claim 1.

11. The processor, When the occurrence of the data loss satisfies a predetermined condition, a process of re-determining the accumulation time is performed.

11. The data processing system of claim 10.

12. The processor, The determined storage time is displayed.

2. The data processing system of claim 1.

13. 1. A data processing method for a data processing system that receives data, stores it in a buffer, and outputs the data stored in the buffer to a predetermined data storage destination, comprising: The data processing system includes: receiving a reference cycle for referencing data in the data storage destination; determining a storage time, which is a time for storing the data stored in the buffer until it is output to the data storage destination, to a time within the reference period based on the presence or absence of a data loss that prevents the data to be stored in the buffer from being stored in the buffer; The data in the buffer is output to the data storage destination according to the determined storage time. Data processing methods.

14. A data processing program that causes a computer to execute a process of receiving data, storing the data in a buffer, and outputting the data stored in the buffer to a predetermined data storage destination, The computer includes: A reference period for referring to the data in the data storage destination is received. determining a storage time, which is a time for storing the data stored in the buffer until it is output to the data storage destination, to a time within the reference period based on the presence or absence of a data loss that prevents the data to be stored in the buffer from being stored in the buffer; The data in the buffer is output to the data storage destination in accordance with the determined storage time. Data processing program.

Images