Data Linkage System

The data linkage system addresses processing failures by monitoring and re-executing processes, ensuring uninterrupted data integration and storage across multiple information systems.

JP7583350B2Active Publication Date: 2024-11-14KYOCERA DOCUMENT SOLUTIONS INC
View PDF 4 Cites 0 Cited by

Patent Information

Application Number
JP2020034412
Authority / Receiving Office
JP · JP
Patent Type
Patents
Current Assignee / Owner
Filing Date
2020-02-28
Publication Date
2024-11-14
Estimated Expiration
2040-02-28

AI Technical Summary

Technical Problem

Existing data coordination systems fail to handle processing failures during data linkage, leading to incomplete or interrupted data integration across multiple information systems.

Method used

A data linkage system with a process monitoring system that detects failed processes and automatically restarts them, incorporating a data collection system to collect data and a data accumulation system to store it, ensuring seamless data linkage even in the face of processing failures.

Benefits of technology

Ensures continuous data linkage by automatically re-executing failed processes, maintaining data integrity and availability across multiple information systems.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 0007583350000001
    Figure 0007583350000001
  • Figure 0007583350000002
    Figure 0007583350000002
  • Figure 0007583350000003
    Figure 0007583350000003
Patent Text Reader

Abstract

To provide a data cooperation system and a processing monitoring system that allow cooperation of data even if processing fails during the cooperation.SOLUTION: A data cooperation system comprises: a data collection system that collects data held by information systems; and a data storage system that stores the data collected by the data collection system and held by the plurality of information systems, and the data cooperation system comprises a pipeline orchestrator that monitors processing on every stage on the data in the data cooperation system. When the pipeline orchestrator detects failed processing (S401), it re-executes the processing (S404).SELECTED DRAWING: Figure 14
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] The present invention relates to a data linkage system and a process monitoring system that collect and accumulate data held by a plurality of information systems. [Background technology]

[0002] 2. Description of the Related Art Conventionally, information processing systems that cooperate with a plurality of external systems are known (see, for example, Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2016-006624 A Summary of the Invention [Problem to be solved by the invention]

[0004] However, Patent Document 1 does not describe behavior when processing fails midway through a data linkage system that collects and accumulates data held by multiple information systems.

[0005] SUMMARY OF THE PRESENT DISCLOSURE In view of the above, an object of the present invention is to provide a data linkage system and a process monitoring system that are capable of linking data even if processing fails midway through the linkage. [Means for solving the problem]

[0006] The data linkage system of the present invention is a data linkage system comprising a data collection system which collects data held by information systems, and a data storage system which stores data held by multiple of the information systems and collected by the data collection system, and is further characterized in that it comprises a process monitoring system which monitors each stage of processing of data in the data linkage system, and which re-executes the processing if it detects that the processing has failed.

[0007] With this configuration, if the data linkage system of the present invention detects a failed process at each stage of processing of data in the data linkage system itself, it will re-execute the process, making it possible to link data even if the process fails midway through the linkage.

[0008] The data linkage system of the present invention includes an application unit that uses data stored in the data storage system, and the processing monitoring system may cause the data collection system to execute the process of collecting the data requested to be updated by the application unit when the application unit requests updating of the data stored in the data storage system.

[0009] With this configuration, when the application unit requests the processing monitoring system to update the data stored in the data storage system, the data linkage system of the present invention has the data collection system execute a process to collect the data requested to be updated by the application unit, thereby making it possible to easily update the data stored in the data storage system.

[0010] The processing monitoring system of the present invention is a processing monitoring system that monitors each stage of processing of data in a data integration system that includes a data collection system that collects data held by information systems, and a data storage system that stores data held by multiple of the information systems and collected by the data collection system, and is characterized in that if the processing monitoring system detects that the processing has failed, it re-executes the processing.

[0011] With this configuration, if the processing monitoring system of the present invention detects a failed process at each stage of processing for data in the data linkage system, it re-executes the process, so that data can be linked by the data linkage system even if the processing fails midway through the linkage by the data linkage system. Effect of the Invention

[0012] The data linkage system and process monitoring system of the present invention are capable of linking data even if a process fails during the linkage. [Brief description of the drawings]

[0013] [Figure 1] FIG. 1 is a block diagram of a system according to an embodiment of the present invention. [Diagram 2] FIG. 2 is a block diagram of a pipeline provided in the data storage system shown in FIG. [Diagram 3] Figure 1 shows a block diagram of the pipeline orchestrator. [Figure 4] FIG. 2 is a diagram showing an example of the flow of operation of the system shown in FIG. 1 when data held by the information system is collected by a POST connector and sent to a pipeline. [Diagram 5] 5 is a flowchart of the operation of the POST connector shown in Figure 4 when sending a file to a pipeline. [Figure 6]FIG. 2 is a diagram showing an example of the flow of operation of the system shown in FIG. 1 when data held by the information system is collected by a GET connector and passed to a pipeline. [Figure 7] FIG. 2 is a diagram showing an example of the flow of operation of the system shown in FIG. 1 when data held by the information system is collected by a POST agent and transmitted to a pipeline. [Figure 8] 8 is a flowchart of the operation of the POST agent shown in FIG. 7 when sending a file to a pipeline. [Figure 9] 2 is a diagram showing an example of the flow of operations of the system shown in FIG. 1 when data held by the information system is collected by a GET agent and passed to a pipeline. FIG. [Figure 10] 2 is a sequence diagram of a portion of the operation of the data linkage system shown in FIG. 1 when the data storage system stores data. [Figure 11] 11 is a sequence diagram of operations subsequent to the operations shown in FIG. 10. FIG. [Figure 12] 11 is a flowchart showing the operation of a masking processing unit in the masking process shown in FIG. 10. [Figure 13] 13 is a diagram showing an example of a data management table used in the operation shown in FIG. 12. FIG. [Figure 14] 10 is a sequence diagram of the operation of the data linkage system shown in FIG. 1 when a masking processing unit fails to process data. [Figure 15] 1 when an application unit requests updating of data of a specific information system stored in the data storage system. FIG. [Figure 16] 10 is a flowchart of the operation of the data linkage system shown in FIG. 1 when changing its own configuration in response to a change in the configuration of a specific information system. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0014] Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

[0015] First, the configuration of a system according to an embodiment of the present invention will be described.

[0016] FIG. 1 is a block diagram of a system 10 according to the present embodiment.

[0017] As shown in FIG. 1, the system 10 includes a data source unit 20 that generates data, and a data linking system 30 that links the data generated by the data source unit 20.

[0018] The data source unit 20 includes an information system 21 that generates data. The information system 21 includes a configuration management server 21a that stores the configuration and settings of the information system 21. The data source unit 20 may include at least one other information system in addition to the information system 21. Examples of information systems include an IoT (Internet of Things) system such as a remote management system that remotely manages image forming devices such as MFPs (Multifunction Peripherals) and dedicated printers, and in-house systems such as ERP (Enterprise Resource Planning) and production management systems. Each of the information systems may be configured by one computer or multiple computers. The information system may hold a file of structured data. The information system may hold a file of unstructured data. The information system may hold a database of structured data.

[0019] The data source unit 20 includes a POST connector 22 as a data collection system that acquires files of structured data or unstructured data held by an information system and transmits the acquired files to a pipeline of the data linkage system 30, which will be described later. The data source unit 20 may include at least one POST connector having a similar configuration to the POST connector 22, in addition to the POST connector 22. The POST connector may be configured by a computer that constitutes the information system from which the POST connector itself acquires files. The POST connector is also a component of the data linkage system 30.

[0020] The data source unit 20 includes a POST agent 23 as a data collection system that acquires structured data from a database of structured data held by the information system and transmits the acquired structured data to a pipeline (described later) of the data linkage system 30. The data source unit 20 may include at least one other POST agent having a similar configuration to the POST agent 23 in addition to the POST agent 23. The POST agent may be constituted by a computer that constitutes the information system from which the POST agent itself acquires structured data. The POST agent is also a component of the data linkage system 30.

[0021] The data source unit 20 includes a GET agent 24 as a data collection system that generates structured data for linkage based on data held by the information system. The data source unit 20 may include at least one GET agent having a similar configuration to the GET agent 24 in addition to the GET agent 24. The GET agent may be configured by a computer that constitutes the information system that holds the data that was the basis for generating the structured data for linkage. The GET agent is also a component of the data linkage system 30.

[0022] The data linkage system 30 includes a data storage system 40 that stores data generated by the data source unit 20, an application unit 50 that utilizes the data stored in the data storage system 40, and a control service unit 60 that executes various types of control over the data storage system 40 and the application unit 50.

[0023] The data storage system 40 includes a pipeline 41 that stores data generated by the data source unit 20. The data storage system 40 may include at least one other pipeline in addition to the pipeline 41. Since the configuration of data in an information system may differ from one information system to another, the data storage system 40 basically includes a pipeline for each information system. Each of the pipelines may be configured by one computer or may be configured by multiple computers.

[0024] FIG. 2 is a block diagram of a pipeline 70 included in the data storage system 40.

[0025] As shown in FIG. 2, the pipeline 70 includes a primary storage 71 having a storage area for storing data received from a POST connector, a POST agent, a GET connector described later, or a GET agent described later, a masking processing unit 72 as a data conversion system that performs a masking process as a data conversion process on data related to privacy such as personal information of users of the information system among the data stored in the primary storage 71, a data transfer processing unit 73 that performs a data transfer process for transferring the data masked by the masking processing unit 72 to a big data analysis unit 44 (see FIG. 1) described later, and a secondary storage 74 having a storage area for accumulating data for transfer to the big data analysis unit 44. The primary storage 71 is provided because, in the case where a process fails in a process subsequent to the process of storing data in the primary storage 71, such as a masking process process or a data transfer process process, the failed process can be re-executed using the data stored in the primary storage 71 without re-transmitting the data from the data source unit 20 to the data linkage system 30, which involves high network communication costs. Each of the primary storage 71 and the secondary storage 74 is not simply a storage device, but a system capable of executing various processes described below.

[0026] 1, the data accumulation system 40 includes a GET connector 42 as a data collection system that acquires files of structured data or unstructured data held by an information system and links the acquired files to a pipeline. The data accumulation system 40 may include at least one GET connector having a similar configuration to the GET connector 42 in addition to the GET connector 42. The GET connector may be configured by a computer that configures a pipeline in which the GET connector itself links files.

[0027] Note that system 10 has a POST connector in data source unit 20 for information systems that do not support retrieval of structured data or unstructured data files from data storage system 40. On the other hand, system 10 has a GET connector in data storage system 40 for information systems that support retrieval of structured data or unstructured data files from data storage system 40.

[0028] The data accumulation system 40 includes a GET agent 43 as a data collection system that acquires structured data generated by a GET agent and links the acquired structured data to a pipeline. The data accumulation system 40 may include at least one other GET agent having a similar configuration to the GET agent 43 in addition to the GET agent 43. The GET agent may be configured by a computer that configures a pipeline in which the GET agent itself links structured data.

[0029] For information systems that do not support structured data being acquired from data storage system 40, system 10 is provided with a POST agent in data source unit 20. On the other hand, for information systems that support structured data being acquired from data storage system 40, system 10 is provided with a GET agent in data source unit 20 and a GET agent in data storage system 40.

[0030] The data accumulation system 40 includes a big data analysis unit 44 as a data conversion system that executes a final conversion process as a data conversion process that converts data accumulated through a plurality of pipelines into a form that can be searched or tabulated using a query language, such as a database language, such as SQL. The big data analysis unit 44 can also execute a search or tabulation on the data that has been subjected to the final conversion process in response to a search request or tabulation request from the application unit 50. The big data analysis unit 44 may be configured by one computer or may be configured by multiple computers.

[0031] The final conversion process may include a data integration process for integrating data from a plurality of information systems as a data conversion process. When the system 10 includes a remote management system arranged in Asia to remotely manage a large number of image forming devices arranged in Asia, a remote management system arranged in Europe to remotely manage a large number of image forming devices arranged in Europe, and a remote management system arranged in America to remotely manage a large number of image forming devices arranged in America as information systems, each of these three remote management systems includes a device management table for managing the image forming devices managed by the remote management system itself. The device management table is information indicating various information of the image forming device in association with an ID assigned to each image forming device. Here, since each of the three remote management systems includes its own device management table, there is a possibility that the same ID is assigned to different image forming devices among the device management tables of the three remote management systems. Therefore, when the big data analysis unit 44 integrates the device management tables of the three remote management systems to generate one device management table, it reassigns the ID of the image forming device so that there is no duplication.

[0032] The application unit 50 includes an application service 51 that uses data managed by the big data analysis unit 44 to execute a specific operation instructed by a user, such as displaying data or analyzing data. The application unit 50 may include at least one application service in addition to the application service 51. Each of the application services may be configured by one computer or multiple computers.

[0033] The application unit 50 includes an API platform 52 that provides an API (Application Program Interface) that executes a specific operation by using data managed by the big data analysis unit 44. The API platform 52 may be configured by one computer or may be configured by multiple computers. For example, the APIs provided by the API platform 52 include an API that transmits data on the remaining amount of consumables collected from the image forming device by a remote management system to a consumable ordering system outside the system 10, which orders consumables when the remaining amount of consumables such as toner in the image forming device is equal to or less than a specific amount, and an API that transmits various data collected from the image forming device by a remote management system to a failure prediction system outside the system 10, which predicts failures of the image forming device.

[0034] The control service unit 60 includes a pipeline orchestrator 61 as a processing monitoring system that monitors each stage of processing of data in the data source unit 20, the data storage system 40, and the application unit 50. The pipeline orchestrator 61 may be configured by one computer or may be configured by multiple computers.

[0035] FIG. 3 is a block diagram of the pipeline orchestrator 61.

[0036] As shown in FIG. 3, the pipeline orchestrator 61 includes a trigger processing unit 81 that processes triggers for the operation of the pipeline orchestrator 61, an action description unit 82 that stores multiple operation scenarios for the pipeline orchestrator 61, and an action processing unit 83 that executes the operation of the pipeline orchestrator 61.

[0037] 1, the control service unit 60 includes a configuration management server 62 that stores the configuration and settings of the data storage system 40 and automatically executes deployment as necessary. The configuration management server 62 may be configured with one computer or multiple computers. The configuration management server 62 constitutes a configuration change system that changes the configuration of the data linkage system 30.

[0038] The control service unit 60 includes a configuration management gateway 63 that connects to a configuration management server of the information system and collects information for detecting changes in the configuration of databases and unstructured data in the information system, i.e., changes in the configuration of data in the information system. The configuration management gateway 63 may be configured by one computer or multiple computers.

[0039] The control service unit 60 includes a key management service 64 that encrypts and stores security information such as key information and connection strings required for linking systems such as information systems. The key management service 64 may be configured by one computer or multiple computers.

[0040] The control service unit 60 includes a management API 65 that receives requests from the data storage system 40 and the application unit 50. The management API 65 may be configured by one computer, or may be configured by multiple computers.

[0041] The control service unit 60 includes an authentication and authorization service 66 that executes authentication and authorization of the application service of the application unit 50. The authentication and authorization service 66 may be configured with one computer or may be configured with multiple computers. The authentication and authorization service 66 can, for example, confirm whether the application service is permitted to request updating of the information system data stored in the data storage system 40.

[0042] The operation of the system 10 will now be described.

[0043] First, the operation of the system 10 when data held by the information system 21 is collected by the POST connector 22 and transmitted to the pipeline 41 will be described.

[0044] FIG. 4 is a diagram showing an example of the flow of operation of the system 10 when data held by the information system 21 is collected by the POST connector 22 and transmitted to the pipeline 41.

[0045] In the example shown in FIG. 4, the information system 21 is a production management system 100.

[0046] As shown in FIG. 4, the production management system 100 includes a production management server 101 that executes production management, and a storage 102 for storing files of structured data or unstructured data.

[0047] The production management server 101 executes a backup by storing a file of structured data or unstructured data in the storage 102 by batch processing (S201).

[0048] After the process of S201, the production management server 101 instructs the POST connector 22 to transfer the file stored in the storage 102 in S201 to the pipeline (S202). Here, the production management server 101 includes identification information of the file stored in the storage 102 in S201 in the instruction in S202.

[0049] Upon receiving the instruction in S202, the POST connector 22 acquires from the storage 102 the file specified by the identification information included in the instruction in S202 (S203).

[0050] After the process of S203, the POST connector 22 transmits the file acquired in S203 to the pipeline 41 to which the POST connector 22 itself is associated (S204).

[0051] FIG. 5 is a flowchart of the operation of the POST connector 22 when sending a file to the pipeline 41.

[0052] 5, the POST connector 22 assigns a transaction ID as identification information to the current transaction of transmitting a file to the pipeline 41 (S221). Here, the transaction ID is, for example, a numerical value, and is incremented every time a new transaction occurs in the POST connector 22.

[0053] The POST connector 22 determines whether the target data of the current transaction is larger than a specific processing unit (S222), where the specific processing unit is, for example, a specific number of files.

[0054] If the POST connector 22 determines in S222 that the target data of the current transaction is larger than the specific processing unit, it divides the target data of the current transaction into specific processing units (S223).

[0055] When the POST connector 22 determines in S222 that the data of the current transaction is equal to or smaller than a specific processing unit, or when the processing of S223 ends, it assigns a processing ID as identification information to each processing unit of data (S224). Here, the processing ID is, for example, a numerical value, and is incremented each time data of a specific processing unit is newly generated in the POST connector 22 in the same transaction.

[0056] After the process of S224, the POST connector 22 starts to transmit the target data of the current transaction to the pipeline 41 in units of processing (S225).

[0057] Next, the POST connector 22 determines whether the number of files sent to the pipeline 41 per specific unit time has exceeded a specific number (S226).

[0058] If the POST connector 22 determines in S226 that the number of files sent to the pipeline 41 per specific unit time does not exceed a specific number, it determines whether or not the sending of the data that is the subject of this transaction to the pipeline 41 has been completed (S227).

[0059] If the POST connector 22 determines in S227 that the transmission of the target data of the current transaction to the pipeline 41 has not been completed, it executes the process of S226.

[0060] When the POST connector 22 determines in S227 that the transmission of the target data of the current transaction to the pipeline 41 is completed, the operation shown in FIG. 5 ends.

[0061] When the POST connector 22 determines in S226 that the number of files transmitted to the pipeline 41 per specific unit time has exceeded a specific number, it instructs the pipeline orchestrator 61 to scale out the pipeline 41 and to start parallel processing by the pipeline 41 (S228). Therefore, in accordance with the instruction in S227, the pipeline orchestrator 61 scales out the pipeline 41 to a specific state and instructs the pipeline 41 to start parallel processing.

[0062] Next, the POST connector 22 determines whether or not the transmission of the data that is the subject of the current transaction to the pipeline 41 has been completed until it determines that the transmission of the data that is the subject of the current transaction to the pipeline 41 has been completed (S229).

[0063] When the POST connector 22 determines in S229 that the transmission of the target data of the current transaction to the pipeline 41 is completed, it instructs the pipeline orchestrator 61 to scale in the pipeline 41 and to end the parallel processing by the pipeline 41 (S230). Therefore, in accordance with the instruction in S230, the pipeline orchestrator 61 scales in the pipeline 41 to its original state and instructs the pipeline 41 to end the parallel processing.

[0064] After the process of S230, the POST connector 22 ends the operation shown in FIG.

[0065] The operation of the system 10 will now be described when data held by the information system is collected by the GET connector 42 and passed to the pipeline.

[0066] FIG. 6 is a diagram illustrating an example of the flow of operation of the system 10 when data held by the information system is collected by a GET connector 42 and passed to the pipeline.

[0067] In the example shown in Fig. 6, the information system is an image forming apparatus remote management system 120. The example shown in Fig. 6 is an example of an operation when a user instructs the remote management system 120 to acquire a maintenance report including sensor information including output values ​​of various sensors of the image forming apparatus.

[0068] 6, the remote management system 120 includes a user communication server 121 for receiving instructions from a user, a back-end processing server 122 for executing processing according to the instructions from the user, a command server 123 for transmitting various commands to the image forming apparatus, a device communication server 124 for receiving data from the image forming apparatus, a database 125 for storing various data of the image forming apparatuses managed by the remote management system 120, and a storage 126 for storing files of structured data or unstructured data. The remote management system 120 manages a large number of image forming apparatuses including an image forming apparatus 130. The database 125 stores device IDs as identification information of the image forming apparatuses for the image forming apparatuses managed by the remote management system 120.

[0069] A user of the remote management system 120 can send an instruction to the remote management system 120 to obtain a maintenance report of the image forming device 130. This instruction includes the device ID of the image forming device 130 from which the maintenance report is to be obtained. When the user communication server 121 of the remote management system 120 receives the instruction to obtain the maintenance report, it sends the received instruction to the backend processing server 122 (S251).

[0070] When the backend processing server 122 receives the instruction to obtain the maintenance report sent by the user communication server 121 in S251, it sends a request to the command server 123 to send a maintenance report obtainment command to obtain the maintenance report (S252). This request includes the device ID included in the instruction to obtain the maintenance report.

[0071] When the command server 123 receives a request to send a maintenance report acquisition command sent by the back-end processing server 122 in S252, it sends the maintenance report acquisition command to the image forming device 130 identified by the device ID included in the request (S253).

[0072] When the image forming apparatus 130 receives the maintenance report acquisition command transmitted by the command server 123 in S253, the image forming apparatus 130 transmits a maintenance report of the image forming apparatus 130 itself to the remote management system 120 (S254). Here, the image forming apparatus 130 includes its own device ID in the maintenance report.

[0073] When the device communications server 124 of the remote management system 120 receives the maintenance report transmitted by the image forming apparatus 130 in S254, it determines whether the device ID included in the received maintenance report is included in the database 125 (S255).

[0074] When the device communication server 124 determines in S255 that the device ID included in the received maintenance report is included in the database 125, the device communication server 124 stores the received maintenance report in the storage 126 (S256).

[0075] The GET connector 42 of the data linkage system 30 periodically searches the storage 126 of the remote management system 120, which is the information system to which the GET connector 42 itself is associated, for a maintenance report file of a specific image forming apparatus (S257).

[0076] When the GET connector 42 confirms that the maintenance report file for the specific image forming device 130 exists in the storage 126, it obtains this file from the storage 126 (S258).

[0077] After the process of S258, the GET connector 42 passes the file acquired in S258 to the pipeline to which the GET connector 42 itself is associated (S259).

[0078] When passing a file to a pipeline, the GET connector 42 executes the same operation as that shown in FIG. 5. That is, the GET connector 42 assigns a transaction ID to the current transaction. When the data targeted by the current transaction is larger than a specific processing unit, the GET connector 42 divides the data targeted by the current transaction into specific processing units. The GET connector 42 assigns a processing ID to each processing unit of data. When the number of files passed to the pipeline per specific unit time exceeds a specific number, the GET connector 42 instructs the pipeline orchestrator 61 to scale out the pipeline and start parallel processing by the pipeline, and thereafter, when the data targeted by the current transaction has been passed to the pipeline, the GET connector 42 instructs the pipeline orchestrator 61 to scale in the pipeline and end parallel processing by the pipeline.

[0079] Next, the operation of the system 10 when data held by the information system is collected by the POST agent 23 and sent to the pipeline will be described.

[0080] FIG. 7 is a diagram showing an example of the flow of operation of the system 10 when data held by the information system is collected by the POST agent 23 and sent to the pipeline.

[0081] In the example shown in Fig. 7, the information system is a remote management system 120 for an image forming device, similar to the example shown in Fig. 6. Database 125 stores event information indicating an event that has occurred in an image forming device that is a target of management by remote management system 120. The example shown in Fig. 7 is an example of the operation of system 10 in the case where image forming device 130 that is a target of management by remote management system 120 transmits event information indicating an event that has occurred in image forming device 130 itself to remote management system 120.

[0082] When an event such as an error occurs in the image forming device 130 itself, the image forming device 130 transmits event information indicating the event that occurred in the image forming device 130 itself to the device communication server 124 of the remote management system 120 (S271). For example, errors that occur in the image forming device 130 include a paper jam indicating that paper is jammed inside the image forming device 130 and a cover open indicating that the cover of the image forming device 130 is open.

[0083] When the device communications server 124 of the remote management system 120 receives the event information transmitted by the image forming apparatus 130 in S271, the device communications server 124 updates the database 125 with the received event information (S272).

[0084] The POST agent 23 checks at a specific timing whether the event information stored in the database 125 has been changed (S273). The check in S273 may be performed, for example, when the database 125 is periodically backed up, when the database 125 itself detects a change in the database 125, or when an API for changing the database 125 is called in the remote management system 120.

[0085] If the POST agent 23 detects a change in the event information in the database 125 as a result of the check in S273, it obtains data indicating the content of the change in the event information from the database 125 (S274).

[0086] After the process of S274, the POST agent 23 transmits the data acquired in S274 to the pipeline of the data linkage system 30 with which the POST agent 23 itself is associated (S275).

[0087] FIG. 8 is a flowchart showing the operation of the POST agent 23 when sending a file to a pipeline.

[0088] 8, the POST agent 23 assigns a transaction ID to the current transaction of transmitting a file to the pipeline (S291). Here, the transaction ID is, for example, a numerical value, and is incremented every time a new transaction occurs in the POST agent 23.

[0089] The POST agent 23 determines whether the target data of the current transaction is larger than a specific processing unit (S292), where the specific processing unit is, for example, a specific number of tables.

[0090] If the POST agent 23 determines in S292 that the target data of the current transaction is larger than the specific processing unit, it divides the target data of the current transaction into specific processing units (S293).

[0091] When the POST agent 23 determines in S292 that the target data of the current transaction is equal to or smaller than a specific processing unit, or when the processing of S293 ends, it assigns a processing ID as identification information to each processing unit of data (S294). Here, the processing ID is, for example, a numerical value, and is incremented each time data of a specific processing unit is newly generated in the POST agent 23 in the same transaction.

[0092] After the process of S294, the POST agent 23 starts to transmit the target data of the current transaction to the pipeline in units of processing (S295).

[0093] Next, the POST agent 23 determines whether the amount of data sent to the pipeline per specific unit time has exceeded a specific amount (S296).

[0094] If the POST agent 23 determines in S296 that the amount of data sent to the pipeline per specific unit time does not exceed a specific amount, it determines whether or not the sending of the data that is the subject of this transaction to the pipeline has been completed (S297).

[0095] If the POST agent 23 determines in S297 that the transmission of the target data of the current transaction to the pipeline has not been completed, it executes the process of S296.

[0096] When the POST agent 23 determines in S297 that the transmission of the target data of the current transaction to the pipeline is completed, the operation shown in FIG. 8 ends.

[0097] When the POST agent 23 determines in S296 that the amount of data sent to the pipeline per specific unit time has exceeded a specific amount, it instructs the pipeline orchestrator 61 to scale out the pipeline and start parallel processing by the pipeline (S298). Therefore, the pipeline orchestrator 61 scales out the pipeline to a specific state in accordance with the instruction in S297, and instructs the pipeline to start parallel processing.

[0098] Next, the POST agent 23 determines whether or not the transmission of the data that is the subject of the current transaction to the pipeline has been completed until it determines that the transmission of the data that is the subject of the current transaction to the pipeline has been completed (S299).

[0099] When the POST agent 23 determines in S299 that the transmission of the target data of the current transaction to the pipeline is completed, it instructs the pipeline orchestrator 61 to scale in the pipeline and to end the parallel processing by the pipeline (S300). Therefore, the pipeline orchestrator 61 scales in the pipeline to its original state in accordance with the instruction in S300, and instructs the pipeline to end the parallel processing.

[0100] After the process of S300, the POST agent 23 ends the operation shown in FIG.

[0101] Next, the operation of system 10 when data held by the information system is collected by GET agent 43 and passed to the pipeline will be described.

[0102] FIG. 9 is a diagram showing an example of the flow of operation of the system 10 when data held by the information system is collected by a GET agent 43 and passed to the pipeline.

[0103] In the example shown in FIG. 9, the information system is a production management system 100, similar to the example shown in FIG.

[0104] As shown in FIG. 9, the GET agent 24 of the production management system 100 generates structured data for linkage at a specific timing based on the data stored in the storage 102 (S321).

[0105] The GET agent 43 of the data linkage system 30 periodically inquires of the GET agent 24 of the production management system 100, which is the information system with which the GET agent 43 itself is associated, about the presence or absence of structured data for linkage (S322).

[0106] When the GET agent 43 confirms that the structured data for linkage exists in the GET agent 24, it acquires this structured data from the GET agent 24 (S323).

[0107] After the process of S323, the GET agent 43 passes the structured data acquired in S323 to the pipeline to which the GET agent 43 itself is associated (S324).

[0108] When passing a file to a pipeline, the GET agent 43 executes the same operation as that shown in FIG. 8. That is, the GET agent 43 assigns a transaction ID to the current transaction. When the target data of the current transaction is larger than a specific processing unit, the GET agent 43 divides the target data of the current transaction into specific processing units. The GET agent 43 assigns a processing ID to each processing unit of data. When the amount of data passed to the pipeline per specific unit time exceeds a specific amount, the GET agent 43 instructs the pipeline orchestrator 61 to scale out the pipeline and start parallel processing by the pipeline, and thereafter, when the passing of the target data of the current transaction to the pipeline is completed, the GET agent 43 instructs the pipeline orchestrator 61 to scale in the pipeline and end parallel processing by the pipeline.

[0109] Next, the operation of the data linkage system 30 when the data storage system 40 stores data will be described.

[0110] FIG. 10 is a sequence diagram of part of the operation of the data linkage system 30 when the data storage system 40 stores data.

[0111] 10, when the primary storage 71 of the pipeline 70 receives data of a specific processing unit from a data collection system, i.e., a POST connector, a POST agent, a GET connector, or a GET agent, the primary storage 71 stores the received data (S341). Next, the primary storage 71 notifies the pipeline orchestrator 61 of an event indicating completion of data storage (S342).

[0112] When the trigger processing unit 81 of the pipeline orchestrator 61 receives the event notified by the primary storage 71 in S342, it analyzes the contents of this event, calls a scenario corresponding to this event, i.e., a scenario for masking processing, from the action description unit 82 (S343), and notifies the action processing unit 83 of the scenario called in S343 (S344). Therefore, the action processing unit 83 instructs the masking processing unit 72 of the pipeline 70 to execute processing based on the scenario notified in S344, i.e., to execute masking processing on the data stored in the primary storage 71 in S341 (S345).

[0113] Upon receiving the instruction in S345, the masking processing unit 72 executes masking processing on the data stored in the primary storage 71 in S341. That is, the masking processing unit 72 first acquires the data stored in the primary storage 71 in S341 from the primary storage 71 (S346). Next, the masking processing unit 72 executes masking processing on the data acquired in S346 (S347). Next, the masking processing unit 72 transfers the data on which the masking processing has been executed in S347 to the data transfer processing unit 73 (S348). Then, the masking processing unit 72 notifies the pipeline orchestrator 61 of an event indicating the completion of the masking processing (S349).

[0114] When the trigger processing unit 81 of the pipeline orchestrator 61 receives the event notified by the masking processing unit 72 in S349, it analyzes the contents of this event, calls a scenario corresponding to this event, i.e., a scenario for data transfer processing, from the action description unit 82 (S350), and notifies the action processing unit 83 of the scenario called in S350 (S351). Therefore, the action processing unit 83 instructs the data transfer processing unit 73 of the pipeline 70 to execute processing based on the scenario notified in S351, i.e., to execute data transfer processing for the data on which masking processing was performed in S347 (S352).

[0115] FIG. 11 is a sequence diagram of operations subsequent to the operations shown in FIG.

[0116] 11, upon receiving the instruction in S352, the data transfer processing unit 73 executes data transfer processing on the data masked by the masking processing unit 72. That is, the data transfer processing unit 73 first accumulates the data passed from the masking processing unit 72 in S348 in the secondary storage 74 as data to be transferred to the big data analysis unit 44 (S353). Next, the data transfer processing unit 73 transfers the data accumulated in the secondary storage 74 in S353 to the big data analysis unit 44 via the secondary storage 74 (S354). Then, the data transfer processing unit 73 notifies the pipeline orchestrator 61 of an event indicating the completion of the data transfer processing (S355).

[0117] When the trigger processing unit 81 of the pipeline orchestrator 61 receives the event notified by the data transfer processing unit 73 in S355, it analyzes the contents of this event, calls a scenario corresponding to this event, i.e., a scenario for final conversion processing, from the action description unit 82 (S356), and notifies the action processing unit 83 of the scenario called in S356 (S357). Therefore, the action processing unit 83 instructs the big data analysis unit 44 to execute processing based on the scenario notified in S357, i.e., to execute final conversion processing on the data accumulated in the secondary storage 74 in S354 (S358).

[0118] Upon receiving the instruction in S358, the big data analysis unit 44 executes a final conversion process on the data transferred by the data transfer processing unit 73. That is, the big data analysis unit 44 first converts the data transferred from the data transfer processing unit 73 in S354 into a form that can be searched and aggregated using a specific query language (S359). Then, the big data analysis unit 44 notifies the pipeline orchestrator 61 of an event indicating the completion of the final conversion process (S360).

[0119] Next, the operation of the masking processing unit 72 in the masking process of S347 will be described.

[0120] FIG. 12 is a flowchart showing the operation of the masking processor 72 in the masking process.

[0121] The masking processor 72 executes the operation shown in FIG. 12 for each processing unit of data.

[0122] As shown in FIG. 12, the masking processing unit 72 writes information indicating that masking processing is being performed on the data that is the target of this masking processing, into the data management table 90 (see FIG. 13) as data management information that manages the processing history of the data that is the target of linkage (S381).

[0123] FIG. 13 is a diagram showing an example of the data management table 90 used in the operation shown in FIG.

[0124] The data management table 90 shown in FIG. 13 includes a transaction ID, a processing ID, a storage type indicating the storage in which the data identified by the combination of the transaction ID and the processing ID is stored, a storage name indicating the name of the file when the data identified by the combination of the transaction ID and the processing ID is stored in the storage, a last update date and time indicating the date and time when the information was stored in the data management table 90, a processing name indicating the name of the processing for the data identified by the combination of the transaction ID and the processing ID, and a processing status indicating the status of the processing indicated by the processing name.

[0125] There are two types of storage: primary storage and secondary storage.

[0126] The process name includes Masking, which indicates a masking process, and Transfer, which indicates a data transfer process. In S381, Masking is written.

[0127] The processing status includes "Processing," which indicates that the processing indicated by the processing name is being executed, "Completed," which indicates that the processing indicated by the processing name has ended normally, and "Error," which indicates that the processing indicated by the processing name has failed. "Processing" is written in S381.

[0128] As shown in FIG. 12, after the process of S381, the masking processor 72 starts the masking process on the target data (S382).

[0129] Next, the masking processor 72 determines whether or not a failure in the masking process started in S382, that is, a failure in data conversion, has been detected (S383).

[0130] If the masking processor 72 determines in S383 that a failure in the masking process has not been detected, it determines whether or not the masking process started in S382 has ended (S384).

[0131] If the masking processor 72 determines in S384 that the masking process has not been completed, it executes the process of S383.

[0132] When the masking processing unit 72 determines in S383 that a failure in the masking processing has been detected, it notifies the pipeline orchestrator 61 of an event indicating the failure of the masking processing (S385). This event includes the transaction ID and the processing ID of the target data.

[0133] Next, the masking processor 72 writes information indicating that the masking process has failed for the data that is the target of the current masking process to the data management table 90 (S386), and ends the operation shown in Fig. 12. The "processing name" and "processing status" in the information written in S386 are "Masking" and "Error", respectively.

[0134] When the masking processor 72 determines in S384 that the masking process has ended, it writes information indicating that the masking process has ended normally for the data that is the subject of this masking process to the data management table 90 (S387), and ends the operation shown in Fig. 12. The "processing name" and "processing status" in the information written in S387 are "Masking" and "Completed", respectively.

[0135] The above describes the operation of the masking processing unit 72 in the masking processing of S347, but the same applies to the operation of the data transfer processing unit 73 in the data transfer processing of S354 and the operation of the big data analysis unit 44 in the final conversion processing of S359.

[0136] Next, the operation of the data linkage system 30 when the masking processor 72 fails to process data will be described.

[0137] FIG. 14 is a sequence diagram of the operation of the data linkage system 30 when the masking processor 72 fails to process data.

[0138] When the masking process fails during the execution of the operation shown in Fig. 10, the masking processing unit 72 notifies the pipeline orchestrator 61 of an event indicating the failure of the masking process (S401), as shown in Fig. 14. The notification of S401 corresponds to the notification of S385 (see Fig. 12).

[0139] When the trigger processing unit 81 of the pipeline orchestrator 61 receives the event notified by the masking processing unit 72 in S401, it analyzes the contents of this event, calls a scenario corresponding to this event, i.e., a scenario for re-executing the masking process, from the action description unit 82 (S402), and notifies the action processing unit 83 of the scenario called in S402 (S403). Therefore, the action processing unit 83 instructs the masking processing unit 72 of the pipeline 70 to execute a process based on the scenario notified in S403, i.e., to execute a masking process on the data stored in the primary storage 71 in S341 (S404). Here, the action processing unit 83 identifies the information with the most recent last update date and time among the information contained in the data management table 90 for the data identified by the combination of the transaction ID and processing ID included in the event notified by the masking processing unit 72 in S401, and if the processing status of the identified information is not Completed, i.e., Processing or Error, instructs the masking processing unit 72 of the pipeline 70 to perform masking processing on this data.

[0140] After the process of S404, the process of S346 and subsequent processes shown in FIG. 10 are executed.

[0141] The above describes the operation of the data linkage system 30 when the masking processing unit 72 fails to process data. However, the data linkage system 30 can also use a similar mechanism to re-execute processing when a component in the data accumulation system 40 other than the masking processing unit 72, such as the data transfer processing unit 73 or the big data analysis unit 44, fails to process data, or when a component in the data linkage system 30 other than the data accumulation system 40, such as a data collection system, fails to process data.

[0142] Note that data stored in the primary storage 71 is not frequently used. Therefore, the primary storage 71 may move data that has been stored in the primary storage 71 for a certain period of time to a certain storage area outside the pipeline. When the primary storage 71 moves data to a certain storage area outside the pipeline, the primary storage 71 may compress the data before moving it. After moving the data to the certain storage area outside the pipeline, the primary storage 71 notifies the pipeline orchestrator 61 of a combination of the transaction ID and the process ID of the data moved to the certain storage area outside the pipeline. When the pipeline orchestrator 61 instructs the masking processing unit 72 of the pipeline 70 to execute a masking process on the data moved to the certain storage area outside the pipeline, the pipeline orchestrator 61 instructs the primary storage 71 to restore the data to the primary storage 71. Therefore, the primary storage 71 acquires data specified by the pipeline orchestrator 61 from the certain storage area outside the pipeline and stores it in the primary storage 71 itself. Here, if the data specified by the pipeline orchestrator 61 is compressed, the primary storage 71 decompresses the data before storing it in the primary storage 71 itself.

[0143] Although the data stored in the primary storage 71 has been described above, the same applies to the data stored in the secondary storage 74. That is, the secondary storage 74 may move data that has been stored in the secondary storage 74 for a specific period of time to a specific storage area outside the pipeline, and restore the data moved to the specific storage area outside the pipeline to the secondary storage 74 itself in response to an instruction from the pipeline orchestrator 61. Furthermore, when the secondary storage 74 moves data to a specific storage area outside the pipeline, the secondary storage 74 may compress the data before moving it.

[0144] Next, the operation of the data linkage system 30 when the application unit 50 requests updating of data of a specific information system stored in the data storage system 40 will be described.

[0145] Figure 15 is a sequence diagram of the operation of the data linkage system 30 when the application unit 50 requests updating of data of a specific information system (hereinafter referred to as the "target information system" in the description of the operation shown in Figure 15) stored in the data storage system 40.

[0146] In addition, cases where the application unit 50 requests updating of the data of the target information system stored in the data storage system 40 include cases where the application service of the application unit 50 requests updating of the data of the target information system stored in the data storage system 40 in response to instructions from a user of the application service of the application unit 50.

[0147] As shown in FIG. 15, the application unit 50 requests the management API 65 to update the data of the target information system stored in the data storage system 40 (S421).

[0148] When the management API 65 receives the request in S421, it notifies the pipeline orchestrator 61 of an event indicating the received request (S422).

[0149] When the trigger processing unit 81 of the pipeline orchestrator 61 receives the event notified by the management API 65 in S422, it analyzes the contents of this event, calls a scenario corresponding to this event, that is, a scenario for updating the data of the target information system stored in the data storage system 40, from the action description unit 82 (S423), and notifies the action processing unit 83 of the scenario called in S423 (S424). Therefore, the action processing unit 83 executes processing based on the scenario notified in S424. That is, the action processing unit 83 first checks whether the data of the target information system stored in the data storage system 40 is the latest (S425). If the data of the target information system stored in the data storage system 40 is not the latest as a result of the check in S425, the action processing unit 83 instructs the data collection system for the target information system to execute transmission of the data of the target information system (S426).

[0150] Therefore, the data collection system acquires data from the target information system (S427), and passes the data acquired in S427 to the pipeline associated with the data collection system itself (S428).

[0151] After the process of S428, the processes shown in FIG. 10 and FIG. 11 are executed.

[0152] When the pipeline 70 and the big data analysis unit 44 process data in response to a request from the application unit 50 to update the data of the target information system stored in the data storage system 40, it is preferable that the final conversion process by the big data analysis unit 44 is completed early. Therefore, in regard to the process of S354, the data transfer processing unit 73 may directly transfer the data passed from the masking processing unit 72 in S348 to the big data analysis unit 44, rather than transferring the data stored in the secondary storage 74 in S353 to the big data analysis unit 44 via the secondary storage 74.

[0153] The above describes the updating of the data of the target information system stored in the data storage system 40. Here, the data linkage system 30 can also update only specific data of the data of the target information system stored in the data storage system 40. For example, the data linkage system 30 can also update only data of a specific table, such as a device management table, of the data of the target information system stored in the data storage system 40.

[0154] Next, the operation of the data linkage system 30 when changing its own configuration in response to a change in the configuration of a specific information system will be described.

[0155] FIG. 16 is a flowchart of the operation of the data linkage system 30 when changing its own configuration in response to a change in the configuration of a specific information system (hereinafter, referred to as a "target information system" in the explanation of the operation shown in FIG. 16).

[0156] The configuration management gateway 63 executes the operations shown in FIG. 16 at specific timing.

[0157] As shown in FIG. 16, the configuration management gateway 63 connects to the configuration management server of the target information system (S441) and determines whether there has been a change in the configuration of the data to be linked based on information from the configuration management server of the target information system (S442).

[0158] When the configuration management gateway 63 determines in S442 that there is no change in the configuration of the data to be linked, it ends the operation shown in FIG.

[0159] When it is determined in S442 that there is a change in the configuration of the data to be linked, the configuration management server 62 determines whether or not the content of the change in the configuration of the data collection system and the data storage system 40 that should be changed in response to the content of the change in the configuration of the data to be linked has been defined (S443). Here, the configuration management server 62 stores change content correspondence information indicating the correspondence between the content of the change in the configuration of the data to be linked and the content of the change in the configuration of the data collection system and the data storage system 40 that should be changed in response to the content of the change in the configuration of the data to be linked. When the correspondence regarding the content of the change in the configuration of the data to be linked is stored in the change content correspondence information, the configuration management server 62 determines that the content of the change in the configuration of the data collection system and the data storage system 40 that should be changed in response to the content of the change in the configuration of the data to be linked has been defined. On the other hand, when the correspondence regarding the content of the change in the configuration of the data to be linked is not stored in the change content correspondence information, the configuration management server 62 determines that the content of the change in the configuration of the data collection system and the data storage system 40 that should be changed in response to the content of the change in the configuration of the data to be linked has not been defined.

[0160] If the configuration management server 62 determines in S443 that the contents of the configuration changes to be made in the data collection system and data storage system 40 in response to the contents of the configuration changes of the data to be linked have not been defined, it stops the processing of the data collection system and data storage system 40 related to the data to be linked (S444). Next, the configuration management server 62 notifies a predetermined destination, such as the destination of a person involved in the target information system, that the configuration of the data linkage system 30 cannot be changed in response to the configuration changes of the target information system (S445), and ends the operation shown in FIG.

[0161] When the configuration management server 62 determines in S443 that the content of the change in the configuration of the data collection system and the data storage system 40 that should be changed in response to the content of the change in the configuration of the data to be linked is defined, the configuration management server 62 changes the content of the change in the configuration of the data collection system and the data storage system 40 that should be changed in response to the content of the change in the configuration of the data to be linked, with the content of the change defined in the change content correspondence information (S446). Here, the content of the change in the configuration of the data collection system may be, for example, a change in the range of data to be linked, or a change in the frequency of linking. Note that, when changing the configuration of the data collection system, the configuration management server 62 may deploy a new data collection system with a changed configuration. The content of the change in the configuration of the data storage system 40 may be, for example, a change in the processing content of the masking process by the masking processing unit, or a change in the processing content of the final conversion process in the big data analysis unit 44.

[0162] After the processing of S446, the configuration management server 62 ends the operation shown in FIG.

[0163] As described above, when the data linkage system 30 detects (S401) that a process has failed among the various stages of processing of data in the data linkage system 30 itself, it re-executes this process (S404), so that data can be linked even if the process fails during the linkage.

[0164] For example, when a failure such as a communication error occurs with the data source unit 20, the data linkage system 30 automatically and to a minimum extent recovers from the failure, i.e., re-executes the data collection process, so that the overall operating costs of the data linkage system 30 can be reduced even when linking large amounts of data.

[0165] When the application unit 50 requests the pipeline orchestrator 61 to update the data stored in the data storage system 40 (S421-S422), the data linkage system 30 causes the data collection system to execute a process of collecting the data requested to be updated by the application unit 50 (S426), thereby making it possible to easily update the data stored in the data storage system 40.

[0166] In this embodiment, the pipeline includes a masking processor as a data conversion system. However, the pipeline may include at least one data conversion system other than the masking processor instead of or in addition to the masking processor. [Explanation of symbols]

[0167] 21 Information Systems 22 POST connector (data collection system) 23 POST Agent (Data Collection System) 24 GET Agent (Data Collection System) 30 Data Linkage System 40 Data Storage System 42 GET connector (data acquisition system) 43 GET Agent (Data Collection System) 50 Application Section 61 Pipeline Orchestrator (Processing Monitoring System) 100 Production management system (information system) 120 Remote Management System (Information System)

Claims

[Claim 1] a data collection system that collects data held by the information system; a data storage system that stores data held by the plurality of information systems and collected by the data collection system; an application unit that utilizes data stored in the data storage system; Equipped with When data of a transaction for collecting data held by the information system is larger than a specific processing unit, the data collection system divides the data of the transaction into the specific processing units. A data linkage system, a processing monitoring system for monitoring each stage of processing for the specific processing unit data in the data linkage system, the process monitoring system, upon detecting a failed process, causes the process to be re-executed; The data storage system includes: a first data conversion system that performs a first data conversion process on the data collected by the data collection system; a second data conversion system that performs a second data conversion process on the data that has been subjected to the first data conversion process by the first data conversion system, and manages the data that has been subjected to the second data conversion process as data to be used by the application unit; a data transfer processing unit that transfers data on which the first data conversion process has been performed by the first data conversion system to the second data conversion system; a transfer data storage system for storing data for transfer to the second data conversion system; Equipped with the data transfer processing unit transfers data, on which the first data conversion process has been performed by the first data conversion system, to the second data conversion system via the transfer data storage system; the processing monitoring system, when the application unit requests updating of the data stored in the data storage system, causes the data collection system to execute the processing of collecting the data requested to be updated by the application unit; A data linkage system characterized in that the data transfer processing unit transfers data that has been requested to be updated by the application unit and collected by the data collection system, and that has undergone the first data conversion process by the first data conversion system, directly to the second data conversion system without going through the transfer data storage system.

Citation Information

Patent Citations

  • Information processing system, information processing method, information processor, and program

    JP2016006624A

  • Information processing apparatus and information processing system

    JP2019074856A

  • Resource acquisition method and device

    JP2019506785A

  • Synchronized resource planning

    WO2014172564A1