Data processing system and cache update control method

Abstract

To allow update of a cache in a database only under planned constraints to reduce a load due to an access to the database.SOLUTION: In a data processing system 1, a data processing apparatus comprises: an analysis part cache 150 that caches data in a database 210; a data processing development module 130 that is used by a developer 3 and develops analysis processing for data analysis; a data processing module 140 that is used by the developer 3 and performs the data analysis through the execution of the analysis processing; a cache management module 110 that sets update constraints input from a DB administrator in relation to cache update from the database in the analysis part cache; and an update determination module 120 that determines whether the cache update can be performed based on the update constraints. When the cache update is requested, if it is determined that the cache update can be performed, the update determination module 1 executes the requested cache update.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a data processing system and a cache update control method, and is suitable for use in a data processing system and a cache update control method for managing updates to a cache used for database analysis. [Background technology]

[0002] In recent years, data analysis has become increasingly used in manufacturing factories to generate new value by analyzing sensor data, log data, and other data stored in core databases. In this data analysis, data analysts refer to and analyze data from core databases to improve production efficiency and visualize the data.

[0003] In general data analysis implementation, data to be analyzed is selected first, and then a development phase is carried out to streamline the data analysis, followed by an operation phase in which data analysis is carried out by periodically referencing and analyzing the database. During the development phase, various data in the database is examined through trial and error, which generates a load due to frequent access to the database. On the other hand, since the main role of a core database is to store operation log data, it is desirable to reduce inadvertent data access that interferes with the storage process of operation log data.

[0004] In the conventional data analysis development phase, databases were accessed by specialized data analysts under waterfall-type management and monitoring, so access was planned and there was no inadvertent load. However, in recent years, the trend toward democratization of analytics has created an environment in which many non-specialist users can easily handle data analysis, and as a result, unmanaged data analysis has generated a lot of inadvertent load on database access.

[0005] By the way, it is generally known that caching is effective in reducing the load on a database. This method involves temporarily storing data required for data analysis as a cache, and by referencing the cache during analysis, it is possible to avoid accessing the database each time an analysis is performed.

[0006] Regarding updates to cache data, for example, Patent Document 1 discloses a technique for reducing access to a master database. Specifically, Patent Document 1 discloses that "when a transmission request is made for cache data whose update period has elapsed, the state of an update flag associated with the cache data is determined" and that "when the update flag is off, the update flag is turned on and update data corresponding to the cache data whose update period has elapsed is obtained." [Prior art documents] [Patent documents]

[0007] [Patent Document 1] JP 2012-146076 A Summary of the Invention [Problem to be solved by the invention]

[0008] In a core database, there are times when you do not want to put a load on the core database by updating old cache data that should be updated. Also, during the development phase of data analysis, changing the cache to different data during the analysis can lead to incorrect analysis results, so this must be avoided.

[0009] However, in the method described in Patent Document 1, by adding an update flag to cache data, when an old cache is referenced, the database is accessed at that timing, and the cache is updated. In other words, in the method described in Patent Document 1, the cache is updated automatically depending on the passage of time, and the update timing cannot be controlled and the cache cannot be updated spontaneously, which is a problem in that the requirements for cache management in the above-mentioned core database cannot be satisfied.

[0010] The present invention has been made in consideration of the above points, and aims to propose a data processing system and a cache update control method that enable updating of a database cache only under planned constraints, thereby reducing the load caused by accessing the database. [Means for solving the problem]

[0011] In order to solve the above problem, the present invention provides a data processing system that is connected to a database and performs data analysis of the database, the system comprising: an analysis location cache that caches data of the database referenced in a development phase of the data analysis; a data processing development module that is used by a developer of the data analysis and develops an analytical process for the data analysis; a data processing module that is used by the developer and performs data analysis by executing the analytical process developed by the data processing development module; a cache management module that sets an update constraint input by an administrator of the database regarding cache updates from the database in the analysis location cache; and an update determination module that determines whether or not to allow cache updates of the analysis location cache based on the update constraints set by the cache management module, The data of the database has metadata including at least one of a schema name, a table name, and a column name, and column data representing the contents of a table, and the update determination module makes the determination when the update constraint is input by the administrator and when the database is specified by the developer, and if the determination when the update constraint is input is that the cache update is permitted, obtains the metadata from the database and stores the obtained metadata in the analysis location cache, thereby executing the cache update, and if the determination when the database is specified is that the cache update is permitted, transmits a determination result to the data processing development module that the update is permitted, and when the cache update is requested from the data processing development module in response to the transmission of the determination result, obtains at least one of the metadata and the column data from the database, In response to the developer's operation The cache update is performed by acquiring the analysis location cache and storing the acquired data in the analysis location cache.

[0012] In addition, in order to solve the above problem, the present invention provides a cache update control method by a data processing system that performs data analysis of a database, the data processing system including: an analysis location cache that caches data of the database referenced in a development phase of the data analysis; a data processing development module that is used by a developer of the data analysis and develops an analytical process for the data analysis; a data processing module that is used by the developer and performs data analysis by executing the analytical process developed by the data processing development module; a cache management module that sets an update constraint input by an administrator of the database regarding cache updates from the database in the analysis location cache; and a cache update control module that updates the analysis location cache based on the update constraint set by the cache management module. and an update determination module which determines whether or not a cache update is possible, wherein the database data includes metadata including at least one of a schema name, a table name, and a column name, and column data which indicates the contents of a table, wherein the determination is made by the update determination module when the update constraint is input by the administrator, and if the determination is that the cache update is possible, the metadata is obtained from the database and the obtained metadata is stored in the analysis location cache, thereby executing the cache update, and wherein the determination is made by the update determination module when the database is specified by the developer, and if the determination is that the cache update is possible, the metadata and at least one of the column data are obtained from the database when a cache update is requested by the data processing development module, In response to the developer's operation The cache update is performed by acquiring the analysis location cache and storing the acquired data in the analysis location cache. Effect of the Invention

[0013] According to the present invention, the database cache can be updated only under planned constraints, thereby reducing the load caused by accessing the database. [Brief description of the drawings]

[0014] [Figure 1] 1 is a block diagram showing an example of the configuration of a data processing system 1 according to a first embodiment of the present invention. [Diagram 2] FIG. 2 is a diagram for explaining the relationship between the data processing system 1 and a user. [Diagram 3] 13 is a sequence diagram illustrating an example of a procedure for a cache update constraint registration process. FIG. [Figure 4] FIG. 11 is a sequence diagram illustrating an example of a procedure for an arbitrary cache update process. [Diagram 5] 13 is a flowchart showing an example of a processing procedure of a process performed by the DB information input function 112. [Figure 6] FIG. 2 is a diagram showing an example of DB information 101. [Figure 7] 13 is a flowchart showing an example of a processing procedure of a process performed by an update constraint input function 111. [Figure 8] FIG. 2 is a diagram illustrating an example of a cache update constraint 102. [Figure 9] 13 is a flowchart showing an example of a processing procedure of a process performed by an update determination module 120. [Figure 10] 13 is an example of a code for performing an update determination. [Figure 11] Here is a code example of a connection query. [Figure 12] FIG. 13 is a diagram showing an example of an update log 163. [Figure 13] 13 is a flowchart showing an example of a processing procedure of a process performed by an analysis flow design function 131. [Figure 14] 13 is a flowchart showing an example of a processing procedure of a process performed by the DB information browsing function 132. [Figure 15] 13 is a flowchart (part 1) illustrating an example of a processing procedure of a process performed by a drawing unit 170. [Figure 16] 13 is a second flowchart illustrating an example of a processing procedure of the drawing unit 170. [Figure 17] FIG. 13 is a diagram showing an example of a registration DB list screen. [Figure 18]FIG. 13 is a diagram showing an example of a DB information input screen. [Figure 19] FIG. 13 is a diagram illustrating an example of a cache update constraint input screen. [Figure 20] FIG. 13 is a diagram illustrating an example of an analysis flow design screen. [Figure 21] FIG. 13 is a diagram showing an example of a DB designation screen. [Figure 22] FIG. 13 is a diagram showing an example of a DB data display screen. [Figure 23] 11 is a block diagram showing an example of the configuration of a data processing system 4 according to a second embodiment of the present invention. FIG. [Figure 24] FIG. 13 is a diagram illustrating an example of a cache update constraint input screen. [Diagram 25] FIG. 2 is a diagram illustrating an example of a cache update constraint 102. [Figure 26] FIG. 11 is a sequence diagram illustrating an example of a procedure for a timely cache update process. [Figure 27] 13 is a flowchart showing an example of a processing procedure of a process performed by the DB information browsing function 132. [Figure 28] 13 is a flowchart showing an example of a processing procedure of a process performed by an update determination module 120. [Figure 29] 13 is a flowchart showing an example of a processing procedure of a management drawing unit 510. [Diagram 30] 13 is a flowchart showing an example of a processing procedure of a process performed by an analysis drawing unit 620. [Diagram 31] 1 is a block diagram showing an example of the hardware configuration of each device (computer device 10) constituting a data processing system 1, 4 according to a first or second embodiment. [Diagram 32] FIG. 11 is an image diagram for explaining a cache data replacement process. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0015] Hereinafter, an embodiment of the present invention will be described in detail with reference to the drawings.

[0016] (1) First embodiment (1-1) Configuration FIG. 1 is a block diagram showing an example of the configuration of a data processing system 1 according to the first embodiment of the present invention.

[0017] 1, the data processing system 1 according to this embodiment is configured by connecting one or more data processing devices 100 used by a data analysis developer (developer 3 in FIG. 2 described later) to a database device 200 managed by a database administrator (DB administrator 2 in FIG. 2 described later). Note that the detailed relationship between the data processing system 1, the DB administrator 2, and the developer 3 will be described later with reference to FIG. 2.

[0018] 1, DB information 101 is information of the database 210 that the DB administrator 2 assigns to the data processing device 100, and cache update constraints 102 are constraints imposed at the time of cache update described by the DB administrator 2. The DB information 101 and the cache update constraints 102 are input to the data processing device 100 by the DB administrator 2.

[0019] The data processing device 100 includes a cache management module 110 , an update determination module 120 , a data processing development module 130 , a data processing module 140 , an analysis location cache 150 , a data storage unit 160 , and a rendering unit 170 .

[0020] The cache management module 110 has, as functions for accepting information input from the DB administrator 2 to the data processing device 100, an update constraint input function 111 that accepts cache update constraints 102, which are constraints when updating the cache, and a DB information input function 112 that accepts DB information 101 for connecting to the database 210.

[0021] The update determination module 120 has a function of referring to cache update constraint information 162 that summarizes the cache update constraints 102 received by the cache management module 110, and determining whether an update is possible when an update request is made to the analysis location cache 150.

[0022] The data processing development module 130 has an analysis flow design function 131 and a DB information viewing function 132. The analysis flow design function 131 has a database reference method for performing data analysis, methods for combining, separating and analyzing data, a method for outputting analysis results, etc., and the developer 3 can perform data analysis by using these methods. The DB information viewing function 132 has a function of displaying information about the database 210 (DB information 101) and providing a field for entering a database query when a method including a connection to the database 210 is called by the analysis flow design function 131.

[0023] The data processing module 140 has a function of actually executing the analysis processing developed by the developer 3 in the data processing development module 130 .

[0024] The analysis part cache 150 is a cache that stores data required for data analysis. As an example, in a relational database, metadata 151 such as a schema name, a table name, and a column name, and column data 152 of the contents of a table are stored.

[0025] The data storage unit 160 stores registered DB information 161, cache update constraint information 162, and an update log 163. The registered DB information 161 is information that summarizes DB information 101 accepted by the DB information input function 112 of the cache management module 110, and the cache update constraint information 162 is information that summarizes cache update constraints 102 accepted by the update constraint input function 111 of the cache management module 110. The update log 163 is information that saves a log of the cache update process.

[0026] The drawing unit 170 accepts a user operation and displays functions and information of the cache management module 110 and the data processing development module 130. In addition, an update button 171, which will be described later, can be displayed on the display screen of the drawing unit 170. The update button 171 is pressed by the user as necessary, whereby the target cache of the analysis location cache 150 is updated.

[0027] The database device 200 includes a database 210 and a database management system 220. In the database 210, factory operation data such as sensor data (sensor data 201 in FIG. 2, which will be described later) and log data (log data 202 in FIG. 2, which will be described later) are stored as metadata 211 and column data 212. Access to the database 210 and the like are managed by the database management system 220.

[0028] In this embodiment, it will be explained that, using the data processing system 1 configured as above, the DB administrator 2 imposes a cache update constraint on the analysis location cache 150 of the data processing device 100, and the developer 3 operates the update button 171 under the above constraint, thereby allowing the analysis location cache 150 to be updated arbitrarily under the above constraint. However, the present invention is not limited to such an embodiment, and the specific configuration may be changed without departing from the concept or spirit of the present invention.

[0029] In this embodiment, the data processing system 1 shown in FIG. 1 can assume that users include, for example, an administrator (or a management department) of a core database and a developer of data analysis.

[0030] Fig. 2 is a diagram for explaining the relationship between the data processing system 1 and users. Assumed users of the data processing system 1 according to this embodiment are a database administrator (DB administrator 2) and a data analysis developer (developer 3). Note that Fig. 2 omits the illustration of part of the configuration of the data processing system 1 shown in Fig. 1.

[0031] As shown in FIG. 2, the DB administrator 2 manages a database 210 in a database device 200 that stores sensor data 201 and log data 202. The developer 3 operates the data processing development module 130 in the data processing device 100 through the drawing unit 170 to design a data processing flow. The processing flow designed by the developer 3 is actually processed by the data processing module 140. In addition, between the data processing device 100 and the database device 200, updates to the analysis location cache 150 and access to refer to data occur. Here, in order for the developer 3 to access the database 210 on the data processing device 100, it is necessary for the DB administrator 2 to grant access authority to the data processing device 100 and provide DB information 101 (specifically, host, port number, user, password, etc.), and the developer 3 cannot access and refer to the database 210 without the permission of the DB administrator 2.

[0032] (1-2) Overall processing In the following, a cache update constraint registration process and an optional cache update process will be described as the overall process executed by the data processing system 1.

[0033] Fig. 3 is a sequence diagram showing an example of the procedure of a cache update constraint registration process. The series of processes shown in Fig. 3 are executed in the data processing system 1 when the DB administrator 2 assigns DB information 101 of the database 210 and cache update constraints 102 to the data processing device 100 so that the developer 3 can access the database 210 on the data processing device 100. It should be noted that the series of processes shown in Fig. 3 are executed before the developer 3 performs data analysis.

[0034] In addition, in the sequence diagram of FIG. 3, a step number is assigned to each of a series of processes that are successively executed by the same functional unit or module, and this is the same in other sequence diagrams described later.

[0035] Step S101 shown in FIG. 3 is a process by the drawing unit 170. In step S101, when the DB administrator 2 accesses the cache management module 110, the drawing unit 170 first notifies the DB information input function 112 of the access. Then, the drawing unit 170 receives data on the registered database (at least the registered DB information 161 and the cache update constraint information 162) from the DB information input function 112 and displays the data. After that, when the DB administrator 2 inputs information on a database to be newly registered (newly registered DB information 101), the drawing unit 170 passes the information to the DB information input function 112. After that, the drawing unit 170 receives from the DB information input function 112 the registered DB information 161 to which the previously passed newly registered DB information 101 has been added, and displays the data. Next, when the DB administrator 2 inputs to register the cache update constraint 102, the drawing unit 170 passes the cache update constraint 102 to the update constraint input function 111. Thereafter, the updated cache update constraint information 162 is received from the update constraint input function 111 and displayed.

[0036] Step S102 shown in FIG. 3 is a process performed by the DB information input function 112. In step S102, when the DB information input function 112 receives an access notification from the drawing unit 170, it acquires information on the current database 210 from the registered DB information 161 in the data storage unit 160, acquires a currently registered cache update constraint from the cache update constraint information 162 in the data storage unit 160, acquires a cache update log from the update log 163 in the data storage unit 160, and transmits information on the registered database 210 to the drawing unit 170 based on the acquired information. In addition, when the DB information input function 112 receives DB information 101 to be newly registered from the drawing unit 170, the DB information input function 112 passes the DB information 101 to the update constraint input function 111 and stores it in the registered DB information 161. Furthermore, the DB information input function 112 transmits updated information on the registered database 210 to the drawing unit 170.

[0037] Step S103 shown in FIG. 3 is a process performed by the update constraint input function 111. In step S103, when the update constraint input function 111 receives DB information 101 to be newly registered from the DB information input function 112, the update constraint input function 111 waits for the transmission of the cache update constraint 102 from the drawing unit 170. Then, when the cache update constraint 102 is input, the update constraint input function 111 passes the DB information 101, the cache update constraint 102, and a registration time flag to the update determination module 120. The registration time flag is a flag indicating that the transmission data is transmitted at the time of registration. Then, the update constraint input function 111 saves the input cache update constraint 102 in the cache update constraint information 162, and transmits the updated cache update constraint information 162 to the drawing unit 170.

[0038] 3 is a process performed by the update determination module 120. In step S104, the update determination module 120 determines whether or not the database 210 is accessible and the cache can be updated based on the DB information 101, the cache update constraint 102, and the registration flag received from the update constraint input function 111. If the registration flag is included in the received data (the update flag owned by the update determination module 120 is ON) and it is determined that the cache can be updated, the update determination module 120 transmits a metadata acquisition query to the database 210 and stores the metadata 211 received from the database 210 in the cache of the analysis location cache 150 (metadata 151).

[0039] As a result of the above-described processing steps in FIG. 3 being performed, the DB information 101 and cache update constraints 102 for connecting to the database 210 input by the DB administrator 2 are assigned to the data processing device 100, and thereafter, the developer 3 becomes able to access the database 210 on the data processing device 100.

[0040] 4 is a sequence diagram showing an example of the procedure of an arbitrary cache update process. The series of processes shown in FIG. 4 is executed by the data processing system 1 when the developer 3 updates the analysis point cache 150 while using the analysis flow design function 131.

[0041] Step S201 shown in FIG. 4 is a process by the drawing unit 170. In step S201, when the developer 3 accesses the analysis flow design function 131, the drawing unit 170 notifies the analysis flow design function 131 of the access. The drawing unit 170 receives information on the analysis flow being designed from the analysis flow design function 131 and displays it. When the developer 3 inputs a request to create a flow design, the drawing unit 170 requests the analysis flow design function 131 to create a flow design. When the developer 3 inputs a database designation (input DB designation), the drawing unit 170 transmits the input DB designation to the analysis flow design function 131. When the drawing unit 170 receives an instruction to display a cache update button from the DB information browsing function 132, the drawing unit 170 displays the update button 171 for the developer 3. When the developer 3 presses the update button 171, the drawing unit 170 passes the information that the update button 171 has been pressed to the DB information browsing function 132. Thereafter, the drawing unit 170 receives the cached and updated information of the database 210 from the DB information viewing function 132 and displays it.

[0042] 4 is a process by the analysis flow design function 131, and is started when an access occurs from the drawing unit 170 to the data processing development module 130. At this time, the analysis flow design function 131 acquires connection information of the already registered database 210 from the registered DB information 161, and transmits it to the drawing unit 170. Next, in step S202, when an input DB designation is received from the drawing unit 170, the analysis flow design function 131 transmits the input database to the DB information browsing function 132. When specifying the input DB, the developer 3 specifically writes the host name, database name, schema name, table name, user, and password of the database as the input DB.

[0043] Step S203 shown in FIG. 4 is a process performed by the DB information viewing function 132. In step S203, when the DB information viewing function 132 receives connection information related to the input DB from the analysis flow design function 131, the DB information viewing function 132 transmits the input DB to the cache update constraint information 162 and acquires the cache update constraint set for the target database 210. In addition, the DB information viewing function 132 checks whether or not a cache of the target database 210 exists in the analysis location cache 150 by transmitting the input DB to the analysis location cache 150. Thereafter, the DB information viewing function 132 transmits the input DB and the cache update constraint to the update determination module 120 and acquires a result of whether or not the developer 3 can update the cache. Here, if the cache update is possible, the DB information viewing function 132 notifies the drawing unit 170 to display a cache update button (update button 171). In addition, when the DB information viewing function 132 receives a notification from the drawing unit 170 that the update button 171 has been pressed, the DB information viewing function 132 issues a cache update query to the update determination module 120. Then, when the cache is updated and the updated data of the database 210 is received from the update determination module 120 , the DB information browsing function 132 transmits the received data to the drawing unit 170 .

[0044] Step S204 shown in FIG. 4 is a process by the update determination module 120, and is executed when an input DB is designated by the developer 3. In step S204, when the update determination module 120 receives an input DB and a cache update constraint from the DB information browsing function 132, the update determination module 120 determines whether a cache update is possible for the database 210 indicated by the input DB based on the cache update constraint, and transmits the determination result (update determination result) to the DB information browsing function 132. After that, the update determination module 120 writes the result of the update in the update log 163 regardless of the update determination result. At the time of step 204, the update determination module 120 has not received a registration time flag from the update constraint input function 111 (see step S103 in FIG. 3), and therefore does not acquire metadata 211 from the database 210 (see step S104 in FIG. 3).

[0045] 4 is a process by the update determination module 120, and is executed when the developer 3 presses the update button 171. In step S205, when the update determination module 120 receives a cache update query from the DB information browsing function 132, it issues the query to the database 210 indicated by the input DB, receives the result, and updates the analysis location cache 150. Thereafter, the update determination module 120 writes the update result in the update log 163.

[0046] (1-3) Individual processing In the following, individual processes executed by each function, module, etc. in the overall process by the data processing system 1 described above will be described in detail with reference to FIGS.

[0047] (1-3-1) DB information input function 112 5 is a flowchart showing an example of a processing procedure of the processing by the DB information input function 112. The processing shown in FIG. 5 corresponds to the processing of step S102 in FIG.

[0048] According to FIG. 5, first, the DB information input function 112 receives from the drawing unit 170 a notification that the cache management module 110 has been accessed (step S301).

[0049] Next, the DB information input function 112 acquires the registered DB information 161, the cache update constraint information 162, and the update log 163 that are already registered and stored in the data storage unit 160, and transmits them to the drawing unit 170 (step S302).

[0050] Next, the DB information input function 112 receives the DB information 101 to be newly registered input by the DB administrator 2 from the drawing unit 170 (step S303), and transmits the received DB information 101 to the update constraint input function 111 (step S304).

[0051] Then, the DB information input function 112 stores the DB information 101 received in step S303 in the registered DB information 161 (step S305).

[0052] FIG. 6 is a diagram showing an example of DB information 101. DB information 300 shown in FIG. 6 shows an example of the code of DB information 101 when stored in registered DB information 161, and database information is described in JSON format. The description content of DB information 300 shown in FIG. 6 corresponds to the input content on DB information input screen 360 in FIG. 18 described later. By saving such DB information in registered DB information 161, developer 3 can access the database. Note that the description format of DB information 101 is not limited to JSON format, and may be, for example, a database format or the like.

[0053] Finally, the DB information input function 112 transmits information of the registered database 210 including the newly registered DB information 101 (i.e., the registered DB information 161 after update in step S305) to the drawing unit 170 (step S306), thereby completing the series of processes.

[0054] (1-3-2) Update constraint input function 111 7 is a flowchart showing an example of a processing procedure of the processing by the update constraint input function 111. The processing shown in FIG. 7 corresponds to the processing of step S103 in FIG.

[0055] 7, first, the update constraint input function 111 receives the DB information 101 input by the DB administrator 2 from the DB information input function 112 (step S401). After step S401, the update constraint input function 111 waits until the cache update constraint 102 for the database of the DB information 101 received in step S401 is input on the drawing unit 170 (step S402).

[0056] Then, when the update constraint input function 111 receives the input cache update constraint 102 from the drawing unit 170 (step S403), it transmits the DB information 101 received in step S401 and the cache update constraint 102 received in step S403 to the update determination module 120 (step S404). At this time, the update constraint input function 111 transmits the DB information 101 received in step S401 and the cache update constraint 102 received in step S403 together with a registration time flag.

[0057] Next, the update constraint input function 111 stores the cache update constraint 102 in the cache update constraint information 162 (step S405).

[0058] FIG. 8 is a diagram showing an example of the cache update constraint 102. The cache update constraint 310 shown in FIG. 8 shows an example of the code of the cache update constraint 102 when stored in the cache update constraint information 162, and the cache update constraint is described in JSON format. The description content of the cache update constraint 310 shown in FIG. 8 corresponds to the input content in a cache update constraint input screen 370 shown in FIG. 19 described later. Specifically, the cache update constraint 310 describes that the schema to be constrained is "public" (code 311), and the contents of the constraint are "update is prohibited from 8:00 to 12:00" (code 312), and "update is allowed once an hour from 12:00 to 17:00" (code 313). Note that the description format of the cache update constraint 102 is not limited to the JSON format, and may be, for example, a database format or the like.

[0059] Finally, the update constraint input function 111 transmits the cache update constraint information 162 including the newly input cache update constraint 102 (i.e., the cache update constraint information 162 updated in step S405) to the drawing unit 170 (step S406), thereby completing the series of processes.

[0060] (1-3-3) Update decision module 120 Fig. 9 is a flowchart showing an example of a processing procedure of the processing by the update determination module 120. The processing shown in Fig. 9 corresponds to the processing of step S104 in Fig. 3 and steps S204 and S205 in Fig. 4. The update determination module 120 is in a standby state until some input is made from the update constraint input function 111, and starts the processing of Fig. 9 in response to the input from the update constraint input function 111.

[0061] 9, first, the update determination module 120 judges whether the DB information 101 and the cache update constraint 102 have been input (step S501). If the input has been input (YES in step S501), the process proceeds to step S502, and if another input has been input (NO in step S501), the process proceeds to step S510, which will be described later.

[0062] In step S502, the update determination module 120 determines whether or not an update flag is ON in addition to the DB information 101 and the cache update constraint 102 confirmed in step S501. The update flag is a flag that is ON when a registration time flag is received from the update constraint input function 111. Therefore, the process of step S502 can also be said to be a process of determining whether or not a registration time flag is input from the update constraint input function 111. If the update flag is ON in step S502 (YES in step S502), it is considered to be a process of a situation in which the DB administrator 2 inputs the DB information 101 and the cache update constraint 102, and the process proceeds to step S503. On the other hand, if the update flag is not ON (OFF) in step S502 (NO in step S502), it is considered to be a process of a situation in which an update determination is made for the analysis location cache 150 corresponding to the database 210 when the developer 3 specifies the database 210, and the process proceeds to step S506.

[0063] First, the process when moving from step S502 to step S503 will be described. The process of steps S503 to S505 corresponds to the process of step S104 in FIG.

[0064] In step S503, the update determination module 120 determines whether or not the analysis location cache 150 can be updated based on the input cache update constraint 102. If the analysis location cache 150 can be updated (YES in step S503), the process proceeds to step S504, and if the analysis location cache 150 cannot be updated (NO in step S503), the update determination module 120 ends the process.

[0065] Fig. 10 is an example of code for performing update judgment. In the update judgment module 320 shown in Fig. 10, whether or not an update is possible is judged based on a time constraint. Specifically, code 321 describes a process for checking the current date and time against the constraint, and sending a query to the database if they match. Also, code 322 describes a process for returning "1" if the current date and time are between the constraint dates and times and the constraint schema correctly matches the input database schema.

[0066] When the analysis part cache 150 can be updated, in step S504, the update determination module 120 sends a metadata acquisition query to the database 210. This metadata acquisition process is performed to acquire the database schema and table names that the developer 3 can specify in the analysis flow design function 131. In addition, metadata is frequently referenced in data analysis, and storing it in the cache in advance can reduce the frequency of updates during development.

[0067] Fig. 11 is a code example of a connection query. The connection query 330 shown in Fig. 11 is a query executed in the update determination module 120 to send a metadata acquisition query, and first specifies the destination database (verifies communication with the database), and then in code 331, describes the process of sending the query.

[0068] After step S504 is completed, the update determination module 120 stores the result (metadata 211) received from the database 210 in step S504 as metadata 151 in the analysis part cache 150 (step S505), and ends the process.

[0069] Next, a description will be given of the process when moving from step S502 to step S506. The processes in steps S506 to S509 correspond to the process in step S204 in FIG.

[0070] In step S506, the update determination module 120 determines whether or not the analysis location cache 150 can be updated based on the input cache update constraint 102, similarly to step S503.

[0071] If the analysis location cache 150 can be updated in step S506 (YES in step S506), the update determination module 120 sends a message to the DB information viewing function 132 that the update is possible (step S507), writes the update result in the update log 163 (step S509), and ends the process. On the other hand, if the analysis location cache 150 cannot be updated in step S506 (NO in step S506), the update determination module 120 sends a message to the DB information viewing function 132 that the update is not possible (step S508), writes the update result in the update log 163 (step S509), and ends the process.

[0072] Next, a description will be given of the process when moving from step S501 to step S510. The processes in steps S510 to S513 and S509 correspond to the process in step S205 in FIG.

[0073] In step S510, the update determination module 120 determines whether or not a cache update query has been input from the update constraint input function 111. If the input is a cache update query (YES in step S510), it is assumed that the update button 171 has been pressed by the developer 3, and the process proceeds to step S511 to execute the update of the analysis location cache 150. If the input is something other than a cache update query (NO in step S510), the update determination module 120 ends the process.

[0074] In step S511, the update determination module 120 transmits the input cache update query to the database 210 and receives the result. Then, the update determination module 120 updates the analysis location cache 150 with the result received in step S511 (step S512).

[0075] Thereafter, the update determination module 120 transmits the update result in step S512 (that the cache update has been completed) to the DB information browsing function 132 (step S513), writes the update result in the update log 163 (step S509), and ends the process.

[0076] Fig. 12 is a diagram showing an example of the update log 163. The update log 340 shown in Fig. 12 is a specific example of the update log 163 in which a log is recorded for each update, and includes information on whether the update was successful, the date and time when the update was performed, and the update content. Specifically, for example, in the case of log 341, it is shown that at 14:15 on September 15, 2020, a SELECT query was executed on the schema "public" of the database name "Factory-C" and the cache was updated.

[0077] (1-3-4) Analysis flow design function 131 Fig. 13 is a flowchart showing an example of a processing procedure of the analysis flow design function 131. The processing shown in Fig. 13 corresponds to the processing of step S202 in Fig. 4, and is started when an access from the drawing unit 170 to the data processing development module 130 occurs.

[0078] 13, first, the analysis flow design function 131 receives from the drawing unit 170 a notification that the data processing development module 130 has been accessed (step S601). In response, the analysis flow design function 131 instructs the drawing unit 170 to display the analysis flow design function (step S602).

[0079] Next, when the analysis flow design function 131 receives from the drawing unit 170 that a creation operation has been performed in the flow design function on the drawing unit 170 (step S603), it obtains information on all databases 210 registered by the DB administrator 2 from the registered DB information 161 (step S604) and waits for the next input from the drawing unit 170 (step S605).

[0080] Thereafter, when an input DB is designated on the drawing unit 170 (input DB designation), the analysis flow design function 131 receives database, schema, table information, etc. from the drawing unit 170 (step S606). In this case, the analysis flow design function 131 transmits the information received in step S606 to the DB information browsing function 132 (step S607), and then returns to the standby state of step S605.

[0081] (1-3-5) DB information viewing function 132 Fig. 14 is a flowchart showing an example of a processing procedure of processing by the DB information browsing function 132. The processing shown in Fig. 14 corresponds to the processing of step S203 in Fig. 4. The DB information browsing function 132 has a function of managing the display / non-display of the update button 171 of the drawing unit 170 (more specifically, the schema update button 395 and the table update button 396 in the DB designation screen 390 shown in Fig. 21 described later), and managing the data in the analysis location cache 150 (for example, the metadata 151 and the column data 152).

[0082] 14, the DB information browsing function 132 is normally in a standby state (step S701). Then, when the developer 3 inputs database, schema, and table information on the drawing unit 170, as described in step S607 of FIG. 13, this input information is transferred by the analysis flow design function 131 and received by the DB information browsing function 132 (step S702).

[0083] Next, the DB information viewing function 132 acquires the cache update constraint set for the target database 210 from the cache update constraint information 162 based on the input information received in step S702 (step S703). Also, the DB information viewing function 132 checks whether or not a cache of the target database 210 exists in the analysis location cache 150 based on the input information received in step S702 (step S704).

[0084] Next, the DB information browsing function 132 transmits the input information (DB information) received in step S702 and the cache update constraint acquired in step S703 to the update determination module 120 (step S705). The update determination module 120 that has received these determines whether the analysis location cache 150 can be updated, as described in steps S506 to S508 of FIG. 9, and transmits the result to the DB information browsing function 132.

[0085] Next, the DB information browsing function 132 judges whether the information received from the update judgment module 120 (the update judgment result of the analysis part cache 150) is updateable or not (step S706). If it is updateable (YES in step S706), the process proceeds to step S707, and if it is not updateable (NO in step S706), the process proceeds to step S710 described later.

[0086] In step S707, since the analysis part cache 150 can be updated, the DB information browsing function 132 transmits a notification to the drawing unit 170 to display the update button 171. Note that, with regard to the expression of display / non-display of the update button 171, in this description, a display mode that can accept an operation from the developer 3 is referred to as "display", and a display mode that does not accept an operation from the developer 3 is referred to as "non-display". Therefore, for example, when the table update button 396 shown in FIG. 21 described later is displayed in a form that cannot be operated by the developer 3, this also corresponds to "non-display".

[0087] Thereafter, the DB information browsing function 132 judges whether or not the developer 3 has pressed the update button 171 (pressing the cache update button) from the drawing unit 170 (step S708). If the notification has been received in step S708 (YES in step S708), the process proceeds to step S709, and if the notification has not been received (NO in step S709), the process proceeds to step S710.

[0088] In step S709, the DB information browsing function 132 issues a cache update query to the update determination module 120. Then, the process proceeds to step S710 when a completion notification of the cache update is received from the update determination module 120. When the completion notification is received, the data of the database 210 after the cache update is also received.

[0089] In step S710, the DB information browsing function 132 transmits the data of the database 210 to the drawing unit 170 in order to display the specified table data in the analysis part cache 150. After that, the process returns to step S701 and waits.

[0090] (1-3-6) Drawing section 170 15 and 16 are flowcharts (part 1 and part 2) showing an example of a processing procedure of the processing by the drawing unit 170. The processing shown in Fig. 15 mainly corresponds to the processing of step S101 in Fig. 3, and the processing shown in Fig. 16 corresponds to the processing of step S201 in Fig. 4.

[0091] 15, first, when the drawing unit 170 is accessed by the DB administrator 2 or developer 3, it decides whether to display the cache management module 110 or the data processing development module 130 depending on the access destination (step S801). If the access destination is the cache management module 110, the process proceeds to step S802, and if the access destination is the data processing development module 130, the process proceeds to step S810 in FIG.

[0092] First, the process when the process moves from step S801 to step S802 will be described. As described above, the processes in steps S802 to S809 correspond to the process by the drawing unit 170 in the cache update constraint registration process (step S101 in FIG. 3).

[0093] If the access destination is the cache management module 110, the drawing unit 170 requests the DB information input function 112 to acquire the registered DB information 161 in order to display the registered DB information 161 that has already been registered (step S802).

[0094] The drawing unit 170 then receives data relating to the currently registered database 210 (at least the registered DB information 161 and the cache update constraint information 162) from the DB information input function 112, and displays the received data on the registered DB list screen (step S803).

[0095] Fig. 17 is a diagram showing an example of a registered DB list screen. The registered DB list screen 350 shown in Fig. 17 is a screen that displays data related to each registered database 210, and displays information from an update log 163 in addition to the registered DB information 161 and the cache update constraint information 162. Specifically, the registered DB list screen 350 in Fig. 17 displays, for each registered database, a host 351 indicating a host name, a port 352 indicating a port number, a DB name 353 indicating a database name, a user 354 indicating a user name, a path 355 indicating a password, a constraint 356 indicating a cache update constraint, and a log 357 indicating an update log.

[0096] Next, the drawing unit 170 displays an input screen (DB information input screen, cache update constraint input screen) for the DB administrator 2 to input the DB information 101 or the cache update constraint 102, and waits until the input is made (step S804).

[0097] Fig. 18 is a diagram showing an example of a DB information input screen. The DB information input screen 360 shown in Fig. 18 is a screen for inputting DB information 101 required for connecting to the database 210 to be registered, and specifically has a host name input field 361, a port number input field 362, a database name input field 363, a user name input field 364, and a password input field 365. Note that the types of input fields provided on the DB information input screen are not limited to the example shown in Fig. 18, and it is necessary to enter information as database information that is appropriate depending on the type of database. Then, when an OK button 366 is pressed, the input is confirmed.

[0098] FIG. 19 is a diagram showing an example of a cache update constraint input screen. The cache update constraint input screen 370 shown in FIG. 19 is a screen for inputting the cache update constraint 102 to be imposed on the database 210 to be registered. Specifically, a target schema or table requiring a constraint is input in an input field 371, and detailed constraint content is input in an input field 373. In the case of FIG. 19, the input field 373 is implemented in a form that allows detailed update restrictions by time. Also, when registration of a constraint is not required, the cache can be updated immediately, so it is sufficient to check the check field 372. Then, when an OK button 374 is pressed, the input is confirmed.

[0099] 19, specific contents of the update constraints are that from midnight to 8am, cache updates can be made immediately without restrictions, from 8am to 12pm, cache updates are not permitted, from 12pm to 5pm, cache updates are limited to once per hour, and from 5pm to midnight, only metadata updates are permitted (see input field 373).The format of the input fields provided on the cache update constraint input screen is not limited to that shown in FIG. 19, and in addition to the time constraints described above, restrictions may also be imposed based on update frequency, data size required for updating, etc.

[0100] Next, the drawing unit 170 determines whether the input was confirmed on the DB information input screen or the cache update constraint input screen (whether DB information 101 or cache update constraint 102 was input) (step S805), and if DB information 101 was input, proceeds to step S806, and if cache update constraint 102 was input, proceeds to step S808.

[0101] When the DB information 101 is input, the drawing unit 170 transmits the input DB information 101 to the DB information input function 112 (step S806). The DB information input function 112 performs a process of storing the DB information 101 transmitted in step S806 in the registered DB information 161. The drawing unit 170 then receives the registered DB information 161 reflecting the input DB information 101 from the DB information input function 112 (step S807), and returns to the process of step S803. Note that the information received in step S807 is reflected in the display contents of the registered DB list screen when the process proceeds to step S803.

[0102] On the other hand, when the cache update constraint 102 is input, the drawing unit 170 transmits the input cache update constraint 102 to the update constraint input function 111 (step S808). The update constraint input function 111 performs a process of storing the cache update constraint 102 transmitted in step S808 in the cache update constraint information 162. The drawing unit 170 then receives the cache update constraint information 162 reflecting the input cache update constraint 102 from the update constraint input function 111 (step S809), and returns to the process of step S803 again. Note that the information received in step S809 is reflected in the display contents of the registered DB list screen when the process proceeds to step S803.

[0103] Next, the process when the process moves from step S801 to step S810 will be described. As described above, the processes in steps S810 to S821 correspond to the process by the drawing unit 170 in the optional cache update process (step S201 in FIG. 4).

[0104] If the access destination is the data processing development module 130, the drawing unit 170 goes to the analysis flow design function 131 to receive information on the analysis flow currently being designed by the developer 3 (step S810). Specifically, the drawing unit 170 notifies the analysis flow design function 131 of the access, and receives the analysis flow information in response.

[0105] Then, the drawing unit 170 displays the information received from the analysis flow design function 131 in step S810 on the analysis flow design screen (step S811).

[0106] FIG. 20 is a diagram showing an example of an analysis flow design screen. In the analysis flow design screen 380 shown in FIG. 20, each procedure is displayed as one node, and the screen allows the user to design the order of processing by connecting the nodes with arrows. For example, the processing procedure shown in FIG. 20 is a process in which a join node 382 executes a join process on a database table specified by a table input node 381, and then CSV output is performed to an output location specified by a CSV output node 383. In the analysis flow design screen 380, a button 384 is a button for creating a new analysis flow, and a button 385 is a button for deleting the analysis flow being displayed. Note that the display form of the analysis flow design screen is not limited to the example shown in FIG. 20, and when the analysis flow design screen has a function for designing each data analysis process including its procedure, the analysis flow design function 131 is used.

[0107] Next, the drawing unit 170 displays a screen (DB designation screen) on which the developer 3 can input the database designation (settings), and waits until it receives input from the developer 3 or a notification of the display of a cache update button from the DB information viewing function 132 (step S812).

[0108] FIG. 21 is a diagram showing an example of a DB designation screen. When the developer 3 places a table input node 381 on the analysis flow design screen 380 of FIG. 20, a DB designation screen 390 as shown in FIG. 21 is displayed. On the DB designation screen 390 shown in FIG. 21, a database can be designated (set). Specifically, in a selection field 391, a database to be connected to can be selected, in an input field 392, a query for data acquisition can be designated, in a selection field 393, a target schema can be selected, and in a selection field 394, a target table can be selected. In addition, a schema update button 395 and a table update button 396 are displayed by the update button 171, which is displayed when the cache of the analysis location cache 150 can be updated, by schema and table.

[0109] In the example of Fig. 21, since the schema update button 395 is displayed as operable, the schema of the target database can be updated. On the other hand, the table update button 396 is displayed as inoperable (hidden in a broader sense), so the table of the target database cannot be updated. Then, the designation of the connection DB (selection field 391), schema (selection field 393), and table (selection field 394) on the DB designation screen 390 of Fig. 21 is regarded as a database designation operation in step S813 described later.

[0110] In the standby state of step S812, when an input from the developer 3 or a display notification of a cache update button from the DB information browsing function 132 is received, the drawing unit 170 judges what kind of input it is, and assigns a destination according to the contents (step S813). Specifically, when there is a flow design creation operation (operation of the button 384 on the analysis flow design screen 380) from the developer 3, the process proceeds to step S814, when there is a database designation operation (the above-mentioned operation on the DB designation screen 390), the process proceeds to step S815, and when there is a pressing operation of the cache update button 171 (operation of the schema update button 395 or the table update button 396 on the DB designation screen 390), the process proceeds to step S818. Also, when a display notification of the update button 171 is input from the DB information browsing function 132, the process proceeds to step S816. The flow design creation operation and the database designation operation are input by the user operating a button or an input field on the screen displayed by the drawing unit 170.

[0111] In step S814, the drawing unit 170 notifies (requests) the analysis flow design function 131 to create a flow design, and then returns to step S812.

[0112] In step S815, the drawing section 170 sends the specified database name to the analysis flow design function 131, and then the process returns to step S812.

[0113] When the process proceeds to step S816, the drawing unit 170 receives a display notification of update buttons 171 (schema update button 395, table update button 396) for updateable target caches from the DB information browsing function 132. When the schema update button 395 and the table update button 396 are provided as the update buttons 171, the caches targeted by each update button are the metadata 211 and column data 212 of the database 210, respectively, and the cache update buttons 171 can be displayed for each data.

[0114] Next, in response to the display notification received in step S816, the drawing unit 170 displays the update button 171 on the connection screen, such as the schema update button 395 and the table update button 396 illustrated in FIG. 21 (step S817), and then returns to step S812.

[0115] When the process proceeds to step S818, the drawing unit 170 detects that the developer 3 has pressed the update button 171. Next, the drawing unit 170 notifies the DB information viewing function 132 that the update button 171 has been pressed (step S819). At this time, the notification includes information indicating the target cache. As described in FIG. 4, FIG. 9, and FIG. 13, the DB information viewing function 132 that has received the notification that the update button 171 has been pressed transmits a cache update query to the update determination module 120 to update the target cache of the analysis location cache 150.

[0116] Next, the drawing unit 170 receives the updated data or metadata of the database 210 from the DB information browsing function 132 (step S820). Then, the drawing unit 170 displays the updated database data or metadata on the DB data display screen based on the data received in step S820 (step S821), and then returns to step S812.

[0117] Fig. 22 is a diagram showing an example of a DB data display screen. For example, in the DB data display screen 400 in Fig. 22, databases, schemas, and data in the tables are listed in a tabular format by a table 401. The display of the DB data display screen 400 plays a supporting role in allowing the developer 3 to easily check the internal data of the cache when performing data analysis.

[0118] As described above, the data processing system 1 according to this embodiment has a function that allows the DB administrator 2 to impose a cache update constraint that restricts cache updates on the data processing device 100 on which the developer 3 performs data analysis. Furthermore, the data processing system 1 according to this embodiment has a function that allows the developer 3 to voluntarily execute cache updates in the data processing device 100 during the development phase of data analysis, and when this function is executed, a determination is made as to whether or not the update is possible based on the cache update constraint imposed by the DB administrator 2, and the cache update is executed only if the update is possible (i.e., under the constraint of the cache update constraint).

[0119] According to such a data processing system 1, cache updates are executed only under the constraints planned by the DB administrator 2, and in the development phase of data analysis, the developer 3 can update the cache at any timing under the above constraints. As a result, access to the database 210 due to cache updates occurs only within the scope required by the data analyst (developer 3) and permitted by the database administrator (DB administrator 2), thereby reducing the frequency of access to the database 210 and the load caused by such access.

[0120] Also, from a user's perspective, the data processing system 1 according to this embodiment is implemented with a function that allows the DB administrator 2 to impose cache update constraints on cache updates of the analysis point cache 150 used in the development phase of data analysis, and also with a function (update button 171) that allows the developer 3 to voluntarily execute cache updates. When the developer 3 requests execution of a cache update, the cache update is performed only if the update is possible under an update judgment based on the cache update constraint. In other words, the developer 3 can perform a cache update at any timing as long as the conditions satisfy the cache update constraints.

[0121] (2) Second embodiment In the above-mentioned first embodiment, the configuration in which the data processing system 1 includes one data processing device 100 as shown in Fig. 2 has been mainly described. However, in the development phase of actual data analysis, it is common for multiple developers 3 to simultaneously perform data analysis, and in order to meet such demand, a configuration including multiple data processing devices 100 may be used. In that case, however, an analysis location cache 150 is required for each data processing device 100. However, if each data processing device 100 has an analysis location cache 150, there is a risk that access to the database 210 will increase due to cache updates, and further, problems such as a loss of consistency in the target data of the data analysis due to different cache update timings are anticipated.

[0122] Therefore, in a data processing system 4 according to a second embodiment described below, the cache management part (mainly the analysis location cache 150 and the cache management module 110) in the data processing system 1 according to the first embodiment is separated from the data analysis part (mainly the data processing development module 130 and the data processing module 140) and implemented as an independent cache DB management device 500. In the data processing system 4 configured in this way, it is possible to unify the analysis location cache 150 into one on the cache DB management device 500 side, without having it in the data analysis side device (data analysis device 600).

[0123] In the description of the second embodiment, the same reference numerals will be used to designate configurations, processes, and the like common to the first embodiment, and description thereof will be omitted.

[0124] (2-1) Configuration FIG. 23 is a block diagram showing a configuration example of a data processing system 4 according to the second embodiment of the present invention. As shown in FIG. 23, the data processing system 4 is configured to include a cache DB management device 500 and a data analysis device 600 instead of the data processing device 100 in the data processing system 1 (see FIG. 1 and FIG. 2) according to the first embodiment. The data analysis devices 600 are a plurality of devices on the data analysis side used by a data analysis developer 3, and are connected to the cache DB management device 500. The cache DB management device 500 is a cache management device managed by a DB administrator 2 that provides a cache (analysis location cache 150) for a plurality of data analysis devices 600 with one device, and is connected to each data analysis device 600 and a database 210. The database 210 suitable for the second embodiment is, for example, a core database.

[0125] 23, configurations that are not important when describing the second embodiment are omitted, but the data processing system 4 may have the configuration of the data processing system 1 described in the first embodiment as appropriate. Specifically, for example, the data processing system 4 may be considered to have an update constraint input function 111 and a DB information input function 112 in a cache management module 110, and the database 210 may be considered to be located in a database device 200, etc.

[0126] The cache DB management device 500 includes an analysis location cache 150, an update determination module 120, registered DB information 161, cache update constraint information 162, an update log 163, a cache management module 110, request count information 501, and a management drawing unit 510. Of these, new configurations not included in the data processing device 100 of the data processing system 1 according to the first embodiment are the request count information 501 and the management drawing unit 510.

[0127] Of the functions of the drawing unit 170 in the first embodiment, the management drawing unit 510 performs only drawing related to the cache DB management device 500 (drawing on the cache management side).

[0128] The request count information 501 is data that records the number of times that a cache (analysis location cache 150) update has been requested from all data analysis devices 600 for each piece of data held in the database 210. Details will be described later in the explanation of Fig. 26 and Fig. 28, but in this embodiment, the action of each developer 3 specifying the database 210 on the data analysis device 600 is regarded as a data update request, and when a certain number of data update requests restricted by the DB administrator 2 have been made (when the number of requests held in the request count information 501 exceeds the restriction), a cache update process is executed.

[0129] In this embodiment, the DB administrator 2 can register DB information 101 (not shown in FIG. 23) and cache update constraints 102 in the cache DB management device 500. As described above, the cache DB management device 500 is connected to the database 210.

[0130] The data analysis device 600 includes a data processing development module 130 , a data processing module 610 , and an analysis and drawing unit 620 .

[0131] The data processing module 610 has an analysis processing execution function 611 that executes processing for data analysis using the analysis flow designed by the analysis flow design function 131 of the data processing development module 130 .

[0132] Of the functions of the drawing unit 170 in the first embodiment, the analysis drawing unit 620 performs only drawing related to the data analysis apparatus 600 (drawing on the data analysis side).

[0133] In the data processing system 4 configured as above, multiple developers 3 can perform data analysis using different data analysis devices 600. At this time, the analysis location cache referred to by all the data analysis devices 600 is the analysis location cache 150 of the cache DB management device 500, and the update decision of the cache depends on the cache update constraint information 162.

[0134] (2-2) Overall processing The overall processing executed by the data processing system 4 according to the second embodiment will be described with a focus on the differences from the overall processing in the first embodiment. In the second embodiment, cache update constraint processing and timely cache update processing can be executed as the overall processing.

[0135] The cache update constraint registration process in the second embodiment is a process executed by the data processing system 4 when the DB administrator 2 assigns the DB information 101 and the cache update constraint 102 of the database 210 to the cache DB management device 500 so that the developer 3 can access the database 210 on the data analysis device 600. The processing procedure of the cache update constraint registration process in the second embodiment may be considered to be similar to the processing shown in FIG. 3 in the first embodiment, except that the drawing unit 170 is replaced with the management drawing unit 510. However, in the cache update constraint registration process in the second embodiment, a constraint on the number of requests can be added to the cache update constraint 102.

[0136] Fig. 24 is a diagram showing an example of a cache update constraint input screen. The cache update constraint input screen 700 shown in Fig. 24 is a screen for inputting the cache update constraint 102 to be imposed on the database 210 to be registered in the second embodiment, and compared with the cache update constraint input screen 370 in the first embodiment illustrated in Fig. 19, an input field 701 for limiting the number of requests is added. The DB administrator 2 can input a desired specified number of times in the input field 701 and press the OK button 374 to specify how many times a data update request from the developer 3 (in other words, a cache update request from the data analysis device 600) is made before the cache update process is executed.

[0137] FIG. 25 is a diagram showing an example of the cache update constraint 102. The cache update constraint 710 shown in FIG. 25 shows an example of the code of the cache update constraint 102 when stored in the cache update constraint information 162 in the second embodiment, and similarly to the cache update constraint 310 in the first embodiment exemplified in FIG. 8, the cache update constraint is described in JSON format. When checking the specific code of the cache update constraint 710, the codes 311, 312, and 313 are similar to the cache update constraint 310 in FIG. 8, and therefore the description will be omitted. Furthermore, in the cache update constraint 710 in FIG. 25, the code 711 relating to the number of update requests describes, as a constraint, that the cache update process is executed when there are "10" update requests.

[0138] The timely cache update process in the second embodiment is a process executed by the data processing system 4 when the developer 3 specifies an input database (database 210) while using the analysis flow design function 131 on the data analysis device 600, and a cache update of the analysis location cache 150 is requested in response to a data update request based on the specification of the input DB.

[0139] Fig. 26 is a sequence diagram showing an example of the procedure of the timely cache update process. As described above, in the second embodiment, the number of update requests is determined by the cache update constraint 102, and therefore, unlike the optional cache update process (see Fig. 4) in the first embodiment, in the timely cache update process shown in Fig. 26, the cache update of the analysis location cache 150 can be executed only under the constraints of the cache update constraints including the number of update requests.

[0140] 26 differs from Fig. 4 in that the drawing unit 170 is replaced with the analysis drawing unit 620 as the processing subject, and request count information 501 is added as a reference destination. Also, in Fig. 26, as the processing by each processing subject, the analysis drawing unit 620 executes the processing of step S901, the analysis flow design function 131 executes the processing of step S902, the DB information browsing function 132 executes the processing of step S903, and the update determination module 120 executes the processing of step S904. However, the processing of step S902 is the same as the processing of step S202 in Fig. 4, and a detailed description thereof will be omitted.

[0141] Step S901 shown in Fig. 26 is processing by the analysis drawing unit 620. A difference from the processing of step S201 by the drawing unit 170 in Fig. 4 is that in step S901, the analysis drawing unit 620 does not receive information on update button display from the DB information browsing function 132 and does not display the update button. Therefore, the analysis drawing unit 620 does not detect that the developer 3 has pressed the update button 171, and does not pass the cache update target to the DB information browsing function 132. The other processing is similar to the processing of step S201 in Fig. 4.

[0142] Step S902 shown in Fig. 26 is a process performed by the analysis flow design function 131. The process of step S902 is similar to the process of step S202 in Fig. 4, and therefore a description thereof will be omitted.

[0143] Step S903 shown in FIG. 26 is a process performed by the DB information viewing function 132. In step S903, when the DB information viewing function 132 receives connection information related to the input DB from the analysis flow design function 131, it transmits the input DB to the cache update constraint information 162 and acquires the cache update constraint set for the target database 210. The acquired cache update constraint includes the number of update requests. Thereafter, the DB information viewing function 132 checks whether or not a cache of the target database 210 exists in the analysis location cache 150 by transmitting the input DB to the analysis location cache 150, and transmits the DB information and cache update constraint of the input DB to the update determination module 120. Thereafter, when the cache update is performed and the data of the database 210 after the update is received from the update determination module 120, the DB information viewing function 132 transmits the received data to the analysis drawing unit 620.

[0144] Step S904 shown in FIG. 26 is a process performed by the update determination module 120. In step S904, when the update determination module 120 receives database information (input DB and cache update constraint) from the DB information browsing function 132, it determines whether a cache update is possible for the database 210 indicated by the input DB based on the cache update constraint, writes the determination result (update determination result) to the update log 163, and increases the current request count held in the request count information 501 by "1". After that, the update determination module 120 obtains the increased request count from the request count information 501, and if the update is possible based on the cache update constraint and the request count is equal to or greater than the specified request count limit, it obtains the target database 210 and updates the analysis location cache 150. Then, the update determination module 120 writes these update results to the update log 163, and transmits the updated cache data to the DB information browsing function 132.

[0145] (2-3) Individual Processing In the following, individual processes executed by each function, module, etc. in the overall process in the second embodiment will be described in detail, focusing on the processes different from those in the first embodiment.

[0146] Fig. 27 is a flowchart showing an example of a processing procedure of processing by the DB information browsing function 132. The processing shown in Fig. 27 corresponds to the processing of step S903 in Fig. 26. In comparison with the processing by the DB information browsing function 132 in the first embodiment (see Fig. 14), the processing of steps S1001 to S1005 in Fig. 27 is similar to the processing of steps S701 to S705 shown in Fig. 14, respectively. Moreover, in Fig. 27, there is no processing equivalent to the processing of steps S706 to S709 in Fig. 14, and after step S1005, the processing of step S1006 (corresponding to step S710 in Fig. 14) is performed.

[0147] FIG. 28 is a flowchart showing an example of a processing procedure of the process by the update determination module 120. The process shown in FIG. 28 corresponds to the process of step S104 in FIG. 3 and step S904 in FIG. 26. In comparison with the process by the update determination module 120 in the first embodiment (see FIG. 9), in FIG. 28, if YES is determined in step S1101 (similar to step S501 in FIG. 9), the processes up to the subsequent steps S1102 to S1109 are similar to the processes of steps S502 to S509 in FIG. 9, and the processes of steps S1110 to S1115 are newly added thereafter. Also, in FIG. 28, if NO is determined in step S1101 (similar to step S501 in FIG. 9), the processes after step S510 in FIG. 9 for determining the cache update query are not executed, and the process is terminated as it is.

[0148] The process of steps S1110 to S1115 newly added in Fig. 28 will be described. First, the update determination module 120 adds "1" to the request count information 501 held in the cache DB management device 500 (step S1110). Next, the update determination module 120 acquires the current request count held in the cache DB management device 500 (i.e., the request count after the addition in step S1110) (step S1111).

[0149] Then, the update judgment module 120 compares the current request count acquired in step S1111 with the request count limit stipulated in the cache update constraint imposed on the target database, and judges whether the current request count is equal to or greater than the request count limit (step S1112). If the current request count is equal to or greater than the request count limit in step S1111 (YES in step S1111), the process proceeds to step S1113, and if the current request count is less than the request count limit (NO in step S1111), the process ends.

[0150] When the process proceeds to step S1113, the update determination module 120 transmits a cache update query to the database 210 and receives the result. Then, the update determination module 120 updates the analysis portion cache 150 with the result received in step S1113 (step S1114).

[0151] Thereafter, the update determination module 120 transmits the update result in step S1114 (that the cache update has been completed) to the DB information browsing function 132 (step S1115), and ends the process.

[0152] Fig. 29 is a flowchart showing an example of a processing procedure of the management drawing unit 510. The processing shown in Fig. 29 is similar to the processing by the drawing unit 170 when the cache management module 110 is accessed in the first embodiment, and specifically corresponds to the processing of step S101 in Fig. 3 (Fig. 15 as a detailed individual processing).

[0153] 29, it is determined in step S1201 whether the access destination from the user is the cache management module 110 or the data processing development module 130, but since the user accessing the management drawing unit 510 of the cache DB management device 500 is the DB administrator 2, the access destination is the cache management module 110, and the management drawing unit 510 executes the processes from step S1202 onwards. Specifically, the processes of steps S1202 to S1209 respectively correspond to the processes of steps S802 to S809 in FIG.

[0154] Fig. 30 is a flowchart showing an example of a processing procedure of the analysis drawing unit 620. The processing shown in Fig. 30 corresponds to the processing by the drawing unit 170 when the data processing development module 130 is accessed in the first embodiment, and specifically corresponds to a part of the processing of step S201 in Fig. 4 (Fig. 16 as a detailed individual processing).

[0155] 30, in step S1301, it is determined whether the access destination from the user is the cache management module 110 or the data processing development module 130. However, since the user who accesses the analysis drawing unit 620 of the data analysis device 600 is basically the developer 3, the access destination is the data processing development module 130, and the analysis drawing unit 620 executes the processing from step S1302 onwards. Specifically, the processing from step 1302 to step S1307 corresponds to the processing from step S810 to step S815 in FIG. 16, respectively. However, since the cache update button is not displayed on the analysis drawing unit 620 of each data analysis device 600 in the second embodiment, the analysis drawing unit 620 does not execute the processing corresponding to steps S816 to S821 in FIG. 16, and performs only the drawing processing related to the data processing development module 130.

[0156] By performing the above-mentioned processing, even if the data processing system 4 includes multiple data analysis devices 600 and multiple developers 3 use each of the data analysis devices 600, the caches used for each data analysis are integrated and managed in the analysis location cache 150 in the cache DB management device 500, and further, the cache updates are permitted only under cache update constraints. Therefore, even if a developer 3 using multiple data analysis devices 600 performs a specific data operation (e.g., specification of an input DB) accompanied by a data update request to the analysis location cache 150, it is possible to limit the increase in access to the core database (database 210).

[0157] Also, from the user's point of view, in the data processing system 4 according to this embodiment, a function is implemented that allows the DB administrator 2 to impose a cache update constraint on the cache update of the analysis location cache 150 used in the development phase of data analysis, in a form including a constraint condition based on the number of update requests for each data. Also, a developer 3 using multiple data analysis devices 600 can request a cache update by voluntarily performing a predetermined data operation (e.g., input DB specification) accompanied by a data update request to the analysis location cache 150. When a data update is requested by the developer 3, the cache update of the target data is performed only if the update is possible based on the update constraint (including the update constraint on the number of update requests) imposed on the target data. Therefore, in order to reduce the access load to the database 210, a situation may occur in which the cache update of the target data is not performed immediately if the number of update requests does not reach a specified condition, but the cache update of the target data can be performed relatively early according to the desire of the developer 3.

[0158] 31 is a block diagram showing an example of a hardware configuration of each device (computer device 10) constituting the data processing system 1, 4 according to the first or second embodiment. Specifically, the example of the hardware configuration shown in FIG. 31 can be implemented in the data processing device 100, the database device 200, the cache DB management device 500, and the data analysis device 600.

[0159] As shown in FIG. 31, a computer device 10 includes an external storage device 11, a RAM (Random Access Memory) 12, a ROM (Read Only Memory) 13, a CPU (Central Processing Unit) 14, an output device 15, a communication IF 16, and an input device 17, and each component is interconnected via a bus 18.

[0160] The functions of each device constituting data processing system 1 (data processing system 4) may be realized in computer device 10, for example, by CPU 14 reading a program stored in ROM 13 into RAM 12 and executing it, or some or all of the functions may be realized by hardware such as a specially designed integrated circuit, or by a combination of software and hardware.

[0161] The external storage device 11 is, for example, a hard disk drive (HDD), a solid state drive (SSD), a flash memory, or an optical storage device. The output device 15 has a function of outputting data, and outputs a screen related to the data processing device 100, the cache DB management device 500, or the data analysis device 600 on an output device such as a monitor or a printer. The communication IF 16 is used for connecting the data processing device 100 and the database device 200 in the data processing system 1, or for connecting the cache DB management device 500 and the data analysis device 600 and the database 210 (database device) in the data processing system 4. The input device 17 is a keyboard, a mouse, or the like that accepts input operations by users (DB administrator 2, developer 3), and is used, for example, in the first embodiment, when the DB administrator 2 inputs DB information 101 and cache update constraints 102 to the data processing device 100, or when the developer 3 designs an analysis flow or specifies a database for the data processing development module 130.

[0162] Although the first and second embodiments have been described above as representative forms for carrying out the present invention, the present invention is not limited to the above-mentioned embodiments and includes various modified examples. For example, the above-mentioned embodiments have been described in detail to clearly explain the present invention, and are not necessarily limited to those having all of the configurations described. In addition, it is possible to replace a part of the configuration of one embodiment with the configuration of another embodiment, and it is also possible to add the configuration of another embodiment to the configuration of one embodiment. In addition, it is possible to add, delete, or replace a part of the configuration of each embodiment with another configuration.

[0163] For example, in the data processing system 1 according to the first embodiment (or the data processing system 4 according to the second embodiment), the cache size of the analysis location cache 150 is finite, and if all of the data used by the data processing device 100 (the multiple data analysis devices 600 in the second embodiment) cannot be stored in the cache, it is necessary to directly refer to the database 210 when accessing data that does not exist in the cache. Therefore, as a modified example of each embodiment, in such a situation, for example, the update determination module 120 may be configured to perform a cache data replacement process such that data that has been referenced more frequently by the data processing device 100 (data analysis device 600) is preferentially stored in the analysis location cache 150. The cache data replacement process will be described below with reference to FIG. 32.

[0164] Fig. 32 is an image diagram for explaining the cache data replacement process. In the case of Fig. 32, the number of times a cache update request has been made (number of update requests) is recorded by data (by schema, by table) in the update log 163. Specifically, for data A721 and data B722 whose caches are stored in the analysis location cache 150, the numbers of update requests are recorded as "100 times" and "10 times," respectively, and for data C723 whose caches are not stored in the analysis location cache 150, the number of update requests is recorded as "50 times."

[0165] In Fig. 32, a cache update is requested for data C723. Here, at the stage of step S511 in Fig. 9 (step S1113 in Fig. 28 in the second embodiment) where the update determination module 120 determines that the cache can be updated and a query is sent to the database 210, if the data capacity in the analysis location cache 150 is full (which may be interpreted as reaching a predetermined upper limit capacity or capacity rate) and the previous cache of data C723 does not exist in the analysis location cache 150, the update determination module 120 executes a cache data replacement process.

[0166] In the cache data replacement process, the update determination module 120 obtains the number of update requests for each data from the update log 163, deletes data that has fewer update requests than the data requested to be updated and that exists in the analysis location cache 150, and stores the data requested to be updated newly in the analysis location cache 150. Specifically, in the case of FIG. 32, data B722 (10 times), which has fewer update requests than data C723 (50 times) and exists in the analysis location cache 150, is selected as the data to be deleted. Note that data A721 (100 times) has a greater number of update requests than data C723 (50 times), so it is not selected as the data to be deleted and the cache in the analysis location cache 150 is maintained.

[0167] In this modified example, by performing the cache data replacement process as described above, data B with a relatively low number of update requests is deleted from the analysis location cache 150, and data C with a relatively high number of update requests is stored in the analysis location, so that data with a high number of references (schemas and tables) can be preferentially cached while suppressing the cache update frequency of the analysis location cache 150, and as a result, the number of times data is directly read into the database 210 can be reduced. In other words, the data processing system 1 (data processing system 4) can reduce the number of times the database 210 is directly referenced and reduce the access load by preferentially storing data with a high number of references from the data processing device 100 (data analysis device 600) in the analysis location cache 150.

[0168] In addition, in each of the drawings referred to in the above description, the control lines and information lines shown are those considered necessary for the description, and not all of the control lines and information lines in the product are necessarily shown. In reality, it can be considered that almost all components are interconnected. [Explanation of symbols]

[0169] 1,4 Data Processing System 2 DB administrator 3 Developer 10 Computer equipment 11 External storage device 12 RAM 13 ROM 14 CPU 15 Output Devices 16 Communication Interface 17 Input Devices 18 Bus 100 Data processing device 110 Cache Management Module 111 Update constraint input function 112 DB information input function 120 Update Decision Module 130 Data Processing Development Module 131 Analysis flow design function 132 DB information viewing function 140 Data Processing Module 150 Analysis Location Cache 151 Metadata 152 Column Data 160 Data storage unit 161 Registered DB Information 162 Cache Update Constraint Information 163 Update Log 170 Drawing section 171 Update button 200 Database device 201 Sensor Data 202 Log Data 210 Database 211 Metadata 212 Column Data 220 Database Management System 350 Registered DB list screen 360 DB information input screen 370 Cache update constraint input screen 380 Analysis flow design screen 390 DB specification screen 400 DB data display screen 500 Cache DB management device 501 Request Count Information 510 Management drawing section 600 Data Analysis Equipment 610 Data Processing Module 611 Analysis processing execution function 620 Analysis drawing section 700 Cache update constraint input screen

Claims

1. A data processing system that connects to a database and performs data analysis on the database, an analysis point cache that caches data of the database referenced in the development phase of the data analysis; a data processing development module used by a data analysis developer to develop analytical processes for the data analysis; a data processing module used by the developer to perform data analysis by executing the analytical process developed by the data processing development module; a cache management module that sets update constraints input by an administrator of the database regarding cache updates from the database in the analysis location cache; an update determination module that determines whether or not to update the analysis location cache based on the update constraint set by the cache management module; Equipped with The database data includes metadata including at least one of a schema name, a table name, and a column name, and column data representing the contents of a table; The update determination module: making the determination when the update constraint is input by the administrator and when the database is designated by the developer; When the update constraint is input and the determination is that the cache update is permitted, the cache update is executed by acquiring the metadata from the database and storing the acquired metadata in the analysis location cache; If the determination is that the cache update is possible when the database is specified, a determination result indicating that the update is possible is transmitted to the data processing development module, and when the cache update is requested from the data processing development module in response to the transmission of the determination result, at least one of the metadata and the column data is obtained from the database in response to an operation by the developer, and the obtained data is stored in the analysis location cache, thereby executing the cache update. A data processing system comprising:

2. a drawing unit that displays a screen to the developer, When the data processing development module receives the determination result from the update determination module, the data processing development module transmits a display notification of an update button for performing an operation to request the cache update to the drawing unit; When the drawing unit receives the display notification from the data processing development module, the drawing unit displays the update button in a form that can be operated by the developer.

2. The data processing system of claim 1.

3. The cache management module enables setting the update constraints on the data stored in the database on a schema-by-schema or table-by-table basis.

2. The data processing system of claim 1.

4. The cache management module allows the time or frequency at which the cache updates are permitted to be set in the update constraint.

2. The data processing system of claim 1.

5. a cache DB management device that includes the analysis location cache, the cache management module, and the update determination module and is used by the administrator; a plurality of data analysis devices used by the developers, each having the data processing development module and the data processing module; Equipped with In the cache DB management device, the analysis location cache caches the data of the database referenced by the plurality of data analysis devices in an integrated manner.

2. The data processing system of claim 1.

6. the cache DB management device further includes an update log that records the number of cache update requests from the plurality of data analysis devices; the cache management module defines a cumulative number of requests for the cache update as at least one of the update constraints; When the developer performs a predetermined data operation on the data analysis device, the data processing development module requests the cache DB management device to update the cache for the target data of the predetermined data operation; The update determination module of the cache DB management device determines whether or not to allow a cache update of the target data in the analysis location cache based on the update constraint including the definition of the cumulative number of times, and executes the requested cache update if the determination determines that the cache update is allowed.

6. The data processing system of claim 5.

7. The cache management module has a function of caching data with a high number of references preferentially in the analysis location cache.

6. The data processing system according to claim 1 or claim 5.

8. A cache update control method for a data processing system that performs data analysis of a database, comprising: The data processing system includes: an analysis point cache that caches data of the database referenced in the development phase of the data analysis; a data processing development module used by a data analysis developer to develop analytical processes for the data analysis; a data processing module used by the developer to perform data analysis by executing the analytical process developed by the data processing development module; a cache management module that sets update constraints input by an administrator of the database regarding cache updates from the database in the analysis location cache; an update determination module that determines whether or not to update the analysis location cache based on the update constraint set by the cache management module; and The database data includes metadata including at least one of a schema name, a table name, and a column name, and column data representing the contents of a table; When the update constraint is input by the administrator, the update determination module makes the determination, and if the determination is that the cache update is permitted, the metadata is acquired from the database, and the acquired metadata is stored in the analysis location cache, thereby executing the cache update; When the developer specifies the database, the update determination module makes the determination, and if the determination indicates that the cache update is permitted, when the data processing development module requests the cache update, the update module obtains at least one of the metadata and the column data from the database in response to the developer's operation, and stores the obtained data in the analysis location cache, thereby executing the cache update.

2. A cache update control method comprising:

9. the data processing system further comprises a drawing unit that displays a screen to the developer; when the database is designated by the developer and the update determination module determines that the cache update is permitted, the data processing development module transmits to the drawing unit a notification to display an update button for performing an operation to request the cache update; When the drawing unit receives the display notification from the data processing development module, the drawing unit displays the update button in a form that can be operated by the developer.

9. The cache update control method according to claim 8.

Images