Data processing system and data processing method
The data processing system quickly identifies updated files by logging operations and creating snapshots, addressing inefficiencies in detecting sensitive data leakage by focusing on newly created or updated data.
Patent Information
- Application Number
- JP2025077914
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Filing Date
- 2025-05-08
- Publication Date
- 2026-11-19
AI Technical Summary
Existing techniques for detecting sensitive data leakage outside a region are inefficient due to the time-consuming process of checking modification dates of all files, which requires significant processing time.
A data processing system that includes storage servers and a data management server, which logs file operations, creates snapshots, and identifies updated files based on operation logs to expedite sensitive data detection.
Enables quick identification of updated files, reducing processing time and load on the data management server by targeting only newly created or updated data for sensitive data detection.
Smart Images

Figure 2026182023000001_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to a technique for managing files stored in a storage server.
Background Art
[0002] In recent years, in data utilization, there is an increasing number of cases where data located at a plurality of on-premise sites is moved and duplicated to sites with abundant computing resources to perform processing, learning, and the like. At the same time, regional regulations that prohibit the leakage of sensitive data such as personal information outside the region have been strengthened.
[0003] To solve these problems, Patent Document 1 discloses a technique for detecting sensitive data such as personal information and suppressing leakage outside the region.
Prior Art Documents
Patent Documents
[0004]
Patent Document 1
Summary of Invention
Problems to be Solved by the Invention
[0005] The technique of Patent Document 1 can avoid data leakage outside the region. However, there is a problem that it takes time to detect sensitive data.
[0006] For example, after sensitive data detection is performed once on all file data in file and object storage, the sensitive data detection time can be shortened by performing sensitive data detection only on newly created data and updated data.
[0007] In this case, for example, in the sensitive data detection process, the modification date and time of all files stored in the file object storage can be checked, files that have been updated since the last sensitive data detection can be extracted, and the sensitive data detection process can be applied to these files.
[0008] Performing this kind of processing requires checking the modification dates of all the large number of files stored in file object storage, which takes a considerable amount of time.
[0009] This invention has been made in view of the above circumstances, and its purpose is to provide a technology that can quickly identify files that have been updated. [Means for solving the problem]
[0010] To achieve the above objective, a data processing system relating to one perspective includes one or more storage servers for storing files, and a data management server for performing processing on the files, wherein the system has one or more processors, and the one or more processors store the details of each operation on the files as an operation log in the storage server, create a snapshot of the file system in the storage server, extract the details of operations performed from the operation log from the creation of the previous snapshot to the current snapshot, and create updated file information that can identify files that have been updated since the creation of the previous snapshot based on the extracted details of operations. [Effects of the Invention]
[0011] According to the present invention, it is possible to quickly identify files that have been updated. [Brief explanation of the drawing]
[0012] [Figure 1] Figure 1 is an overall diagram of the data processing system according to the first embodiment. [Figure 2]Figure 2 is a configuration diagram of the data management server according to the first embodiment. [Figure 3] Figure 3 is a configuration diagram of the storage server according to the first embodiment. [Figure 4] Figure 4 is a configuration diagram of the operation log according to the first embodiment. [Figure 5] Figure 5 is a configuration diagram of the snapshot list according to the first embodiment. [Figure 6] Figure 6 is a configuration diagram of the updated file list according to the first embodiment. [Figure 7] Figure 7 is a flowchart of client I / O processing according to the first embodiment. [Figure 8] Figure 8 is a flowchart of data extraction and processing control processing according to the first embodiment. [Figure 9] Figure 9 is a flowchart of updated file list creation processing according to the first embodiment. [Figure 10] Figure 10 is a configuration diagram of the operation log according to the second embodiment. [Figure 11] Figure 11 is a configuration diagram of the updated file list according to the second embodiment. [Figure 12] Figure 12 is a flowchart of data extraction and processing control processing according to the second embodiment. [Figure 13] Figure 13 is a configuration diagram of the data management server according to the third embodiment. [Figure 14] Figure 14 is a configuration diagram of the storage server according to the third embodiment. [Figure 15] Figure 15 is a flowchart of data extraction and processing control processing according to the third embodiment. [Description of Embodiments of the Invention]
[0013] Embodiments will be described with reference to the drawings. Note that the embodiments described below do not limit the invention according to the claims, and not all of the constituent elements and combinations thereof described in the embodiments are necessarily essential to the solution means of the invention.
[0014] In the following description, information may be described using the expression "AAA list", but the information may be represented by any data structure. That is, to indicate that the information is independent of the data structure, the "AAA list" may be referred to as "AAA information".
[0015] Further, in the following description, processing may be described with a "program" as the subject of operation, but since the program is executed by a processor to perform predetermined processing while appropriately using at least one of a storage unit and an interface unit, the subject of the processing may be the processor (or a computer or computer system having the processor). The program may be installed into the computer from a program source. The program source may be, for example, a program distribution server or a computer-readable recording medium. Further, in the following description, two or more programs may be implemented as one program, or one program may be implemented as two or more programs. Further, at least a part of the processing implemented by executing the program may be implemented by a hardware circuit (for example, an ASIC (Application Specific Integrated Circuit) or an FPGA (Field-Programmable Gate Array)).
[0016] Figure 1 is an overall configuration diagram of a data processing system according to the first embodiment.
[0017] The data processing system 1 includes a client 2, an infrastructure management server 3, a data management server 10, and one or more storage servers 20. The client 2, the data management server 10, and the storage servers 20 are connected via, for example, a data network 5. The data network 5 may be, for example, a LAN (Local Area Network) or a WAN (Wide Area Network). The infrastructure management server 3, the data management server 10, and the storage servers 20 are connected via a management network 6. The management network 6 may be, for example, a LAN or a WAN.
[0018] Client 2 is, for example, a PC (Personal Computer), which receives operation requests from the user regarding files and directories on the file system and sends them to the storage server 20.
[0019] The infrastructure management server 3 executes infrastructure management processing 4, which manages the infrastructure including the data management server 10 and the storage server 20.
[0020] One or more storage servers 20 provide one or more file systems. In this embodiment, a file system is constructed by a cluster consisting of multiple storage servers 20, and file management is performed by file object virtualization processing 21.
[0021] In a cluster, an operation log T0 that stores the details of operations performed on files in the file system and a snapshot of the file system at a given point in time are managed. Each snapshot includes an update file list T2 (update file information) that can identify files that have been updated (added, modified, deleted, etc.) between the time the previous snapshot was created (or the initial state if no previous snapshot exists) and the time the current snapshot was created.
[0022] The data management server 10 executes data management middleware processing 11 to manage the data on the storage server 20.
[0023] Figure 2 is a diagram showing the configuration of the data management server according to the first embodiment.
[0024] The data management server 10 is composed of a computer, such as a PC or a general-purpose server. The data management server 10 includes a CPU (Central Processing Unit) 12, memory 13, storage devices 14, and a network interface 15.
[0025] Network IF15 is an interface such as a wired LAN card or a wireless LAN card, and communicates with other devices (e.g., infrastructure management server 3, storage server 20) via the data network 5 and management network 6.
[0026] The CPU 12 is an example of a processor and performs various processes according to the programs stored in the memory 13 and / or storage device 14.
[0027] Memory 13 is, for example, RAM (RANDOM ACCESS MEMORY) and stores programs executed by the CPU 12 and necessary information. In this embodiment, memory 13 stores the data management middle program 16. The data management middle program 16 includes a data extraction / processing program 17 and an offset conversion program 18. When the data management middle program 16 is executed by the CPU 12, it executes data management middle processing 11. When the data extraction / processing program 17 is executed by the CPU 12, it extracts predetermined data (for example, sensitive data) from a file and performs data extraction / processing, such as anonymizing the sensitive data. When the offset conversion program 18 is executed by the CPU 12, it performs offset conversion, which converts the offset that has become the operation range in the file. Note that the offset conversion program 18 does not function in the first embodiment, but functions in the second embodiment described later.
[0028] The storage device 14 is, for example, a hard disk or flash memory, and stores programs executed by the CPU 12 and data used by the CPU 12. The storage device 14 also stores, for example, information about files containing sensitive data, and processed files after processing (anonymization) of sensitive data on the files.
[0029] Figure 3 is a configuration diagram of the storage server according to the first embodiment.
[0030] The storage server 20 is comprised of a computer, such as a general-purpose server. The storage server 20 includes a CPU 22, memory 23, storage devices 24, and a network interface 25.
[0031] Network IF25 is an interface such as a wired LAN card or a wireless LAN card, and communicates with other devices (e.g., client 2, infrastructure management server 3, data management server 10) via the data network 5 and management network 6.
[0032] The CPU 22 is an example of a processor that performs various processes according to the programs stored in the memory 23 and / or storage device 24.
[0033] Memory 23 is, for example, RAM, and stores programs executed by the CPU 22 and necessary information. In this embodiment, memory 23 stores the virtualization processing program 26. The virtualization processing program 26 includes an operation log recording processing program 27 and an update file list creation processing program 28. When executed by the CPU 22, the virtualization processing program 26 executes a file object virtualization process 21 that virtualizes and manages files in the file system. When executed by the CPU 22, the operation log recording processing program 27 executes an operation log recording process that records the details of operations on files and directories as an operation log T0. When executed by the CPU 22, the update file list creation processing program 28 executes an update file list creation process that creates an update file list T2.
[0034] The storage device 24 is, for example, a hard disk or flash memory, and stores programs executed by the CPU 22 and data used by the CPU 22. The storage device 24 stores the operation log T0, the snapshot list T1, and the updated file list T2. The operation log T0 may be stored in the storage device 24 of any of the multiple storage servers 20 in the cluster, or it may be distributed and stored across the storage devices 24 of the multiple storage servers 20. The snapshot list T1 and the updated file list T2 are stored in the storage device 24 of any of the multiple storage servers 20.
[0035] Figure 4 is a diagram showing the configuration of the operation log according to the first embodiment.
[0036] Operation log T0 stores entries for each operation performed on a file or directory. Each entry in operation log T0 includes fields for timestamp C51, operation type C52, file system name C53, file handler C54, and type C55.
[0037] The timestamp C51 stores the time information (timestamp) when the operation corresponding to the entry was performed. The operation type C52 stores the type of operation corresponding to the entry. Operation types include create, write, delete, etc. The file system name C53 stores the name of the file system targeted by the operation corresponding to the entry. The file handler C54 stores information that identifies the file or directory targeted by the operation corresponding to the entry. This information can be a file path. The type C55 stores the type of object targeted by the operation corresponding to the entry. Object types include files and directories.
[0038] Figure 5 is a diagram showing the configuration of the snapshot list according to the first embodiment.
[0039] Snapshot list T1 stores entries corresponding to file systems and snapshots in each file system. Entries in snapshot list T1 include the fields File system name C11, Type C12, and Snapshot name C13. File system name C11 stores the file system name of the file system corresponding to the entry. Type C12 stores the type of the entry. Types include file system and snapshot. Snapshot name C13 stores the snapshot name (snapshot name) if the entry corresponds to a snapshot. Entries in snapshot list T1 can identify the paths to file system metadata and snapshots in a file system. For example, the path to the metadata of a file system named FS1 is " / FS1 / .metadata.file", and the path to the snapshot of a file system named FS1 with the snapshot name "FS1_20240822_001" is " / FS1 / .snap / FS1_20240822_001 / .metadata.file".
[0040] Figure 6 is a diagram showing the configuration of the update file list according to the first embodiment.
[0041] The updated file list T2 stores entries for each updated operation target (file, directory) on the file system. Each entry in the updated file list T2 includes the path C31, type C32, size C33, creation date and time C34, modification date and time C35, and state change C36.
[0042] The path C31 stores the path of the target (file, directory) corresponding to the entry. The type C32 stores the type of target corresponding to the entry. The size C33 stores the size of the target corresponding to the entry. The creation date and time C34 stores the date and time the target corresponding to the entry was created. The modification date and time C35 stores the date and time the target corresponding to the entry was modified. The state change C36 stores the details of the operation performed on the target corresponding to the entry. Operation details include creation, update, deletion, etc.
[0043] Next, we will explain the processing operation of the data processing system 1.
[0044] First, we will explain the client I / O processing that handles I / O requests from client 2 in the storage server 20.
[0045] Figure 7 is a flowchart of the client I / O processing according to the first embodiment.
[0046] The virtualization processing program 26 (more precisely, the CPU 22 that executes the virtualization processing program 26) receives a file I / O request sent from client 2 (S1). Here, file I / O requests include file creation, file writing, file reading, file deletion, directory creation, directory deletion, etc. For file writing, the request includes information that identifies the file (such as a path name) and offset information that indicates the area of the file to be operated on.
[0047] Next, the virtualization processing program 26 executes file I / O processing in response to the file I / O request (S2).
[0048] Next, the operation log recording processing program 27 records the details of the file I / O operation corresponding to the file I / O request in the operation log T0 (S3).
[0049] Next, the virtualization processing program 26 sends a response to the file I / O request to client 2 (S4).
[0050] Next, we will explain the data extraction and processing control process.
[0051] Figure 8 is a flowchart of the data extraction and processing control process according to the first embodiment.
[0052] The data extraction and processing control process may be executed periodically, for example, by a scheduler, or it may be executed according to instructions from the user.
[0053] The data extraction and processing program 17 of the data management server 10 (more precisely, the CPU 12 that executes the data extraction and processing program 17) sends a request to the storage server 20 to create an updated file list (S11). The request to create an updated file list may include, for example, the file system name of the file system for which the updated file list is to be created, or it may include content that applies to all file systems.
[0054] On the storage server 20, the update file list creation processing program 28 executes the update file list creation process (see Figure 9) and creates the update file list T2 (S12).
[0055] Next, the data extraction and processing program 17 sends a request to the storage server 20 to retrieve the updated file list T2 (S13).
[0056] The update file list creation processing program 28 sends the created update file list T2 to the data management server 10 (S14).
[0057] The data extraction and processing program 17 receives the updated file list T2, processes the files included in the updated file list T2, performs data extraction and data processing, and stores the processed files in a predetermined storage device (S15). In the data extraction process, the data extraction and processing program 17 extracts sensitive data from the files to be processed, for example. In the data processing process, the data extraction and processing program 17 performs anonymization processing on the extracted sensitive data so that its actual content cannot be identified.
[0058] According to this embodiment, since the files to be processed are identified from the update file list T2, the files to be processed can be identified quickly. Furthermore, since only the files in the update file list T2 are targeted for processing, the processing load on the data management server 10 can be reduced, and the processing time can be shortened.
[0059] Next, the process of creating the updated file list (S12) will be explained in detail.
[0060] Figure 9 is a flowchart of the update file list creation process according to the first embodiment.
[0061] The update file list creation processing program 28 of the storage server 20 receives the update file list creation request and creates a snapshot (S21).
[0062] Next, the update file list creation program 28 updates the snapshot list T1 (S22). Specifically, the update file list creation program 28 adds an entry for the snapshot of the file system targeted this time to the snapshot list T1.
[0063] Next, the update file list creation program 28 refers to the snapshot list T1 and determines whether or not there is a previous snapshot of the target file system (S23).
[0064] As a result, if a previous snapshot exists (S23:Yes), the update file list creation program 28 sets the variable "Previous Snapshot Acquisition Date and Time" to the date and time of the previous snapshot (S24), and proceeds to step S26. Note that the snapshot acquisition date and time may be associated with the snapshot file. On the other hand, if there is no previous snapshot (S23:No), that is, if it is the first snapshot for the file system, the update file list creation program 28 sets the variable "Previous Snapshot Acquisition Date and Time" to an initial value (for example, 0) (S25), and proceeds to step S26.
[0065] In step S26, the update file list creation processing program 28 extracts the operation details from the operation log T0 from the variable, which is the date and time of the previous snapshot, up to the date and time when the update file list creation request was received.
[0066] Next, the update file list creation processing program 28 merges the acquired operation details for each file to create the update file list T2 (S27).
[0067] This update file list creation process allows you to create an update file list T2 that includes only the files that have been manipulated since the previous snapshot (or the initial state in the case of the first snapshot) up to the creation of the current snapshot.
[0068] Next, a data processing system according to the second embodiment will be described. Note that the data processing system according to the second embodiment can be configured using the same hardware as the data processing system 1 according to the first embodiment shown in Figures 2 and 3. Therefore, each component will be described using the reference numerals in Figures 2 and 3 as appropriate, and the differences will be the main focus of the explanation.
[0069] The data processing system according to the second embodiment includes an offset conversion processing program 18 and modifies some of the functions of the data extraction and processing program 17. Furthermore, the data processing system uses operation log T01 instead of operation log T0, and update file list T21 instead of update file list T2.
[0070] Figure 10 is a diagram showing the configuration of the operation log according to the second embodiment.
[0071] Operation log T01 further includes fields with offset C56 and length C57, in addition to the entry in operation log T0.
[0072] Offset C56 stores the offset that indicates the beginning of the range (operation range) targeted by the operation content corresponding to the entry. Length C57 stores the length of the range targeted by the operation content corresponding to the entry. The offset and length correspond to the offset information.
[0073] Figure 11 is a diagram showing the configuration of the update file list according to the second embodiment.
[0074] The update file list T21 further includes fields with offset C37 and length C38 for each entry in the update file list T2.
[0075] Offset C37 stores the offset indicating the beginning of the operation range targeted by the operation content for the file corresponding to the entry, if no offset conversion process has been performed. If an offset conversion process has been performed, it stores the offset indicating the beginning of the range that includes the operation range targeted by the operation content for the file corresponding to the entry and is also semantically recognizable (semantically recognizable range). Here, the semantically recognizable range is, for example, from the beginning to the end of the line containing the operation range if the file type is a CSV file, the range that includes the operation range and is enclosed in {} if the file type is a JSON file, and the range from the beginning of the keyframe if the file type is a video file.
[0076] If no offset conversion process has been performed, length C38 stores the length of the operation range targeted by the operation content for the file corresponding to the entry. If an offset conversion process has been performed, it stores the length of the semantically recognizable range of the file corresponding to the entry.
[0077] Next, the data extraction and processing control process according to the second embodiment will be described. The data extraction and processing control process may be executed periodically by, for example, a scheduler, or it may be executed according to instructions from the user.
[0078] Figure 12 is a flowchart of the data extraction and processing control process according to the second embodiment.
[0079] The data extraction and processing program 17 of the data management server 10 (more precisely, the CPU 12 that executes the data extraction and processing program 17) sends a request to the storage server 20 to create an updated file list (S31). The request to create an updated file list includes, for example, the file system name of the file system for which the updated file list is to be created.
[0080] On the storage server 20, the update file list creation processing program 28 executes the update file list creation process and creates the update file list T21 (S32). Here, the update file list creation process is modified in the process shown in Figure 9 to create the update file list T21 instead of the update file list T2, and stores the offset of the beginning of the merged range of the target range in the offset C37 of the entry in the update file list T21, and stores the length of the merged range in the length C38.
[0081] Next, the data extraction and processing program 17 sends a request to the storage server 20 to retrieve the updated file list T21 (S33).
[0082] The update file list creation processing program 28 sends the created update file list T21 to the data management server 10 (S34).
[0083] Next, the data extraction and processing program 17 receives the updated file list T21, and the offset conversion processing program 18 determines whether the offset conversion setting is valid (S35). Here, the offset conversion setting can be set in advance by the user.
[0084] As a result, if the offset conversion setting is not enabled (S35: No), the data extraction and processing program 17 processes the range specified by the offset and length of the files included in the updated file list T21, performs data extraction processing and data processing, and stores the processed files in a predetermined storage device (S36).
[0085] On the other hand, if the offset conversion setting is enabled (S35: Yes), the offset conversion processing program 18 detects the type of file included in the update file list T21 (S37), converts the offset and length for each file type to an offset (update offset) and length corresponding to the semantically recognizable range, sets them in the update file list T21 (S38), performs data extraction processing and data processing for the range indicated by the file's update offset, and stores the processed file in a predetermined storage device (S39).
[0086] According to this embodiment, since the data extraction process and data processing are performed only on the range indicating the offset and length of the files in the updated file list T21, the amount of processing can be reduced compared to when the entire file is processed. Furthermore, in this embodiment, when the offset conversion setting is enabled, the data extraction process and data processing are performed on the semantically recognizable range which expands the file operation range, thereby improving the extraction accuracy of the data extraction process.
[0087] Next, a data processing system according to the third embodiment will be described. The data processing system according to the third embodiment is a system in which the offset conversion process in the data processing system according to the second embodiment is performed on the storage server 20. The data processing system according to the third embodiment includes a data management server 10A instead of the data management server 10 according to the first embodiment, and a storage server 20A instead of the storage server 20.
[0088] Figure 13 is a diagram showing the configuration of the data management server according to the third embodiment. In Figure 13, components similar to those in the data management server 10 according to the first embodiment are denoted by the same reference numerals.
[0089] The data management server 10A has hardware similar to that of the data management server 10, but it does not store the offset conversion processing program 18 in the memory 13.
[0090] Figure 14 is a configuration diagram of the storage server according to the third embodiment. In Figure 14, components similar to those in the storage server 20 according to the first embodiment are denoted by the same reference numerals.
[0091] Storage server 20A has hardware similar to storage server 20, and further stores an offset conversion processing program 29 in memory 23. When executed by CPU 22, the offset conversion processing program 29 performs an offset conversion process that converts the offset that has become the operation range in the file.
[0092] Figure 15 is a flowchart of the data extraction and processing control process according to the third embodiment. The data extraction and processing control process may be executed periodically by, for example, a scheduler, or it may be executed according to instructions from the data management server 10A or the like.
[0093] The virtualization processing program 26 of the storage server 20A (more precisely, the CPU 22 that executes the virtualization processing program 26) sends an update file list creation request to the update file list creation processing program 28 (S41). The update file list creation request includes, for example, the file system name of the file system for which an update file list is to be created.
[0094] The update file list creation processing program 28 executes the update file list creation process and creates the update file list T21 (S42). Here, the update file list creation process is the same as the process in step S32 of Figure 12.
[0095] Next, the offset conversion processing program 29 determines whether or not the offset conversion setting is valid (S43). Here, the offset conversion setting can be set in advance by the user.
[0096] As a result, if the offset conversion setting is not valid (S43: No), the offset conversion processing program 29 sends a data extraction and processing request to the data management server 10A (S44). The data extraction and processing request includes the update file list T21.
[0097] The data extraction and processing program 17 of the data management server 10A processes data within a range specified by the offset and length of the files included in the update file list T21, performs data extraction and data processing, and stores the processed files in a predetermined storage device (S45).
[0098] On the other hand, if the offset conversion setting is enabled (S43: Yes), the offset conversion processing program 29 detects the types of files included in the update file list T21 (S46), converts the offset and length for each file type to offsets (update offsets) and lengths corresponding to the semantically recognizable range, sets them in the update file list T21 (S47), and sends a data extraction and processing request to the data management server 10A (S48). The data extraction and processing request includes the update file list T21.
[0099] The data extraction and processing program 17 of the data management server 10A performs data extraction and data processing on the range indicated by the update offset of the files included in the update file list T21, and stores the processed data in a predetermined storage device (S49).
[0100] In this embodiment, the offset conversion process is performed on the storage server 20A, so the offset conversion process can be performed quickly.
[0101] It should be noted that the present invention is not limited to the embodiments described above, and can be implemented with appropriate modifications without departing from the spirit of the invention.
[0102] For example, the above embodiment showed an example of extracting data containing sensitive data from a file system, but the present invention is not limited to this, and data other than sensitive data, such as a predetermined word, may also be extracted. Furthermore, the data processing is not limited to anonymization, but any data processing may be performed.
[0103] Furthermore, in the above embodiment, the offset was converted to be within a semantically recognizable range depending on the file type, but the present invention is not limited to this, and the offset may be converted to any range that includes the range in which the operation was performed and is part of the file.
[0104] Furthermore, in the above embodiment, it was determined whether or not to perform an offset conversion according to the offset conversion setting, but the present invention is not limited to this, and an offset conversion may always be performed. [Explanation of Symbols]
[0105] 1...Data processing system, 2...Client, 3...Infrastructure management server, 5...Data network, 6...Management network, 10...Data management server, 12...CPU, 13...Memory, 14...Storage device, 15...Network interface, 20...Storage server, 22...CPU, 23...Memory, 24...Storage device, 25...Network interface, T0, T01...Operation log, T2, T21...Update file list
Claims
1. A data processing system comprising one or more storage servers for storing files, and a data management server for performing processing on the files, Having one or more processors, The one or more processors mentioned above are: The details of each operation performed on the aforementioned file are stored as an operation log on the storage server. Create a snapshot of the file system on the aforementioned storage server, From the aforementioned operation log, extract the operations performed from the creation of the previous snapshot to the current snapshot. Based on the extracted operation details, update file information is created that identifies files that have been updated since the last snapshot was created. Data processing system.
2. The aforementioned operation log includes offset information indicating the range of operations on the target file. The updated file information includes offset information indicating the range of operations for the updated file. The data processing system according to claim 1.
3. The aforementioned processor, The offset information is changed to cover a portion of the file that is wider than the operating range. The data processing system according to claim 2.
4. The aforementioned processor, Identify the type of the updated file, and based on that type, change the offset information to a semantically recognizable range where the meaning can be recognized. The data processing system according to claim 3.
5. The aforementioned processor, If the type of the updated file is a CSV file, the semantically recognizable range is defined as the range from the beginning to the end of the line containing the operation range; if the type of the updated file is a JSON file, the semantically recognizable range is defined as the range containing the operation range and enclosed in {}; and if the type of the updated file is a video file, the beginning of the keyframe is defined as the beginning of the semantically recognizable range. The data processing system according to claim 4.
6. The processor of the storage server creates the update file information. The data processing system according to claim 1.
7. The processor of the aforementioned data management server is Based on the aforementioned updated file information, the system identifies the updated files, retrieves the updated files from the storage server, and performs predetermined processing on the updated files. The data processing system according to claim 1.
8. The processor of the aforementioned data management server is Extract specific data from the aforementioned updated file. The data processing system according to claim 7.
9. The processor of the aforementioned data management server is Extract sensitive data from the updated file. The data processing system according to claim 8.
10. The processor of the aforementioned data management server is Anonymization processing is performed on the aforementioned sensitive data. The data processing system according to claim 9.
11. The processor of the aforementioned data management server is The storage server is requested to provide the updated file information. The processor of the aforementioned storage server is When a request for the aforementioned updated file information is received, a snapshot of the file system on the storage server is created, the operations performed from the operation log between the creation of the previous snapshot and the current snapshot are extracted, updated file information is created that identifies files updated since the creation of the previous snapshot based on the extracted operations, and the updated file information is sent to the data management server. The data processing system according to claim 1.
12. A data processing method using a data processing system that includes one or more storage servers for storing files and a data management server for performing processing on the files, The details of each operation performed on the aforementioned file are stored as an operation log on the storage server. Create a snapshot of the file system on the aforementioned storage server, From the aforementioned operation log, extract the operations performed from the creation of the previous snapshot to the current snapshot. Based on the extracted operation details, update file information is created that identifies files that have been updated since the last snapshot was created. Data processing method.
Citation Information
Patent Citations
Storage system, data transfer control method, and data transfer control program
JP2024085152A
