A file processing method, device, apparatus and storage medium

Abstract

The application relates to the technical field of data processing, and discloses a file processing method, device and equipment and a storage medium, which are applied to a node server and include the following steps: storing a preset type file into a first data warehouse in a distributed storage system; wherein the preset type file is obtained by a computing engine from original data and is obtained by the computing engine from the original data; when a first service deployed locally detects that the preset type file exists in the first data warehouse, the preset type file is transferred to a second data warehouse in the distributed storage system by the first service for storage, so that the first service queries the preset type file from the second data warehouse; wherein the first service has access permissions of the first data warehouse and the second data warehouse during file transfer, and the first service has only the access permission of the second data warehouse during file query. In the application, the first service can have different permissions when transferring and querying files, thereby improving access security.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a file processing method, apparatus, device, and storage medium. Background Technology

[0002] HDFS (Distributed Storage System) is a distributed storage system based on a streaming data access model. Its design, grounded in "write once, read many times," provides high-throughput, high-fault-tolerant data access, effectively solving the problem of storing massive amounts of data. However, the existing HDFS distributed storage system cannot isolate access permissions between storage directories, leading to reduced data security. For example, in a log service, it's desirable for users to only access log files and not other files, but due to the storage characteristics of the HDFS distributed file system, data access permission isolation between storage directories cannot be achieved.

[0003] Therefore, the above-mentioned technical problems urgently need to be solved by those skilled in the art. Summary of the Invention

[0004] In view of this, the purpose of this invention is to provide a data processing method, apparatus, device, and storage medium capable of separately storing files of preset types, thereby improving access security. The specific solution is as follows:

[0005] The first aspect of this application provides a data processing method applied to a node server, comprising:

[0006] A preset type file is stored in the first data warehouse of the distributed storage system; wherein, the preset type file is obtained by the computing engine from the original data and the original data is calculated.

[0007] When the first service deployed locally detects the existence of the preset type file in the first data warehouse, it uses the first service to transfer the preset type file to the second data warehouse in the distributed storage system for storage, so that the first service can query the preset type file from the second data warehouse; wherein, during the file transfer process, the first service has access to both the first data warehouse and the second data warehouse, and during the file query process, the first service only has access to the second data warehouse.

[0008] Optionally, the file processing method further includes:

[0009] Retrieve the file request sent by the client, which contains information about the target file;

[0010] The first service is used to determine the target address corresponding to the file request; wherein, the file at the target address is a file of a preset type that has been transferred from the first data warehouse to the second data warehouse by the first service.

[0011] The target address is returned to the client so that the client can use the second service deployed on the node server to obtain the target file at the target address mounted locally in the distributed file system.

[0012] Optionally, the first service in the file transfer process is based on a shared user in the first user group transferring the preset type of file in the first data warehouse to the second data warehouse for storage; wherein, the first data warehouse belongs exclusively to the first user group, the second data warehouse belongs exclusively to the second user group, and the shared user is also a user in the second user group.

[0013] Optionally, the first service in the file query process is based on users in the second user group other than the users in the first user group querying the second data warehouse to obtain the target address corresponding to the file request.

[0014] Optionally, the client utilizes a second service deployed on the node server to obtain the target file at the target address mounted on the node server, including:

[0015] The client sends the target address to the node server with the NFS gateway deployed through the proxy server;

[0016] The NFS gateway mounts the file cluster of the distributed file system to the node server to obtain the target file at the target address of the distributed file system.

[0017] Optionally, the step of using the first service to transfer the preset type file to the second data warehouse in the distributed storage system for storage includes:

[0018] The first service is used to rename the preset type file according to a preset naming rule; wherein the preset naming rule is sent to the local machine by the user through the client;

[0019] The first service is used to store the renamed preset type files in the directory corresponding to the second data warehouse according to the preset storage directory; wherein, the preset storage directory is a hierarchical directory containing log data generation time and client user domain name.

[0020] Optionally, before storing the renamed preset type file in the directory corresponding to the second data warehouse according to the preset storage directory using the first service, the method further includes:

[0021] The first service is used to encrypt the client user's domain name to obtain an encrypted directory, so that the returned target address contains the encrypted directory.

[0022] A second aspect of this application provides a data processing apparatus applied to a node server, comprising:

[0023] A computing and storage module is used to store preset type files in a first data warehouse of a distributed storage system; wherein the preset type files are obtained by the computing engine from the original data and performing calculations on the original data;

[0024] The transfer module is used to transfer the preset type file to a second data warehouse in the distributed storage system when a first service deployed locally detects the existence of the preset type file in the first data warehouse, so that the first service can query the preset type file from the second data warehouse; wherein, during the file transfer process, the first service has access to both the first data warehouse and the second data warehouse, and during the file query process, the first service only has access to the second data warehouse.

[0025] A third aspect of this application provides an electronic device comprising a processor and a memory; wherein the memory is used to store a computer program, the computer program being loaded and executed by the processor to implement the aforementioned data processing method.

[0026] A fourth aspect of this application provides a computer-readable storage medium storing computer-executable instructions, which, when loaded and executed by a processor, implement the aforementioned data processing method.

[0027] In this application, a preset type file is stored in a first data warehouse of a distributed storage system. The preset type file is obtained by a computing engine acquiring raw data and performing calculations on it. When a first service deployed locally detects the existence of the preset type file in the first data warehouse, it transfers the preset type file to a second data warehouse in the distributed storage system for storage, enabling the first service to query the preset type file from the second data warehouse. During the file transfer process, the first service has access permissions to both the first and second data warehouses; during the file query process, the first service only has access permissions to the second data warehouse. Therefore, the direct storage address of the preset type file obtained by the computing engine from the raw data is the first data warehouse of the distributed storage system. To separate the preset type file from the first data warehouse, a first service is deployed on the node server to monitor the files in the first data warehouse and transfer detected preset type files to the second data warehouse of the distributed storage system. This separates the preset type file from other types of files in the first data warehouse, allowing the first service to have different permissions when transferring and querying files, thus improving access security. Attached Figure Description

[0028] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0029] Figure 1 A flowchart of a document processing method provided in this application;

[0030] Figure 2 A flowchart of a specific document processing method provided in this application;

[0031] Figure 3 A flowchart of a specific document processing method provided in this application;

[0032] Figure 4 This application provides a specific file processing system architecture diagram;

[0033] Figure 5 This application provides a specific diagram illustrating the interaction between an NFS gateway and HDFS.

[0034] Figure 6 A flowchart of a specific document processing method provided in this application;

[0035] Figure 7 This application provides a schematic diagram of a file directory in a first data warehouse;

[0036] Figure 8 This application provides a schematic diagram of a file directory in a second data warehouse;

[0037] Figure 9 A schematic diagram of a document processing device provided in this application;

[0038] Figure 10 This application provides a structural diagram of a document processing electronic device. Detailed Implementation

[0039] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0040] The existing distributed storage system HDFS cannot isolate access permissions for data between storage directories, leading to reduced data security. For example, in a log service, it is desirable for users to only access log files and not other files. However, due to the storage characteristics of the HDFS distributed file system, it is impossible to isolate data access permissions between storage directories. To address these technical shortcomings, this application provides a file processing solution. A first service is deployed on a node server, and files in a first data warehouse are monitored. Files of a preset type are transferred to a second data warehouse in the distributed storage system. This separates the preset type files from other types of files in the first data warehouse, allowing the first service to have different permissions when transferring and querying files, thereby improving access security.

[0041] Figure 1 A flowchart illustrating a file processing method provided in an embodiment of this application. See also... Figure 1 As shown, this file processing method is applied to the node server and includes:

[0042] S11: Store a preset type file in the first data warehouse of the distributed storage system; wherein the preset type file is obtained by the computing engine from the original data and the original data is calculated.

[0043] In this embodiment, the computing engine first acquires the raw data and performs calculations on it, resulting in a file provided to the user. The raw data can be log data, and the corresponding preset file type is a log file. Log data calculations are generally performed offline, and this embodiment does not impose any limitations on this.

[0044] Then, the node server stores the preset type file obtained by the computing engine through computation into the first data warehouse of the distributed storage system. For example, the node server stores the log file obtained from offline computation in the first data warehouse. It can be understood that offline computation can be performed using Hive. During this process, the distributed storage system HDFS is used to store the data, and the completed data is stored in a data warehouse, directly into the Hive data warehouse, i.e., the first data warehouse, and then displayed from the Hive data warehouse.

[0045] S12: When the first service deployed locally detects that the preset type file exists in the first data warehouse, it uses the first service to transfer the preset type file to the second data warehouse in the distributed storage system for storage, so that the first service can query the preset type file from the second data warehouse; wherein, during the file transfer process, the first service has access permissions to both the first data warehouse and the second data warehouse, and during the file query process, the first service only has access permissions to the second data warehouse.

[0046] In this embodiment, a first service is deployed on the node server. This first service monitors files in the first data warehouse to discover files of the preset type in the first data warehouse. The monitoring operation can be triggered actively or passively; this embodiment does not limit this.

[0047] In this embodiment, the first service also has the function of transferring files. When the first service deployed locally detects the existence of the preset type file in the first data warehouse, the node server uses the first service to transfer the preset type file to the second data warehouse in the distributed storage system for storage. For example, the first service transfers log files from the first data warehouse to the second data warehouse to isolate log files from other files in terms of permissions. This allows users to directly access the log files in the second data warehouse without accessing other types of files in the first data warehouse, thus ensuring data security. During this process, the first service has access permissions to both the first and second data warehouses.

[0048] In this embodiment, the first service also has the function of querying files. Since the preset type files have been transferred to the second data warehouse, when querying the preset type files, the first service only has access permissions to the second data warehouse. Taking the log service as an example, after the first service transfers the log files, all log files are stored in the second data warehouse. When the client retrieves log files, the first service can only query from the second data warehouse. Obviously, the access permissions during file query are less than the access permissions during file transfer, thus preventing access to other file data in the first data warehouse and ensuring data security.

[0049] As can be seen, in this embodiment, the node server first stores the preset type file obtained by the computing engine through offline computation in the first data warehouse of the distributed storage system. The preset type file is obtained by the computing engine acquiring raw data and performing calculations on the raw data. When a first service deployed locally detects the existence of the preset type file in the first data warehouse, it uses the first service to transfer the preset type file to the second data warehouse of the distributed storage system for storage, so that the first service can query the preset type file from the second data warehouse. During the file transfer process, the first service has access permissions to both the first and second data warehouses simultaneously; during the file query process, the first service only has access permissions to the second data warehouse. In this embodiment, the direct storage address of the preset type file obtained by the computing engine through offline computation of raw data is the first data warehouse of the distributed storage system. To separate the preset type file from the first data warehouse, a first service is deployed on the node server to monitor the files in the first data warehouse and transfer the detected preset type files to the second data warehouse of the distributed storage system. This separates the preset type file from other types of files in the first data warehouse, allowing the first service to have different permissions when transferring and querying files, thus improving access security.

[0050] Figure 2 This is a flowchart illustrating a specific file processing method provided in an embodiment of this application. See also... Figure 2 As shown, the file processing method is applied to the node server and includes:

[0051] S21: Store a preset type file in the first data warehouse of the distributed storage system; wherein the preset type file is obtained by the computing engine from the original data and the original data is calculated.

[0052] S22: When the first service deployed locally detects that the preset type file exists in the first data warehouse, the first service transfers the preset type file in the first data warehouse to the second data warehouse for storage based on the shared user in the first user group; wherein, the first data warehouse belongs exclusively to the first user group, the second data warehouse belongs exclusively to the second user group, and the shared user is also a user in the second user group.

[0053] In this embodiment, the computing engine primarily performs calculations offline. For example, the Hive computing engine retrieves raw log data from the HDFS distributed storage system and calculates the log file displayed to the user offline. The process of the computing engine retrieving raw data is executed by the computing engine through users registered in the HDFS distributed storage system.

[0054] Therefore, before performing offline computation, the computing engine needs to register in the distributed storage system to create a user, who becomes the leader of the first user group. The offline computing service uses this user to pull data during offline computation, and the first service also uses this user for file transfer. Each user in the first user group corresponds to a folder in the first data warehouse, and this user has all permissions within that folder. For example, if a customer's hourly logs are generated through Hive offline computation, the offline computing service uses this user to store the final computation results in the `hive` folder, which is located in the first data warehouse.

[0055] The first data warehouse belongs exclusively to the first user group. "Exclusively" means that the data warehouse belongs specifically to a particular member. The first data warehouse can belong to the group leader of the first user group, or it can be exclusive to the first user group, belonging to all users within the first user group.

[0056] In this embodiment, when the first service deployed locally detects the existence of the preset type file in the first data warehouse, the node server uses the first service to transfer the preset type file to the second data warehouse in the distributed storage system for storage, based on a shared user in the first user group. The reason it's called a shared user is that the user transferring the file is a user in both the first and second user groups. Similarly, the second data warehouse belongs exclusively to the second user group, and users in the second user group have access to the second data warehouse. Only when the file transfer is based on a user existing in both user groups can the first service simultaneously access both the first and second data warehouses. That is, only when the user on which the first service is based is a user in both the first and second user groups can it simultaneously access both the first and second data warehouses, and only then can the first service transfer files from the first data warehouse to the second data warehouse. Specifically, the shared user is the group leader in the first user group and a member in the second user group.

[0057] S23: Obtain the file request sent by the client containing information about the target file.

[0058] In this embodiment, the aforementioned steps are the process of storing files in the distributed storage system HDFS. When retrieving a stored file, the client first sends a file request to the node server. The node server retrieves the file request sent by the client. The file request contains target file information, which is the relevant information of the target file that the client wants to obtain.

[0059] S24: The first service queries the second data warehouse based on users in the second user group other than the users in the first user group, and obtains the target address corresponding to the file request.

[0060] S25: Return the target address to the client.

[0061] In this embodiment, after obtaining the file request, the first service is further used to determine and return an address. First, the node server uses the first service to determine the target address corresponding to the file request. Specifically, it queries the second data warehouse based on users in the second user group to obtain the target address corresponding to the file request. It should be noted that the user used for the query is a user in the second user group other than those in the first user group. This user has access to the second data warehouse but not to the first data warehouse. Therefore, the first service performing the query cannot access files in the first data warehouse, thus separating user permissions and ensuring data storage security.

[0062] The first service then returns the target address to the client. The file at the target address is the file of the preset type that the first service has transferred from the first data warehouse to the second data warehouse for storage. The target address is the storage address of the preset type file in the distributed storage system HDFS, which is also the file address in the second data warehouse.

[0063] S26: The client uses a second service deployed on the node server to obtain the target file at the target address of the distributed file system mounted on the node server.

[0064] In this embodiment, after the node server returns the target address to the client through the first service, the client retrieves the target file from the target address. The node server also deploys a second service, which the client uses to access storage files in the distributed storage system.

[0065] Figure 3 This is a flowchart illustrating a specific file processing method provided in an embodiment of this application. See also... Figure 3 As shown, the file processing method is applied to the node server and includes:

[0066] S31: Obtain the target file information corresponding to the file request from the proxy server; wherein, the target file information is the information obtained after the client sends the file request to the parsing server so that the parsing server parses the file request.

[0067] In this embodiment, the client does not directly send a file request to the node server. Instead, the client first sends the file request to a parsing server. The file request contains the target file information, which the parsing server parses to obtain the target file information. The parsing server then sends the parsed target file information to the proxy server, and the node server obtains the target file information corresponding to the file request from the proxy server. The architecture of the above file processing process is as follows: Figure 4 As shown, the Node deploys the nginx service, providing file download services. Nodes can scale horizontally indefinitely, ensuring high availability of the entire download service.

[0068] S32: Determine the target address corresponding to the target file information using a first service deployed locally; wherein, the file under the target address is a file of a preset type that is transferred from the first data warehouse to the second data warehouse by the first service.

[0069] In this embodiment, the specific process of step S32 described above can be found in the corresponding content disclosed in the previous embodiments, and will not be repeated here. The first service is exemplarily the Npetune service, which provides operations such as inserting, deleting, modifying, and querying logs through a RESTful API. Simultaneously, the Npetune service receives log download requests and returns the download path corresponding to the logs based on the requests.

[0070] S33: The target address is returned to the client using a first service deployed locally, so that the client can send the target address to the node server with the NFS gateway deployed through the proxy server.

[0071] S34: The NFS gateway mounts the file cluster of the distributed file system to the node server to obtain the target file in the target address of the distributed file system.

[0072] In this embodiment, the first service in the node server returns the target address to the client after determining it. The client further sends the target address to the node server with an NFS gateway deployed through the proxy server. The NFS gateway is also the second service. NFS (Network File System) is a UNIX presentation layer protocol that allows users to access files on the network as if they were using their own computer. NFS allows users to access remote file systems as if they were accessing a local file system. Introducing NFS into HDFS allows users to read and write files on HDFS as if they were local files, greatly simplifying HDFS usage. This is achieved by introducing an NFS gateway service, which converts the NFS protocol into the HDFS access protocol, specifically as follows... Figure 5 As shown, clients on a Node can communicate with the local NFS gateway via the NFSv3 protocol. The NFS gateway communicates with the HDFS metadata node NameNode via the ClientProtocol protocol, obtaining metadata about HDFS cluster files, including file name, file size, file format, and file path. When a client needs to access a file, the gateway then communicates with the DataNode containing the file via the DataTransferProtocol protocol, transferring the file content back to the Node. This allows the Node to read and write files on HDFS as if they were local files.

[0073] For the offline log download solution, after the log files are generated in the HDFS distributed storage system, it is only necessary to deploy an NFS gateway service on the node server. This service allows the HDFS distributed storage system to be mounted as part of the client's local file system. Therefore, the log files do not need to be pulled to the local machine on the node server. The files on the HDFS distributed storage system can be accessed remotely by accessing the local disk.

[0074] It should be noted that the node server mentioned here can be the same server as the node server described above, or it can be a different server. The node server is one of the servers in a server cluster, and each node server can simultaneously deploy the first service and the second service, such as... Figure 4As shown, only one service can be deployed. The specific deployment method is set according to business needs, and this embodiment does not limit this. Compared with the existing log file-based local download implementation scheme, where log files are generated and stored in HDFS, and then downloaded from HDFS on the node and stored locally, and provided for external download through a local nginx proxy, the node in this embodiment does not require a large disk space to store files. Low-configuration servers can also be used as node nodes. Furthermore, it avoids storing the same files on both the node and HDFS, avoiding redundant file storage, reducing resource waste and additional maintenance costs. At the same time, storing files only on HDFS can fully utilize the fault tolerance and high availability provided by HDFS. Additionally, the download service provided by the node is multi-point; if the machine fails, it cannot provide download services externally. The process does not require multiple manual interventions, resulting in high automation efficiency.

[0075] Figure 6 This is a flowchart illustrating a specific file processing method provided in an embodiment of this application. See also... Figure 6 As shown, this file processing method is applied to the node server and includes:

[0076] S41: Store the preset type file obtained by the computing engine through offline computing to the first data warehouse in the distributed storage system; wherein, offline computing refers to the computing engine obtaining the raw data and performing calculations on the raw data.

[0077] S42: When the first service deployed locally detects that the preset type file exists in the first data warehouse, it renames the preset type file according to the preset naming rules using the first service; wherein, the preset naming rules are sent to the local machine by the user through the client.

[0078] Based on the aforementioned embodiments, when a user wants to obtain the log file corresponding to the service traffic of a certain domain under a certain time period, the first data warehouse contains the corresponding log file. At this time, the directory of the first data warehouse contains the log data generation time and the client user domain name.

[0079] For example, the logs for customer A's www.a1.com domain in the bus service, from 00:00 to 01:00 on January 1, 2022, consist of two files: 0000.gz and 0001.gz. These files are the results of offline calculations. When the node server stores 0000.gz and 0001.gz in the first data warehouse Hive in the distributed storage system, the path to store the calculation results is: / user / hive / warehouse / bus.db / custom_a_log / ptime=20220101000000 / dkey=www.a1.com. Figure 7 As shown, hive is the first data warehouse, bus.db and custom_a_log are the database name and table name under hive, and ptime and dkey correspond to the partition name of this table. Since the logs of customer A need to be stored in categories by time and domain name, there are two levels of partitions here: ptime and dkey. ptime is an integer time unit with the format YYYYMMDDHHIISS, and dkey is a string with the value corresponding to the domain name of customer A.

[0080] Figure 7 In the ` / user / hive / warehouse` directory, all file permissions are inherited from the `hive` user group (the first user group mentioned above), meaning only the `hive` user group has read and write permissions. Since the `hive` user group has permissions over the entire Hive data warehouse, this is excessive, making it unsuitable to use either the `hive` user group or individual `hive` users for accessing the log service. Therefore, this embodiment creates a new data warehouse, `xycloud` (the second data warehouse), and the user group corresponding to the `xycloud` data warehouse is the `xycloud_log` user group (the second user group). The log service is uniformly accessed by users in the `xycloud_log` user group. For the bus service, besides the log service data, the results of other analysis tasks are also stored under ` / user / hive / warehouse / bus.db / `. If permissions for this directory are granted to the `xycloud_log` user group, then users in this group can access other data in this service, resulting in excessive permissions. However, if the log files are placed in the `xycloud_log` user's directory (` / user / xycloud`), then permissions for all log files can be centralized and managed uniformly.

[0081] The first service can proactively detect and locate log files under ` / user / hive / warehouse / bus.db / custom_a_log`, and move the files in that directory to the directory corresponding to the date and the customer's domain name. The first service uses the `hive` user and grants the `hive` user permissions to manipulate the `xycloud_log` file. This service has permissions for both folders, allowing it to move the generated log files from the `hive` folder to the `xycloud_log` folder.

[0082] S43: Using the first service, the renamed preset type file is stored in the directory corresponding to the second data warehouse according to the preset storage directory; wherein, the preset storage directory is a hierarchical directory containing log data generation time and client user domain name.

[0083] When transferring files, if a user sends a preset naming rule via a client, and the preset storage directory is a hierarchical directory containing log data generation time and client user domain name, then the node server can use the first service to rename the preset type files according to the preset naming rule. This addresses the personalized requirements of some clients for log files (filenames and suffixes need to be named according to the client's specifications). The first service can rename files simultaneously to meet client requirements. For example, in the above embodiment, client a's two files will be renamed to www.a1.com_20220101000000_20220101010000.tar.gz.1 and www.a1.com_20220101000000_20220101010000.tar.gz.2 respectively during the transfer. Figure 8 As shown.

[0084] S44: Use the first service to encrypt the client user domain name to obtain an encrypted directory, so that the returned target address contains the encrypted directory.

[0085] In this embodiment, the node server can also use the first service to encrypt the client user's domain name to obtain an encrypted directory, so that the returned target address contains the encrypted directory. The purpose of encryption is to prevent users who obtain the address from having a collision and thus obtaining files under other domain names. For example, by performing MD5 encryption on the value of the client user's domain name, the final storage directory of the aforementioned client's file is: / user / xycloud_log / 20220101 / b54f05eb4b7bab9ca65b790b86c4892e / , where b54f05eb4b7bab9ca65b790b86c4892e corresponds to the MD5 value of the domain name www.a1.com.

[0086] In one specific embodiment, for a file request (time, domain name) sent by a client: www.xycloudlog.com / queryLog?domain=www.a1.com&startTime=20201115000000&endTime=20201115010000&uid=12345, the target address returned by the node server is: cd / user / xycloud_log / 20220101 / b54f05eb4b7bab9ca65b790b86c4892e / .

[0087] See Figure 9 As shown in the figure, this application also discloses a document processing device, including:

[0088] The computing storage module 11 is used to store a preset type file obtained by the computing engine through offline computing into the first data warehouse of the distributed storage system; wherein, the preset type file is obtained by the computing engine acquiring raw data and performing calculations on the raw data;

[0089] The transfer module 12 is used to transfer the preset type file to the second data warehouse in the distributed storage system for storage when the first service deployed locally detects that the preset type file exists in the first data warehouse, so that the first service can query the preset type file from the second data warehouse; wherein, during the file transfer process, the first service has access permissions to both the first data warehouse and the second data warehouse, and during the file query process, the first service only has access permissions to the second data warehouse.

[0090] As can be seen, in this embodiment, the node server first stores the preset type file obtained by the computing engine through offline computation in the first data warehouse of the distributed storage system. The preset type file is obtained by the computing engine acquiring raw data and performing calculations on the raw data. When a first service deployed locally detects the existence of the preset type file in the first data warehouse, it uses the first service to transfer the preset type file to the second data warehouse of the distributed storage system for storage, so that the first service can query the preset type file from the second data warehouse. During the file transfer process, the first service has access permissions to both the first and second data warehouses simultaneously; during the file query process, the first service only has access permissions to the second data warehouse. In this embodiment, the direct storage address of the preset type file obtained by the computing engine through offline computation of raw data is the first data warehouse of the distributed storage system. To separate the preset type file from the first data warehouse, a first service is deployed on the node server to monitor the files in the first data warehouse and transfer the detected preset type files to the second data warehouse of the distributed storage system. This separates the preset type file from other types of files in the first data warehouse, allowing the first service to have different permissions when transferring and querying files, thus improving access security.

[0091] In some specific embodiments, the file processing apparatus further includes:

[0092] The request retrieval module is used to retrieve file requests sent by the client that contain information about the target file.

[0093] The address determination module is used to determine the target address corresponding to the file request using the first service; wherein, the file under the target address is a file of a preset type that has been transferred from the first data warehouse to the second data warehouse by the first service for storage;

[0094] The address return module is used to return the target address to the client so that the client can use the second service deployed on the node server to obtain the target file in the target address of the distributed file system mounted locally;

[0095] In some specific embodiments, the address return module further includes:

[0096] A sending unit is used for the client to send the target address to the node server with an NFS gateway deployed through a proxy server;

[0097] The mounting unit is used by the NFS gateway to mount the file cluster of the distributed file system to the node server in order to obtain the target file in the target address of the distributed file system.

[0098] In some specific embodiments, the transfer module 12 specifically includes:

[0099] The renaming unit is used to rename the preset type file according to the preset naming rules using the first service; wherein the preset naming rules are sent to the local machine by the user through the client;

[0100] A storage unit is used to store the renamed preset type files in the directory corresponding to the second data warehouse according to the preset storage directory using the first service; wherein, the preset storage directory is a hierarchical directory containing log data generation time and client user domain name;

[0101] An encryption unit is used to encrypt the client user domain name using the first service to obtain an encrypted directory, so that the returned target address contains the encrypted directory.

[0102] Furthermore, embodiments of this application also provide an electronic device. Figure 10 This is a structural diagram of an electronic device 20 according to an exemplary embodiment. The content of the diagram should not be construed as limiting the scope of this application.

[0103] Figure 10 This is a schematic diagram of the structure of an electronic device 20 provided in an embodiment of this application. The electronic device 20 may specifically include: at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input / output interface 25, and a communication bus 26. The memory 22 stores a computer program, which is loaded and executed by the processor 21 to implement the relevant steps in the file processing method disclosed in any of the foregoing embodiments.

[0104] In this embodiment, the power supply 23 is used to provide operating voltage for each hardware device on the electronic device 20; the communication interface 24 can create a data transmission channel between the electronic device 20 and external devices, and the communication protocol it follows can be any communication protocol applicable to the technical solution of this application, and is not specifically limited here; the input / output interface 25 is used to acquire external input data or output data to the outside world, and its specific interface type can be selected according to specific application needs, and is not specifically limited here.

[0105] In addition, the memory 22, as a carrier for resource storage, can be a read-only memory, random access memory, disk or optical disk, etc. The resources stored thereon can include operating system 221, computer program 222 and data 223, etc., and the storage method can be temporary storage or permanent storage.

[0106] The operating system 221 manages and controls the various hardware devices and computer programs 222 on the electronic device 20 to enable the processor 21 to perform calculations and processing on the massive amounts of data 223 in the memory 22. The operating system 221 can be Windows Server, Netware, Unix, Linux, etc. In addition to including computer programs capable of performing the file processing methods disclosed in any of the foregoing embodiments, the computer program 222 may further include computer programs capable of performing other specific tasks. The data 223 may include raw data collected by the electronic device 20.

[0107] Furthermore, this application also discloses a storage medium storing a computer program, which, when loaded and executed by a processor, implements the file processing method steps disclosed in any of the foregoing embodiments.

[0108] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to in the method section.

[0109] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0110] The document processing method, apparatus, device, and storage medium provided by the present invention have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only for the purpose of helping to understand the method and core ideas of the present invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A file processing method, characterized in that, Applied to node servers, including: A preset type file is stored in the first data warehouse of the distributed storage system; wherein, the preset type file is obtained by the computing engine from the original data and the original data is calculated. When the first service deployed locally detects the existence of the preset type file in the first data warehouse, it uses the first service to transfer the preset type file to the second data warehouse in the distributed storage system for storage, so that the first service can query the preset type file from the second data warehouse. This achieves the isolation of query permissions between the preset type file and other files, so that when users access the preset type file, they can directly access the preset type file in the second data warehouse and will not access other types of files in the first data warehouse. During the file transfer process, the first service has access to both the first data warehouse and the second data warehouse, while during the file query process, the first service only has access to the second data warehouse.

2. The file processing method according to claim 1, characterized in that, Also includes: Retrieve the file request sent by the client, which contains information about the target file; The first service is used to determine the target address corresponding to the file request; wherein, the file at the target address is a file of the preset type that is transferred from the first data warehouse to the second data warehouse by the first service; The target address is returned to the client so that the client can use the second service deployed on the node server to obtain the target file at the target address mounted on the distributed file system on the node server.

3. The file processing method according to claim 2, characterized in that, The first service in the file transfer process is based on the shared user in the first user group transferring the preset type file in the first data warehouse to the second data warehouse for storage. The first data warehouse belongs exclusively to the first user group, the second data warehouse belongs exclusively to the second user group, and the shared user is also a user in the second user group.

4. The file processing method according to claim 3, characterized in that, The first service in the file query process is based on users in the second user group other than the users in the first user group querying the second data warehouse to obtain the target address corresponding to the file request.

5. The document processing method according to claim 2, characterized in that, The client utilizes a second service deployed on the node server to obtain the target file at the target address of the distributed file system mounted on the node server, including: The client sends the target address to the node server with the NFS gateway deployed through the proxy server; The NFS gateway mounts the file cluster of the distributed file system to the node server to obtain the target file at the target address of the distributed file system.

6. The document processing method according to any one of claims 1 to 5, characterized in that, The step of using the first service to transfer the preset type file to the second data warehouse in the distributed storage system for storage includes: The first service is used to rename the preset type file according to a preset naming rule; wherein the preset naming rule is sent to the local machine by the user through the client; The first service is used to store the renamed preset type files in the directory corresponding to the second data warehouse according to the preset storage directory; wherein, the preset storage directory is a hierarchical directory containing log data generation time and client user domain name.

7. The document processing method according to claim 6, characterized in that, Before storing the renamed preset type file in the directory corresponding to the second data warehouse according to the preset storage directory using the first service, the method further includes: The first service is used to encrypt the client user's domain name to obtain an encrypted directory, so that the returned target address contains the encrypted directory.

8. A document processing device, characterized in that, Applied to node servers, including: A computing and storage module is used to store preset type files in a first data warehouse of a distributed storage system; wherein the preset type files are obtained by the computing engine from the original data and performing calculations on the original data; The transfer module is used to transfer the preset type file to a second data warehouse in the distributed storage system when a first service deployed locally detects the existence of the preset type file in the first data warehouse. This allows the first service to query the preset type file from the second data warehouse, thus isolating the preset type file from other files in terms of query permissions. This ensures that when a user accesses a preset type file, they directly access the preset type file in the second data warehouse and do not access other types of files in the first data warehouse. During the file transfer process, the first service has access permissions to both the first and second data warehouses. During the file query process, the first service only has access permissions to the second data warehouse.

9. An electronic device, characterized in that, The electronic device includes a processor and a memory; wherein the memory is used to store a computer program, which is loaded and executed by the processor to implement the file processing method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, Used to store computer-executable instructions, which, when loaded and executed by a processor, implement the file processing method as described in any one of claims 1 to 7.

Images