Data processing method and device, electronic equipment, storage medium and program product

Abstract

The invention relates to a data processing method and device, electronic equipment, a storage medium and a program product. The data processing method comprises the steps of responding to a data operation request for a storage system, and obtaining operation parameter information from the data operation request; if the obtained operation parameter information contains operation parameters required for executing data operation on the target file, storing the operation parameter information into metadata of the target file; and executing data operation on the target file based on the data operation request to obtain updated file attribute content, and updating the file attribute content in the metadata of the target file based on the updated file attribute content. The metadata of the target file in the storage system can be perfected.

Description

Technical Field

[0001] The present application relates to the field of data storage technology, and in particular to a data processing method, device, electronic device, storage medium and program product. Background Art

[0002] Currently, big data is usually stored in distributed file systems, such as the Hadoop Distributed File System (HDFS), including structured data, semi-structured data, and unstructured data. Summary of the Invention

[0003] The embodiments of the present application provide a data processing method, apparatus, electronic device, storage medium, and program product for improving the metadata of files in a storage system, thereby avoiding errors or inability to process the file data during subsequent storage and processing, and thereby improving the data processing performance of the storage system.

[0004] On the one hand, an embodiment of the present application provides a data processing method, including:

[0005] Responding to a data operation request for a storage system, and obtaining operation parameter information from the data operation request;

[0006] If the acquired operation parameter information includes operation parameters required for performing data operations on the target file, the operation parameter information is saved in the metadata of the target file;

[0007] performing a data operation on the target file based on the data operation request to obtain updated file attribute content of the target file;

[0008] Based on the updated file attribute content, the file attribute content in the metadata of the target file is updated.

[0009] In one aspect, an embodiment of the present application provides a data processing device, comprising:

[0010] an acquiring unit, configured to respond to a data operation request for the storage system and acquire operation parameter information from the data operation request;

[0011] a saving unit, configured to save the acquired operation parameter information into metadata of the target file if the acquired operation parameter information includes operation parameters required for performing a data operation on the target file;

[0012] a data operation unit, configured to perform a data operation on the target file based on the data operation request, and obtain updated file attribute content of the target file;

[0013] An updating unit is configured to update the file attribute content in the metadata of the target file based on the updated file attribute content.

[0014] On the one hand, an embodiment of the present application provides an electronic device, including a processor and a memory, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor performs the steps of any one of the above-mentioned data processing methods.

[0015] On the one hand, an embodiment of the present application provides a computer-readable storage medium, which includes a computer program. When the computer program is run on an electronic device, the computer program is used to enable the electronic device to perform the steps of any one of the above-mentioned data processing methods.

[0016] On the one hand, an embodiment of the present application provides a computer program product, which includes a computer program, and the computer program is stored in a computer-readable storage medium; when a processor of an electronic device reads the computer program from the computer-readable storage medium, the processor executes the computer program, so that the electronic device performs the steps of any one of the above-mentioned data processing methods.

[0017] The embodiments of the present application provide a data processing method, device, electronic device, and storage medium. In response to a data operation request for a storage system, the method obtains operation parameter information from the data operation request. When the obtained operation parameter information contains the operation parameters required for performing data operations on a target file, the operation parameter information is saved in the metadata of the target file. On the one hand, it can ensure that the operation parameters required for performing data operations on the target file can be collected, and after performing the data operation on the target file based on the data operation request, the updated file attribute content of the target file is obtained, and the file attribute content in the metadata of the target file is updated based on the updated file attribute content. On the other hand, the metadata of the target file not only contains the operation parameters required for performing data operations, but also contains the updated file attribute content obtained after performing data operations. Therefore, the embodiments of the present application can improve the metadata of the target file in the storage system, avoid errors or problems that cannot be processed when the data of the target file is subsequently processed, and thus improve the data processing performance of the storage system. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0019] Figure 1 This is a schematic diagram of an application scenario of a data processing method in an embodiment of the present application;

[0020] Figure 2 This is a flow chart of a data processing method in an embodiment of the present application;

[0021] Figure 3 A schematic diagram of a system architecture for implementing a data processing method in an embodiment of the present application;

[0022] Figure 4 This is a functional diagram of a metadata collection module in an embodiment of the present application;

[0023] Figure 5 This is an interactive diagram of a metadata collection process in an embodiment of the present application;

[0024] Figure 6A Schematic diagram of the structure of a data processing device in an embodiment of the present application;

[0025] Figure 6B Schematic diagram of the structure of a data processing device in an embodiment of the present application;

[0026] Figure 7 This is a schematic diagram of the structure of an electronic device in an embodiment of the present application. DETAILED DESCRIPTION

[0027] To make the purpose, technical solutions, and advantages of the embodiments of this application more clear, the technical solutions of this application will be clearly and completely described below in conjunction with the drawings in the embodiments of this application. Obviously, the described embodiments are part of the embodiments of the technical solutions of this application, but not all of them. Based on the embodiments described in this application document, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the technical solutions of this application.

[0028] References to "one embodiment" or "some embodiments" in this specification mean that a particular feature, structure, or characteristic described in conjunction with that embodiment is included in one or more embodiments of the present application. Thus, phrases such as "in one embodiment," "in some embodiments," "in other embodiments," and "in other embodiments" appearing in various places in this specification do not necessarily refer to the same embodiment, but rather mean "one or more but not all embodiments," unless otherwise specifically emphasized. The terms "including," "comprising," "having," and variations thereof all mean "including but not limited to," unless otherwise specifically emphasized.

[0029] In related technologies, distributed file systems are usually only responsible for storing and querying data. After a file is stored in a distributed file system, the distributed file system will record the file's metadata. This metadata usually only includes the most basic information such as the latest write time, the latest access time, and file permissions. Therefore, other information about the file cannot be queried, such as the file creator, the file storage duration, the file operator, the type of data stored in the file, etc.; once the file is stored, the subsequent data processing may cause errors or even failure to process due to missing information. For example, when multiple users write data of the same data type to different files, the files cannot be merged because the data type of the files cannot be queried; for another example, the files cannot be automatically cleaned because the file storage duration cannot be queried. Therefore, how to improve the information contained in the file's metadata is very important for the storage and processing of files.

[0030] An embodiment of the present application provides a data processing method, which includes: responding to a data operation request for a storage system, obtaining operation parameter information from the data operation request, and if the obtained operation parameter information includes the operation parameters required to perform data operations on a target file, saving the operation parameter information to the metadata of the target file, performing a data operation on the target file based on the data operation request, and obtaining updated file attribute content of the target file; and updating the file attribute content in the metadata of the target file based on the updated second attribute content of the file. Since the metadata of the target file not only includes the operation parameters required to perform data operations, but also includes the updated file attribute content obtained after performing data operations, the embodiment of the present application improves the metadata of the target file in the storage system, avoids errors or problems that cannot be processed when the target file data is subsequently stored and processed, and improves the data processing performance of the storage system.

[0031] The preferred embodiments of the present application are described below in conjunction with the drawings in the specification. It should be understood that the preferred embodiments described herein are only used to illustrate and explain the present application and are not used to limit the present application. In addition, the embodiments and features in the embodiments of the present application can be combined with each other if there is no conflict.

[0032] In order to better understand the data processing method, device, electronic device, storage medium and program product provided by the embodiments of the present application, the application environment applicable to the embodiments of the present application is described below.

[0033] See also Figure 1 , Figure 1 Schematic diagram of an application environment of the data processing method provided by an embodiment of the present application is shown. As an implementation method, the data processing method provided by the embodiment of the present application can be applied to an electronic device. The electronic device can be Figure 1The server 110 shown in FIG. 1 can be connected to the terminal device 120 via a network. The network is used to provide a medium for a communication link between the server 110 and the terminal device 120. The network can include various connection types, such as wired communication links, wireless communication links, etc., which are not limited in the embodiments of the present application. Alternatively, in other embodiments, the electronic device can also be a smartphone, a laptop computer, etc.

[0034] It should be understood that Figure 1 The server 110, network, and terminal device 120 are merely illustrative. Any number of servers, networks, and terminal devices may be provided as needed. For example, the server 110 may be a physical server or a server cluster consisting of multiple servers, and the terminal device 120 may be a mobile phone, tablet, desktop computer, laptop computer, or the like. It will be appreciated that embodiments of the present application may also allow multiple terminal devices 120 to access the server 110 simultaneously.

[0035] In some embodiments, the terminal device 120 may be installed with a client for any service, and the client may send a data operation request (e.g., add, delete, modify, query, etc.) for a storage system (e.g., a distributed file system) to the server 110. After the server 110 receives the data operation request from the client, it may process the data operation request using the data processing method of the embodiment of the present application. The storage system may be deployed in the server 110 or in other servers.

[0036] The data processing method of the embodiment of the present application can be used to process the stored data in any storage system. For example, an enterprise's storage system stores a large amount of business data. In order to facilitate the processing and analysis of this business data, the enterprise's data center (data processing and analysis platform) can adopt the data processing method of the embodiment of the present application.

[0037] It should be noted that the above application scenarios are only provided to facilitate understanding of the spirit and principles of the present application, and the implementation of the present application is not limited in this respect. The data processing method provided by the exemplary implementation of the present application is described below with reference to the accompanying drawings in conjunction with the above application scenarios.

[0038] See Figure 2 FIG. 1 is a flowchart of an implementation of a data processing method provided in an embodiment of the present application. Taking a server as an execution subject as an example, the specific implementation process of the method includes the following S201-S204:

[0039] S201: Respond to a data operation request for a storage system and obtain operation parameter information from the data operation request.

[0040] The storage system may be a distributed file system, such as HDFS. The storage system may include multiple storage directories, each storage directory may store multiple files, and each file may include corresponding business data.

[0041] In some embodiments, a server may receive a data operation request from a client for a storage system. The data operation request may include data operation instructions and operation parameter information. Data operation instructions include, but are not limited to, instructions for adding data, deleting data, modifying data, and querying data. The operation parameter information may be provided by the client based on preset operation parameters. Preset operation parameters are the operation parameters required to perform the data operation and can be pre-set by the server and the client.

[0042] When a user needs to perform data operations on a target file in a storage system, a data operation request for the target file can be triggered in the client, and the data operation request carries a data operation instruction; the client responds to the data operation request and displays a preset type of operation parameter that needs to be filled in. The preset type of operation parameter is field information pre-set by the server, and the field content (i.e., the operation parameter value) of the field information is entered by the client. The preset type of operation parameter can be field information configured as needed, for example, it can be configured according to storage processing requirements. The user can enter the operation parameter value (i.e., field content) in the client according to the preset type of operation parameter. The client uses the preset type of operation parameter and the operation parameter value entered by the user as operation parameter information, and sends the operation parameter information and the above-mentioned data operation instruction to the server in the data operation request.

[0043] In some embodiments, after receiving a data operation request for a storage system, the server obtains a preset type of operation parameter based on the data operation request, and then triggers the client to display the preset type of operation parameter. The user can enter the operation parameter value in the client, and the client sends the operation parameter information to the server.

[0044] In a specific implementation, the server may provide different interfaces to the client for different data operation instructions. For different interfaces, the pre-typed operation parameters may be the same or different. The client may display the pre-typed operation parameters on the client according to the pre-typed operation parameters provided by the interface requirements.

[0045] S202: If the acquired operation parameter information includes operation parameters required for performing data operations on the target file, the operation parameter information is saved in metadata of the target file.

[0046] The operation parameter information may include one or more items of information, each item representing a type of operation parameter and a corresponding operation parameter value. After obtaining the operation parameter information from the data operation request, it is determined whether the operation parameter information includes a preset type of operation parameter. If so, the preset type of operation parameter value is verified. If the verification passes, it is determined that the operation parameter information includes the parameters required to perform the data operation on the target file.

[0047] In some embodiments, when determining whether the operation parameter information includes the operation parameters required to perform the data operation on the target file in S202 above, the following steps may be performed:

[0048] If the obtained operation parameter information contains operation parameters of a preset type, the operation parameter value of the preset type is verified; if the verification passes, it is determined that the operation parameter information contains the operation parameters required to perform data operations on the target file, and the operation parameter information is saved in the metadata of the target file.

[0049] The preset operating parameters may include some or all of the following:

[0050] The storage node of the storage system where the file is located; the file name; the storage directory to which the file belongs; the file structure type; the file creator; the file operator; the group to which the file belongs; the file life cycle; the file life cycle calculation method; and the file recycling method.

[0051] Among them, storage nodes can be understood as server nodes in a distributed file system; file structure types specifically include structured types, semi-structured types, and unstructured types; file creators and file operators can have work numbers, names, departments, telephone numbers, positions, and job titles; the group to which the file belongs can be, for example, the department to which the file belongs, the team to which the file belongs, etc.

[0052] The file life cycle refers to the storage duration of a file, which can be in seconds, minutes, hours, days, months, or years, and the storage duration can also be permanent. The file life cycle calculation method refers to the time from which the file storage duration is calculated, for example, from the file creation time or the file update time. File recovery methods include physical deletion, logical deletion, and backup deletion. Physical deletion refers to directly deleting a file. Logical deletion refers to setting the file to be unusable instead of directly deleting the file. Backup deletion refers to backing up the file to be deleted before deleting it, for example, transferring the file to another storage system (such as a disk, CD, etc.) to facilitate recovery after accidental deletion.

[0053] The above verification of the preset type of operation parameter value may include the following two situations:

[0054] In the first case, the first type of operation parameter value in the preset type is compared with the pre-saved first type operation parameter value set. If the first type of operation parameter value is in the operation parameter value set, it is determined that the first type of operation parameter value verification has passed.

[0055] The first type of operation parameters in the preset type may be operation parameters with reference standards, and the first type of operation parameter values ​​input by the user must comply with the first type of reference standards. For example, the first type of operation parameters include: the storage node of the storage system where the file is located, the file name, the storage directory to which the file belongs, the file structure type, the file creator, the file operator, the group to which the file belongs, etc.

[0056] For example, when the first type of operation parameter includes a file name, the file name Z1 of the target file in the operation parameter information can be compared with the pre-saved file names to determine whether the file name Z1 exists in the file names. If so, the file name Z1 verification passes.

[0057] For another example, when the first type of operation parameter includes the storage directory to which the file belongs, the storage directory Y1 to which the target file in the operation parameter information belongs can be compared with the storage directory Y2 to which the target file belongs that was saved in advance. If they are consistent, the storage directory Y1 passes the verification.

[0058] In the second case, it is determined whether the second type of operation parameter value in the preset type meets the parameter value input rule of the second type of operation parameter. If so, it is determined that the second type of operation parameter value verification has passed.

[0059] The second type of operation parameter in the preset type may be an operation parameter with parameter value input rules, and the user may configure the second type of operation parameter value according to the parameter value input rules. For example, the second type of operation parameter may include: file lifecycle, file lifecycle calculation method, file recovery method, etc.; wherein, the parameter value input rules for the file lifecycle may include: weeks, days, hours, etc.; the parameter value input rules for the file lifecycle calculation method may include: calculation from the file creation time, calculation from the file update time, etc.; the parameter value input rules for the file recovery method may include: physical deletion, logical deletion, backup deletion, etc.

[0060] For example, when the second type of operation parameter includes a file life cycle, assuming that the file life cycle of the target file in the operation parameter information is 10 days, the parameter value input rule of the file life cycle includes the number of days, and there is a file with a file life cycle of 10 days in the storage system, so the file life cycle verification passes.

[0061] It should be noted that the server may pre-store metadata for the target file and update the metadata for the target file each time the data processing method of this embodiment is executed. That is, when operation parameter information is obtained in the current data operation request, if the operation parameter information includes the operation parameters required to perform the data operation on the target file, the obtained operation parameter information is saved in the metadata of the target file.

[0062] In an embodiment of the present application, if the operation parameter information obtained by the server includes the operation parameters required to perform data operations on the target file, the operation parameter values ​​are saved in the metadata of the target file to ensure that the operation parameters required to perform data operations on the target file can be collected.

[0063] S203: Perform data operation on the target file based on the data operation request to obtain updated file attribute content.

[0064] In one embodiment, when the storage system and the execution subject of the embodiment of the present application are deployed in the same server, the execution subject can directly perform data operations on the target file in the storage system based on the data operation instructions included in the data operation request.

[0065] In another embodiment, when the storage system and the execution subject of the embodiment of the present application are deployed in different servers, the execution subject can send the above-mentioned data operation request to the storage system so that the storage system responds to the data operation request and performs data operations on the target file.

[0066] At this time, performing data operation on the target file based on the data operation request in S203 to obtain updated file attribute content of the target file may include the following steps:

[0067] The server sends a data operation request to the storage system, so that the storage system performs a data operation on the target file in response to the data operation request; then, the server receives updated file attribute content of the target file returned by the storage system.

[0068] Among them, the storage system can determine that a data operation needs to be performed on the target file based on the operation parameter information in the data operation request, and determine the storage node of the storage system where the target file is located and the storage directory to which the target file belongs. Then, based on the data operation instruction in the data operation request, the data operation is performed on the target file. When the data operation instruction is a file update operation, the corresponding data can be added to the created target file, or the corresponding data can be deleted from the created target file, or the target file can be deleted, or the corresponding data can be modified in the created target file; when the data operation instruction is a file creation operation, the target file can be created or a subfile can be created in the target file; when the data operation instruction is a file query operation, the corresponding data can be queried in the created target file.

[0069] After performing data operations on the target file, updated file attribute content of the target file can be obtained. When different data operations are performed on the target file, the updated file attribute content obtained can be the same or different.

[0070] In some embodiments, performing a data operation on the target file based on the data operation request in S203 to obtain updated file attribute content of the target file may include the following situations:

[0071] Case 1: Based on the data operation request, a file update operation is performed on the target file to obtain updated first file attribute content of the target file.

[0072] The file update operation includes adding data, deleting data, or modifying data in the target file. The first file attribute content may include some or all of the following: file update time, file physical size, file update times, number of structured data records, and structured data field information.

[0073] Case 2: Based on the data operation request, a file creation operation is performed on the target file to obtain updated second file attribute content of the target file.

[0074] The second file attribute content may include part or all of the following: file creation time, file physical size, number of structured data records, and structured data field information.

[0075] Case 3: Based on the data operation request, a file query operation is performed on the target file to obtain updated third file attribute content of the target file.

[0076] The third file attribute content may include one or both of the file access time and the file access count.

[0077] It is understood that the file attribute content may include some or all of the following:

[0078] File creation time; file update time; file access time; file physical size; number of structured data records; structured data field information; file access times; file update times.

[0079] The number of structured data records refers to the number of structured data items recorded in a structured file. By recording the number of structured data records, the number of structured data items recorded in the structured file can be determined without opening the structured file. The structured data field information refers to the data type of the structured data recorded in the structured file.

[0080] In some embodiments, during the execution of the above-mentioned S203 process, before performing corresponding data operations on the target file in the storage system, the operator information of the target file can also be obtained from the data operation request; based on the operator information, it is determined whether the corresponding operator has the operation permission for the target file; if the operator has the operation permission, the corresponding data operation is performed on the target file, thereby preventing irrelevant users from arbitrarily operating the target file and ensuring the data security of the target file.

[0081] Among them, each file in each storage directory in the storage system can be operated by a person with corresponding operation authority, and the specific operation authority can be set as needed.

[0082] S204: Based on the updated file attribute content, update the file attribute content in the metadata of the target file.

[0083] Each time the server executes the data processing method of this embodiment, it updates the file attribute content in the metadata of the target file. That is, when the updated file attribute content is obtained in this data operation request, the file attribute content in the metadata of the target file is updated. Specifically, the following situations may occur:

[0084] The first case: the file attribute content in the metadata of the target file is replaced with the updated file attribute content.

[0085] For example, the updated file attribute content includes the number of file accesses F2, the file physical size S2, and the number of structured data records P2 after this data operation. Then, the number of file accesses F1 in the metadata of the target file is updated to F2, the file physical size S1 in the metadata of the target file is updated to S2, and the number of structured data records P1 in the metadata of the target file is updated to P2.

[0086] The second case: save the updated file attribute content to the metadata of the target file.

[0087] For example, the file attribute content includes the file creation time T1 and structured data field information A after this data operation. The file creation time T1 is saved under the file creation time in the metadata of the target file, and the structured data field information A is saved under the structured data field information in the metadata of the target file.

[0088] The third case: a portion of the updated file attribute content is saved in the metadata of the target file, and a portion of the file attribute content in the metadata of the target file is replaced with another portion of the updated file attribute content.

[0089] For example, the updated file attribute content includes the file access count F2, file physical size S2, and file access time T2 after this data operation, then the file access time T2 is saved to the metadata of the target file, and the file access count F1 in the metadata of the target file is updated to F2, and the file physical size S1 in the metadata of the target file is updated to S2.

[0090] Specifically, the file attribute content (partial or complete file attribute content) stored in the metadata of the target file can be associated with the operation parameter information. For example, the operation parameter information includes the file operator A of this data operation, and the file attribute content includes the file update time T of this data operation. After the operation parameter information and the file attribute content are associated and stored, it can be known later that A updated the target file at time T.

[0091] In an embodiment of the present application, before performing a data operation on a target file, the operation parameters required for performing the data operation on the target file can be collected and saved in the metadata of the target file; after performing the data operation on the target file, the updated file attribute content of the target file can be obtained, and based on the updated file attribute content, the file attribute content in the metadata of the target file is updated. In this way, the metadata of the target file not only includes the operation parameters required for performing the data operation on the target file, but also includes the updated file attribute content obtained after performing the data operation. Therefore, the metadata of the target file is relatively complete. Based on the metadata of the target file, the target file can be conveniently stored and processed, avoiding errors or problems that cannot be processed when the data of the target file is subsequently stored and processed, thereby improving the data processing performance of the storage system.

[0092] In some embodiments, when obtaining operation parameter information from the data operation request in the above S201, if the operation parameter information is not obtained, a request failure information may be returned to the client; wherein the request failure information is used to indicate that the data operation request lacks operation parameter information.

[0093] Based on the request failure information, the client can re-obtain the operation parameter information and resend the data operation request containing the operation parameter information to the server. The server can judge the re-sent operation parameter information so as to obtain the operation parameters required to perform the data operation.

[0094] In other embodiments, when obtaining operation parameter information from the data operation request in the above S201, if the obtained operation parameter information lacks any operation parameter required to perform data operations on the target file, a request failure information can be returned to the client; wherein, the request failure information is used to indicate any operation parameter missing in the data operation request.

[0095] Among them, any operating parameter can be one or more types of operating parameters among the above preset types.

[0096] The client can display the request failure information, so that the operator can re-fill in the missing operation parameters based on the request failure information. Then, the client can send a data operation request containing the missing operation parameters to the server again, so that the server can obtain the operation parameters required to perform the data operation.

[0097] In other embodiments, when obtaining operation parameter information from the data operation request in the above S201, if any operation parameter required to perform data operation on the target file in the obtained operation parameter information fails to be verified, a request failure information can be returned to the client; wherein the request failure information is used to indicate that any operation parameter fails to be verified.

[0098] Among them, any operating parameter can be one or more types of operating parameters among the above preset types.

[0099] The client can display the request failure information, so that the operator can correct any operation parameter based on the request failure information. Then, the client can send a data operation request containing any corrected operation parameter to the server again. The server can verify any corrected operation parameter again to ensure the correctness of the collected operation parameter information.

[0100] In an embodiment of the present application, the operation parameter information in the collected data operation request is judged to determine whether the collected operation parameter information contains the operation parameters required to perform data operations on the target file, that is, to determine whether the operation parameter information is complete, and the operation parameter information is also verified, that is, to determine whether the operation parameter information is correct. If it is incomplete or incorrect, a request failure message can be returned to enable the operator to supplement the missing or correct the incorrect operation parameters, thereby ensuring the integrity and correctness of the operation parameter information.

[0101] In some embodiments, considering the large amount of data stored in a storage system, to facilitate management of the storage system, each storage directory in the storage system can be bound to a corresponding metadata collector. In other words, each storage directory can be managed and data operations performed by a metadata collector. Multiple metadata collectors can be deployed on a server to separately manage each storage directory in the storage system.

[0102] During the execution of S202, the target storage directory to which the target file belongs can be obtained from the operation parameter information, the target metadata collector bound to the target storage directory can be determined, and the operation parameter information can be saved to the metadata of the target file through the target metadata collector.

[0103] During the execution of S203 , when performing a data operation on the target file based on the data operation request, the target metadata collector may send the data operation request to the storage system so that the storage system performs the data operation on the target file.

[0104] Furthermore, the above S204 may also be executed by a target metadata collector bound to the target storage directory to which the target file belongs, that is, the target metadata collector updates the file attribute content in the metadata of the target file based on the updated file attribute content.

[0105] In the embodiment of the present application, the server can build a bridge between the client and the storage system through the metadata collector. Figure 3 As shown, there are multiple metadata collectors, each bound to a storage directory on the storage system. For example, collector 1 (i.e., metadata collector 1) manages the / tmp directory, collector 2 (i.e., metadata collector 2) manages the / data directory, and collector 3 (i.e., metadata collector 3) manages the / dev directory. Data operation requests initiated by clients (e.g., client 1, client 2, client 3, etc.) targeting a storage directory on the storage system are processed by the metadata collector bound to that storage directory.

[0106] Among them, multiple storage directories of the storage system can be defined according to needs. For example, the functions and uses represented by multiple storage directories can be defined according to the enterprise's unified architecture, such as: / tmp stores all temporary data; / data stores all production data; / user stores all user private data, etc.

[0107] It is understandable that all data operation requests initiated by the client to the storage system can be processed by the metadata collector. Specifically, after receiving the data operation request from the client, the server obtains the operation parameter information from the data operation request, judges and verifies the obtained operation parameter information, and when it is determined that the operation parameter information contains the operation parameters required to perform the data operation on the target file, obtains the target storage directory to which the target file belongs from the operation parameter information, determines the target metadata collector bound to the target storage directory, and saves the operation parameter information to the metadata of the target file through the target metadata collector, and then performs the data operation on the target file based on the data operation request.

[0108] Specifically, the target storage directory's operation permissions can be granted to the bound target metadata collector. After the server saves the operation parameter information to the target file's metadata through the target metadata collector, it can also determine whether the operator of the data operation has the operation permission. If the operator has the operation permission, the target metadata collector sends a data operation request to the storage system, allowing the storage system to perform the data operation on the target file. This prevents users without operation permissions from arbitrarily creating and modifying data in the storage system.

[0109] In this embodiment of the present application, a metadata collector is used to bridge the gap between the client and the storage system, allowing all data operation requests from the client to be proxied by the metadata collector. The server obtains operation parameter information from each data operation request from the client. Upon determining that the obtained operation parameter information contains the operation parameters required to perform a data operation on a target file, the server saves the operation parameter information to the target file's metadata through the target metadata collector. The server then performs a data operation on the target file based on the data operation request, obtaining updated file attribute content. Based on this updated file attribute content, the server then updates the file attribute content in the target file's metadata.

[0110] The following takes the storage system as HDFS as an example, combined with Figure 4 The specific interaction process of the data processing method of the embodiment of the present application is introduced.

[0111] like Figure 4 As shown, the specific interaction process of the data processing method of the embodiment of the present application is as follows:

[0112] S401: The client sends a data operation request for a storage system to the server.

[0113] S402: The server obtains operation parameter information from the data operation request, and determines that the obtained operation parameter information includes operation parameters required to perform the data operation on the target file.

[0114] In addition, when the operation parameter information is not obtained, or the obtained operation parameter information lacks any operation parameter required to perform data operations on the target file, or any operation parameter required to perform data operations on the target file in the obtained operation parameter information fails to pass verification, the server can return a request failure message to the client.

[0115] S403: The server obtains the target storage directory to which the target file belongs from the operation parameter information, determines the target metadata collector bound to the target storage directory, and saves the operation parameter information to the metadata of the target file through the target metadata collector.

[0116] S404: The server sends a data operation request to HDFS through the target metadata collector.

[0117] S405. HDFS performs data operations on the target file according to the data operation request.

[0118] S406. After completing the data operation on the target file, HDFS sends the updated file attribute content of the target file to the server.

[0119] Specifically, the target metadata collector obtains the updated file attribute content returned by HDFS and updates the metadata of the target file.

[0120] S407: The server updates the file attribute content in the metadata of the target file based on the updated file attribute content through the target metadata collector.

[0121] S408. HDFS sends the data operation result to the client.

[0122] The following takes the HDFS storage system as an example to introduce the configuration process of the metadata collector for collecting information (including the operation parameter information and updated file attribute content of the above embodiment).

[0123] The collected information can be configured according to storage and processing requirements. For example, the collected information may include some or all of the following:

[0124] Storage node: the name of the HDFS server.

[0125] File name: The physical name of the file, for example: / user / data / test.txt.

[0126] File storage directory: bound to the corresponding metadata collector to distinguish the data types under the storage directory.

[0127] File structure type: structured, semi-structured, unstructured.

[0128] File creator: records the file creator’s information, such as part or all of the employee number, name, department, phone number, position, job title, and email address.

[0129] File operator: Record the file operator’s information, such as part or all of the employee number, name, department, phone number, position, job title, and email address.

[0130] File Department: records the department of the file creator.

[0131] File lifecycle: can be configured to seconds, minutes, hours, days, months, years, forever, etc.

[0132] File lifecycle calculation method: You can configure the calculation of file generation cycle based on creation time, update time, access time, etc.

[0133] For example, the file life cycle is calculated based on the creation time. Assuming that the file generation cycle is 5 days, determine whether the current time minus the creation time is greater than 5 days. If it is greater than 5 days, the file is automatically cleaned up (file cleaning is determined by the file recycling method).

[0134] File recovery methods: physical deletion, logical deletion (not true deletion, but setting it to be unavailable to users), backup deletion (transferring data to other storage media such as CDs, hard drives, etc.), etc.

[0135] File creation time: The time when the file is created. After the file is created, HDFS needs to return the file creation time.

[0136] File update time: The time when a file is opened, modified, and saved. When the file is updated, HDFS needs to return the file update time.

[0137] File access time: If the file already exists, this refers to the time it was read. After the file is read, HDFS needs to return the file access time.

[0138] File physical size: The physical space occupied by the file. After the file is updated, HDFS needs to return the file's physical size.

[0139] Number of structured data records: Record condition (file structure type = "structured").

[0140] By recording the number of structured data records, the number of structured data records in the structured file can be known without opening the structured file.

[0141] Structured data field information: record conditions (file structure type = "structured").

[0142] By recording the structured data field information, the data type of the structured data recorded in the structured file can be known.

[0143] File access count: Each time a file is read is counted as 1.

[0144] Number of file updates: Each time a file is opened, data is modified and successfully saved, it counts as one time.

[0145] When configuring collection information, you can configure default values ​​for HDFS directories. For example, if the / tmp directory is used to store temporary data, you can set the default file lifecycle to 1 day, calculate the file lifecycle based on creation time, and use physical deletion as the file recovery method. This allows you to use the default values ​​for the HDFS directory if information such as the file lifecycle, file lifecycle calculation method, and file recovery method is not configured.

[0146] In addition, when configuring the collection information, you can check whether necessary storage attribute information is missed. Necessary storage attribute information can be set as needed, such as various information of the operator, including work number, name, department, telephone number, position, job title, etc. It can also include various information of the file, including storage node, storage directory, file name, file structure type, file physical size, file life cycle, file life cycle calculation method, file recovery method, etc.

[0147] For storage attribute information that includes enumeration, you can check whether the enumeration is complete, such as file structure type (structured, semi-structured, unstructured), file lifecycle (seconds, minutes, hours, days, months, years, permanent), and file recovery method (physical deletion, logical deletion, backup deletion).

[0148] After obtaining the metadata of the target file based on the above-mentioned embodiments of the present application, the target file can be stored and processed based on the metadata of the target file.

[0149] The metadata of the target file includes: the operation parameter information obtained according to each data operation request, and the updated file attribute content obtained by performing each data operation. Specifically, the target file can be stored and processed based on the operation parameter information and file attribute content. The embodiment of the present application does not limit the storage and processing method.

[0150] The following embodiment provides an exemplary introduction to the storage processing process of the target file.

[0151] In some embodiments, the metadata of the target file may include the following operating parameters: file lifecycle, file lifecycle calculation method, and file recycling method. The storage processing of the target file based on the metadata of the target file may include:

[0152] Based on the file life cycle calculation method of the target file, the file life cycle start time of the target file is determined; when the length between the current time and the file generation cycle start time is equal to the file life cycle of the target file, the target file is recycled based on the file recycling method of the target file.

[0153] Among them, the file life cycle calculation method refers to the time from which the storage duration of the file is calculated, for example: starting from the file update time or the file creation time. When calculating from the file update time, the start time of the file life cycle of the target file is the most recent file update time. When it is determined that the time between the current time and the most recent file update time is equal to the file life cycle, it is considered that the current time has reached the recycling time of the file, and the target file can be recycled according to the file recycling method.

[0154] Optionally, the target file's file recycling method can be physical deletion, logical deletion, or backup deletion. The target file is recycled based on the target file's file recycling method, which can include the following three situations:

[0155] In the first case, if the file recycling method is physical deletion, the target file is deleted from the storage system;

[0156] In the second case, if the file recovery method is logical deletion, the target file will be set to unusable;

[0157] In the third case, if the file recovery method is backup deletion, the target file is backed up to another storage system outside the storage system and then deleted.

[0158] For example, assume that the file lifecycle of a.txt is configured to be 5 days, the file lifecycle calculation method is based on the file update time, and the file recovery method is physical deletion. For example, if file a.txt is updated at 2023-09-15 8:00:00, at 2023-09-20 9:00:00, the current time minus the file update time, if the result is greater than 5 days, file a.txt will be automatically cleaned up. If the file recovery method is physical deletion, file a.txt can be directly deleted. If the file recovery method is backup deletion, file a.txt can be transferred to other storage media (such as disks, CDs, etc.).

[0159] In the above-mentioned embodiments of the present application, when the target file is a temporary file, i.e., its file lifecycle is limited, automatic cleanup can be achieved based on the file lifecycle, file lifecycle calculation method, and file recycling method in the target file's metadata, thereby preventing the target file from occupying storage space for a long time. Furthermore, the embodiments of the present application can also configure the file recycling method of the target file according to actual needs, specifically, one of physical deletion, logical deletion, and backup deletion, to meet different automatic cleanup requirements.

[0160] In other embodiments, the metadata of the target file may include the following file attributes: file structure type, file physical size, and structured data field information; wherein the structured data field information is used to describe the type of structured data stored in the corresponding file; then, when performing storage processing on the target file based on the metadata of the target file, the following operations may be performed:

[0161] For the target file and other files in the target storage directory to which the target file belongs, if the target file and other files meet the following conditions, the target file and other files will be merged:

[0162] (1) The file structure type of the target file and other files is structured type.

[0163] Structured files store structured data. For example, the structured data is an employee table, where the first column is the employee number, the second column is the name, and the third column is the position.

[0164] (2) The structured data field information of the target file and other files is the same.

[0165] The structured data field information may represent: the data type of the structured data stored in the structured type file, for example, in the employee table mentioned above, the first column is the employee number, the second column is the name, the third column is the position, and so on.

[0166] (3) The physical sizes of the target file and other files meet the merging conditions.

[0167] Specifically, the other files may be one or more files, and the merging condition may be that the sum of the physical sizes of the target file and the other files does not exceed a set file size, which may be set according to actual conditions.

[0168] It can be understood that when the target file and other files are both structured files, the information of the stored structured data is the same, and the sum of the physical sizes of the target file and other files does not exceed the set file size, the target file and other files can be automatically merged into one file, thereby realizing automatic management of each file in the storage directory.

[0169] For example, the target file is the file / data / a.csv in the / data directory, and the / data directory also includes the file / data / b.csv. The file structure types of these two files are both structured types, and the stored structured data are all student tables, with the first column being the student ID, the second column being the name, etc., and the sum of the physical sizes of the two files does not exceed the maximum file size. Therefore, / data / a.csv and / data / b.csv are merged into one file / data / c.csv.

[0170] For example, the data stored in / data / a.csv is shown in Table 1:

[0171] Table 1

[0172] Name age gender Xiao Wang 20 female

[0173] The data stored in / data / b.csv is shown in Table 2:

[0174] Table 2

[0175] Name age gender Xiao Zhang 21 male

[0176] The data stored in the merged / data / c.csv is shown in Table 3:

[0177] Table 3

[0178] Name age gender Xiao Wang 20 female Xiao Zhang 21 male

[0179] In the above-mentioned implementation of the present application, file merging can be automatically achieved through the file structure type, file physical size, structured data field information, etc. in the metadata of the target file, thereby avoiding file confusion under the storage directory and facilitating the storage and processing of data in each file under the storage directory.

[0180] Furthermore, the management operations performed on the target file based on the target file's metadata are not limited to the aforementioned automatic file cleanup and file merging processes. For example, if the target file is determined to be a file that has not been accessed or updated for a long time (cold data), the target file can be separated from other files in the storage system by performing a cold-hot separation. The target file can then be automatically transferred to another storage system (e.g., a storage medium, including an optical disc, hard disk, etc.) for storage, separate from frequently accessed or updated files (hot data).

[0181] In another embodiment, the file attributes of the target file's metadata may include: file access count and file update count. If the target file's file access count within a recent set time period is less than a first set count, and the file update count is less than a second set count, the target file may be considered cold data and automatically transferred to another storage system for storage. The first set count and the second set count may be set as needed and may be the same or different.

[0182] In an embodiment of the present application, when the target file is determined to be a file that has not been accessed or updated for a long time through the number of file accesses and file updates in the metadata of the target file, the target file can be automatically transferred to another storage system to facilitate the hot and cold separation of the files in the storage system.

[0183] Based on the above embodiments of the present application, metadata of each file contained in each storage directory in the storage system can be obtained, and storage space of the storage system can be allocated and managed based on the metadata of each file.

[0184] In some embodiments, the metadata of each file in the storage system may include at least: the file creator, the file operator, the group to which the file belongs, and the physical size of the file. Based on these metadata of each file, the group or user to which each file belongs and the storage space occupied by each file can be determined, so that the storage space of the group or user can be allocated.

[0185] For example, each group or user can apply for storage space of appropriate size (i.e., planned storage space) according to their own business situation. When the ratio of the actual storage space size of a group or user to the planned storage space size exceeds a set threshold (e.g., 80%), the group or user's management personnel can be notified to expand or clean up the storage space to avoid business function abnormalities due to insufficient storage space.

[0186] Based on the metadata of each file contained in each storage directory in the storage system, in addition to performing storage processing on the files in the storage system, the storage changes in the storage system can also be analyzed and visualized. The following embodiment provides an exemplary introduction to the storage change analysis and visualization process.

[0187] In some embodiments, a multi-dimensional analysis of the storage system's storage status can be performed based on the metadata of each file contained in each storage directory in the storage system. Optionally, metadata of each file stored in each storage directory is obtained for each storage directory in the storage system; based on the metadata of each file stored in each storage directory, statistics are collected on the storage system's storage change information from at least one statistical dimension; and the obtained storage change information from at least one statistical dimension is presented in the form of a report.

[0188] Among them, at least one statistical dimension can be set according to actual conditions, and specifically, statistics can be collected on measurable information in metadata. The following takes an enterprise's storage system as an example to exemplify at least one statistical dimension.

[0189] For example, for measurable information in metadata, such as the physical size of a file, the changing trend of the storage space occupied by an individual or department's files in the storage system can be counted according to the individual dimension or department dimension. Specifically, the storage space occupied at historical moments and the current moment can be counted, so that the changing trend of the occupied storage space can be depicted in the form of a report.

[0190] For example, for measurable information in metadata, such as the number of file accesses, we can analyze which files are most frequently accessed by individuals or departments based on the individual or departmental dimensions. We can also analyze which files are most frequently accessed based on the file name dimension, thereby displaying the access popularity of each file in the storage system in the form of a report.

[0191] For example, based on measurable information in metadata, such as file lifecycle and file physical size, the amount of storage space that can be automatically cleaned can be estimated, and the changing trend of the storage space automatically cleaned at different times can be displayed in the form of a report.

[0192] For another example, for measurable information in metadata, such as the physical size of files, we can count the number of files in different physical size ranges, such as the number of files with physical sizes between 1-10mb, 10mb-20mb, etc. Then, we can analyze the proportion of small files for different people or departments by individual or departmental dimensions, and present the proportion of small files for different people or departments in the form of a report.

[0193] Based on the above examples, it can be seen that at least one statistical dimension includes but is not limited to: storage space occupied by an individual or department, popularity of file access, storage space automatically cleared, proportion of small files of an individual or department, etc.

[0194] In an embodiment of the present application, based on the metadata of each file stored in each storage directory of the storage system, the storage system's storage changes can be statistically analyzed from multiple statistical dimensions and visualized, allowing users to intuitively understand the storage system's data storage status. Furthermore, for enterprises, storage costs can be allocated to individuals or departments of the enterprise based on the storage changes in multiple statistical dimensions of the storage system. The storage changes in multiple statistical dimensions can be displayed in the form of reports, allowing individuals or departments to better utilize the storage space of the storage system.

[0195] In the following embodiment, the storage system is HDFS as an example to introduce the data processing system for implementing the data processing method of the embodiment of the present application.

[0196] like Figure 5 As shown, the data processing system of the embodiment of the present application includes: metadata collection and classification module 51, data management module 52, and storage report analysis module 53. These modules can be deployed in a server. These modules are introduced below.

[0197] 1. Role and Function of Metadata Collection and Classification Module

[0198] The metadata collection and classification module can receive data operation requests for HDFS and process the data operation requests, including:

[0199] Obtain operation parameter information from the data operation request. If the obtained operation parameter information contains the parameters required to perform data operations on the target file, save the operation parameter information to the metadata of the target file; based on the data operation instructions contained in the data operation request, perform data operations on the target file in HDFS to obtain updated file attribute content; based on the updated file attribute content, update the file attribute content in the metadata of the target file.

[0200] Specifically, the metadata collection and classification module has the following functions:

[0201] 1. Build a bridge between the client and HDFS through the metadata collector.

[0202] 2. All data operation requests from the client are processed by the metadata collector.

[0203] The data operation request initiated by the client is used to perform data operations on the target file in HDFS. The data operations specifically include adding data, deleting data, changing data, querying data, etc.

[0204] 3. After receiving the data operation request from the client, the obtained operation parameter information will be judged and classified.

[0205] Specifically, after obtaining the operation parameter information from the data operation request, the HDFS directory targeted by the data operation request (i.e., the target storage directory to which the target file belongs) is determined, and the various types of parameters contained in the operation parameter information are classified, and then the data operation request is processed based on the target metadata collector bound to the HDFS directory.

[0206] 4. The metadata collector performs data operations in HDFS according to the client's data operation request.

[0207] Specifically, the metadata collector may send a data operation request to the HDFS, so that the HDFS performs the data operation.

[0208] 2. Role and Function of Data Processing Module

[0209] 1. File lifecycle management: Automatically clean up target files based on information such as the file lifecycle, file lifecycle calculation method, and file recycling method in the target file's metadata.

[0210] 2. File merging: The target file and other files can be merged based on the file structure type, file physical size, and structured data field information in the target file's metadata.

[0211] 3. Separate target files into cold and hot data. Based on the file access and update times in the target file metadata, target files that have not been accessed or updated for a long time (cold data) can be automatically transferred to other storage systems (such as storage media, including optical disks, hard disks, etc.) for storage.

[0212] 4. Allocate HDFS storage space. The metadata for each file in the storage system can include the file creator, file operator, group to which the file belongs, and the file's physical size. Based on this metadata, the group or user to which each file belongs and the storage space occupied by each file can be determined, allowing storage space to be allocated to the group or user.

[0213] 3. Role and Function of Storage Report Analysis Module

[0214] 1. Used to monitor HDFS storage changes according to different statistical dimensions.

[0215] Specifically, taking the enterprise's storage system as an example, different statistical dimensions include but are not limited to: storage space occupied by individuals or departments, popularity of file access, storage space automatically cleared, the proportion of small files of individuals or departments, etc.

[0216] 2. HDFS storage costs can be allocated to individuals or departments within the enterprise by analyzing storage changes across different statistical dimensions. Reports presenting storage changes across different statistical dimensions help individuals or departments better utilize HDFS storage space.

[0217] In the embodiment of the present application, the storage system has a clear classification in the storage structure, that is, it contains multiple storage directories, which will not cause data confusion. Each storage directory can be defined according to the unified architecture of the enterprise to avoid arbitrary creation by users. The user's operating authority is not directly bound to the storage system, but indirectly bound to the metadata collector. In this way, all user data operations must go through the metadata collector, and complete metadata can be collected to facilitate subsequent data processing and visual analysis. Only after obtaining complete metadata can more automated management functions be performed. For example, once some temporary data is stored in the storage system, if it is not actively cleaned up, it will take up space for a long time. The embodiment of the present application can define the storage period of the data (i.e., the file life cycle) before the data is written, and it will be automatically cleaned up after expiration.

[0218] In general, the above-mentioned scheme of the embodiment of the present application processes the client's data operation request through the metadata collector, avoids the operator's arbitrariness in the data operation of the storage system, realizes the integrity and configurability of the metadata of the files in the storage system, and realizes the storage processing and visual analysis of the files in the storage system.

[0219] Based on the same inventive concept, a data processing device is also provided in an embodiment of the present application. The principle of solving the problem by this device is similar to the method of the above embodiment. Therefore, the implementation of this device can refer to the implementation of the above method, and the repeated parts will not be repeated.

[0220] See Figure 6A As shown, a data processing device 60 provided in an embodiment of the present application includes:

[0221] An acquiring unit 61 is configured to respond to a data operation request for the storage system and acquire operation parameter information from the data operation request;

[0222] A saving unit 62 is configured to save the operation parameter information to metadata of the target file if the acquired operation parameter information includes operation parameters required for performing data operations on the target file;

[0223] A data operation unit 63 is configured to perform a data operation on the target file based on the data operation request to obtain updated file attribute content;

[0224] The updating unit 64 is configured to update the file attribute content in the metadata of the target file based on the updated file attribute content.

[0225] In the embodiment of the present application, since the metadata of the target file not only includes the operation parameters required to perform data operations on the target file, but also includes the updated file attribute content obtained after performing the data operations, the metadata of the target file is relatively complete. Based on the metadata of the target file, the target file can be conveniently stored and processed, avoiding errors or problems that cannot be processed during subsequent storage and processing of the target file data, thereby improving the data processing performance of the storage system.

[0226] In some optional implementations, the storage unit 62 is specifically configured to:

[0227] If the acquired operation parameter information includes an operation parameter of a preset type, verify the operation parameter value of the preset type;

[0228] If the operation parameter value of the preset type passes the inspection, it is determined that the operation parameter information includes the operation parameters required to perform data operations on the target file, and the operation parameter information is saved in the metadata of the target file.

[0229] In some optional implementations, when verifying the preset type of operating parameter value, the storage unit 62 is specifically configured to:

[0230] Comparing the first type of operation parameter value in the preset type with a pre-stored first type of operation parameter value set, and if the first type of operation parameter value is in the operation parameter value set, determining that the first type of operation parameter value verification has passed; or

[0231] Determine whether the second type of operation parameter value in the preset type meets the parameter value input rule of the second type of operation parameter; if so, determine that the second type of operation parameter value verification passes.

[0232] In some optional implementations, the data operation unit 63 is specifically configured to:

[0233] Perform a file update operation on the target file based on the data operation request to obtain updated first file attribute content of the target file; or

[0234] Performing a file creation operation on the target file based on the data operation request to obtain updated second file attribute content of the target file; or

[0235] A file query operation is performed on the target file based on the data operation request to obtain updated third file attribute content of the target file.

[0236] In some optional implementations, the data operation unit 63 is specifically configured to:

[0237] Sending a data operation request to the storage system, so that the storage system performs the data operation on the target file in response to the data operation request;

[0238] Receive updated file attribute content of the target file returned by the storage system.

[0239] In some optional embodiments, the device further comprises a return unit, configured to:

[0240] If any of the following conditions occurs, a request failure message will be returned to the client:

[0241] The operation parameter information was not obtained;

[0242] The obtained operation parameter information lacks any operation parameter required to perform data operations on the target file;

[0243] Any operation parameter required to perform data operations on the target file in the obtained operation parameter information fails to pass verification.

[0244] In some optional implementations, the storage system includes multiple storage directories, each storage directory is bound to a metadata collector;

[0245] The storage unit 62 is specifically used for:

[0246] Obtain the target storage directory to which the target file belongs from the obtained operation parameter information;

[0247] The target metadata collector bound to the target storage directory is determined, and the operation parameter information is saved in the metadata of the target file through the target metadata collector.

[0248] In some optional implementations, the metadata of the target file includes the following operating parameters: file life cycle, file life cycle calculation method, file recycling method; Figure 6B As shown, the device further includes a storage processing unit 65, which is used to:

[0249] Determine the start time of the target file's file life cycle based on the target file's file life cycle calculation method;

[0250] When the duration between the current time and the start time of the file generation cycle is equal to the file life cycle of the target file, the target file is recycled based on the file recycling method of the target file.

[0251] In some optional implementations, when recycling the target file based on the file recycling method of the target file, the storage processing unit 65 is specifically configured to:

[0252] If the file recovery method is physical deletion, the target file is deleted in the storage system;

[0253] If the file recovery method is logical deletion, the target file will be set to unusable;

[0254] If the file recycling method is backup deletion, the target file is backed up to a storage system other than the storage system and then deleted.

[0255] In some optional implementations, the metadata of the target file includes the following file attributes: file structure type, file physical size, and structured data field information; wherein the structured data field information is used to describe the type of structured data stored in the corresponding file; the storage processing unit 65 is further used to:

[0256] For the target file and other files in the target storage directory to which the target file belongs, if the target file and other files meet the following conditions, the target file and other files will be merged:

[0257] The file structure type of the target file and other files is structured type;

[0258] The structured data field information of the target file and other files is the same;

[0259] The physical sizes of the target file and other files meet the merging conditions.

[0260] In some optional embodiments, the apparatus further comprises a statistical analysis unit, configured to:

[0261] For multiple files stored in a storage system, obtaining metadata of the multiple files stored;

[0262] Based on metadata of multiple files, statistics are collected on storage change information of the storage system from at least one statistical dimension;

[0263] The storage change information of at least one statistical dimension obtained is presented in the form of a report.

[0264] In some optional implementations, based on the data operation content included in the data operation request, before performing the corresponding data operation on the target file in the storage system, the method further includes:

[0265] Obtain the operator information of the target file from the data operation request;

[0266] Based on the operator information, it is determined that the corresponding operator has the operation authority for the target file.

[0267] For the convenience of description, the above parts are divided into modules (or units) according to their functions and described separately. Of course, when implementing this application, the functions of each module (or unit) can be implemented in the same or multiple software or hardware.

[0268] After introducing the content retrieval method and apparatus according to an exemplary embodiment of the present application, an electronic device according to another exemplary embodiment of the present application will be introduced next.

[0269] Those skilled in the art will appreciate that various aspects of the present application can be implemented as systems, methods, or program products. Therefore, various aspects of the present application can be specifically implemented in the following forms: a complete hardware implementation, a complete software implementation (including firmware, microcode, etc.), or an implementation that combines hardware and software aspects, which may be collectively referred to herein as a "circuit," "module," or "system."

[0270] Based on the same inventive concept as the above method embodiment, an electronic device is also provided in the embodiment of the present application. In one embodiment, the electronic device may be a server, such as Figure 1 Server 120 is shown.

[0271] In some possible implementations, the electronic device according to the present application may include at least one processor and at least one memory. The memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of the data processing method according to various exemplary embodiments of the present application described above in this specification. For example, the processor may execute the following steps: Figure 2 Steps 201 to 204 are shown in FIG.

[0272] Refer to the following Figure 7 An electronic device 70 according to this embodiment of the present application will be described. Figure 7 The electronic device 70 shown is only an example and should not limit the functions and scope of use of the embodiments of the present application. Figure 7 As shown, the electronic device 70 is in the form of a general-purpose smart terminal (or Bluetooth headset). Components of the electronic device 70 may include, but are not limited to: at least one processor 71, at least one memory 72, and a bus 73 connecting different system components (including the memory 72 and the processor 71).

[0273] Bus 73 represents one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, a processor, or a local bus using any of a variety of bus architectures. Memory 72 may include readable media in the form of volatile memory, such as random access memory (RAM) 721 and / or cache memory 722, and may further include read-only memory (ROM) 723. Memory 72 may also include a program / utility 725 having a set (at least one) of program modules 724, such as, but not limited to, an operating system, one or more application programs, other program modules, and program data, each of which, or some combination thereof, may include an implementation of a network environment.

[0274] The electronic device 70 may also communicate with one or more external devices 74 (e.g., keyboards, pointing devices, etc.), and / or any device that enables the electronic device 70 to communicate with one or more other intelligent terminals (e.g., routers, modems, etc.). Such communication may be performed via an input / output (I / O) interface 75. Furthermore, the electronic device 70 may also communicate with one or more networks (e.g., local area networks (LANs), wide area networks (WANs), and / or public networks, such as the Internet) via a network adapter 76. As shown, the network adapter 76 communicates with other modules of the electronic device 70 via a bus 73. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device 70, including but not limited to microcode, device drivers, redundant processors, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0275] In some possible implementations, various aspects of the data processing method provided by the present application may also be implemented in the form of a program product, which includes a computer program. When the program product is run on a computer device, the computer program is used to cause the computer device to execute the steps of the image processing method according to various exemplary embodiments of the present application described above in this specification. For example, the processor may execute the following steps: Figure 2 Steps 201 to 204 are shown in FIG.

[0276] The program product may employ any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or component, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof.

[0277] The program product for time-domain noise processing according to an embodiment of the present application may be a portable compact disc read-only memory (CD-ROM) and include a computer program, and may be run on a smart terminal. However, the program product of the present application is not limited thereto. In this document, a readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0278] A readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying a readable computer program. This propagated data signal may take a variety of forms, including, but not limited to, electromagnetic signals, optical signals, or any suitable combination thereof. A readable signal medium may also be any readable medium other than a readable storage medium that can transmit, propagate, or transfer a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0279] The computer program embodied on the readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0280] The computer program for performing the operations of the present application can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The computer program can be executed entirely on the user electronic device, partially on the user electronic device, as a separate software package, partially on the user electronic device and partially on a remote electronic device, or entirely on a remote electronic device or server. In cases involving remote electronic devices, the remote electronic device can be connected to the user electronic device through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external electronic device (for example, using an Internet service provider to connect through the Internet).

[0281] It should be noted that although several units or subunits of the device are mentioned in the detailed description above, this division is merely exemplary and not mandatory. In fact, depending on the embodiment of the application, the features and functions of two or more units described above can be embodied in a single unit. Conversely, the features and functions of a single unit described above can be further divided and embodied by multiple units.

[0282] Furthermore, although the operations of the method of the present application are described in a particular order in the accompanying drawings, this does not require or imply that the operations must be performed in this particular order, or that all illustrated operations must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps.

[0283] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the present application. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0284] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0285] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0286] Obviously, those skilled in the art may make various changes and modifications to this application without departing from the spirit and scope of this application. Thus, if these modifications and variations of this application fall within the scope of the claims of this application and their equivalents, this application is intended to include these modifications and variations.

Claims

1. A data processing method, characterized in that: include: Responding to a data operation request for a storage system, and obtaining operation parameter information from the data operation request; If the acquired operation parameter information includes operation parameters required for performing data operations on the target file, the operation parameter information is saved in metadata of the target file; performing a data operation on the target file based on the data operation request to obtain updated file attribute content of the target file; Based on the updated file attribute content, the file attribute content in the metadata of the target file is updated.

2. The method according to claim 1, characterized in that If the acquired operation parameter information includes operation parameters required for performing data operations on the target file, saving the operation parameter information to metadata of the target file includes: If the acquired operation parameter information includes an operation parameter of a preset type, verifying the operation parameter value of the preset type; If the operation parameter value of the preset type passes the inspection, it is determined that the operation parameter information includes the operation parameters required to perform data operations on the target file, and the operation parameter information is saved in the metadata of the target file.

3. The method according to claim 2, characterized in that The verifying the operation parameter value of the preset type includes: Comparing the first type of operation parameter value in the preset type with a pre-saved set of operation parameter values ​​of the first type, and if the first type of operation parameter value is in the set of operation parameter values, determining that the first type of operation parameter value verification has passed; or Determine whether the second type of operation parameter value in the preset type meets the parameter value input rule of the second type of operation parameter; if so, determine that the second type of operation parameter value verification passes.

4. The method according to claim 1, wherein The performing of the data operation on the target file based on the data operation request to obtain updated file attribute content of the target file includes: Perform a file update operation on the target file based on the data operation request to obtain updated first file attribute content of the target file; or Performing a file creation operation on the target file based on the data operation request to obtain updated second file attribute content of the target file; or A file query operation is performed on the target file based on the data operation request to obtain updated third file attribute content of the target file.

5. The method according to claim 1 or 4, characterized in that The step of performing a data operation on the target file based on the data operation request to obtain updated file attribute content of the target file includes: Sending the data operation request to the storage system, so that the storage system performs the data operation on the target file in response to the data operation request; Receive updated file attribute content of the target file returned by the storage system.

6. The method according to any one of claims 1 to 4, characterized in that The method further comprises: If any of the following conditions occurs, a request failure message will be returned to the client that sent the data operation request: The operation parameter information is not obtained; The acquired operation parameter information lacks any operation parameter required for performing data operations on the target file; Any operation parameter required for performing data operations on the target file in the obtained operation parameter information fails to pass verification.

7. The method according to any one of claims 1 to 4, characterized in that The storage system includes multiple storage directories, each storage directory is bound to a metadata collector; Saving the operation parameter information to the metadata of the target file includes: Obtaining the target storage directory to which the target file belongs from the obtained operation parameter information; A target metadata collector bound to the target storage directory is determined, and the operation parameter information is saved into the metadata of the target file through the target metadata collector.

8. A data processing device, characterized in that: include: an acquiring unit, configured to respond to a data operation request for the storage system and acquire operation parameter information from the data operation request; a saving unit, configured to save the acquired operation parameter information into metadata of the target file if the acquired operation parameter information includes an operation parameter value required for performing a data operation on the target file; a data operation unit, configured to perform a data operation on the target file based on the data operation request, and obtain updated file attribute content of the target file; An updating unit is configured to update the file attribute content in the metadata of the target file based on the updated file attribute content.

9. An electronic device, characterized in that: The method comprises a processor and a memory, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of the storage processing method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that The invention comprises a computer program. When the computer program is run on an electronic device, the computer program is used to enable the electronic device to execute the steps of the storage processing method according to any one of claims 1 to 7.

11. A computer program product, characterized in that The invention comprises a computer program, which is stored in a computer-readable storage medium; when a processor of an electronic device reads the computer program from the computer-readable storage medium, the processor executes the computer program, so that the electronic device performs the steps of the storage processing method according to any one of claims 1 to 7.

Images