Metadata backup method and device, metadata retrieval method and device, computer equipment and storage medium

Abstract

The invention relates to a metadata backup method and device, a metadata retrieval method and device, computer equipment and a storage medium. The method comprises the steps of obtaining access log data of an object storage system from an access interface of the object storage system; identifying an operation behavior type for metadata included in at least one object based on a request information field in the access log data; based on the access log data and the operation behavior type, a metadata backup instruction corresponding to the at least one object is generated, and the metadata backup instruction is used for indicating a target metadata backup system independent of the object storage system to execute data updating. By adopting the method, the metadata backup process and the business of the object storage system can be decoupled, and the blockage or interference of the original object storage service caused by the performance fluctuation or network delay of the target metadata backup system is avoided, so that the inter-system coupling degree is effectively reduced; and high-availability backup and synchronization of the metadata in the heterogeneous system are realized.

Description

Technical Field

[0001] This application relates to the field of new information technology, and in particular to a metadata backup and retrieval method and apparatus, computer equipment, and storage medium. Background Technology

[0002] Object-based storage is a distributed storage system. Unlike traditional file systems and block storage, object storage does not use a folder hierarchy to organize data; instead, it stores all objects in a flat address space. With its significant advantages such as simple data access, high service and data availability, and strong scalability, object storage has become a popular technology choice for storing massive amounts of unstructured data.

[0003] Objects in object storage consist of data and its corresponding metadata. The metadata records the data's identifier, attributes, and characteristics, and corresponds one-to-one with the actual stored data. It is crucial information for managing and accessing object data. Problems with the metadata can directly lead to data inaccessibility, data loss, and even system performance degradation and service interruption. Therefore, effective backup of metadata is essential. Summary of the Invention

[0004] Therefore, it is necessary to provide a metadata backup and retrieval method, apparatus, computer equipment, and storage medium that can effectively back up metadata, addressing the aforementioned technical problems.

[0005] Firstly, this application provides a metadata backup method, including:

[0006] Obtain access log data of the object storage system from the access interface of the object storage system; the access log data is used to record operation behaviors on at least one object in the object storage system;

[0007] Based on the request information field in the access log data, identify the operation behavior type for the metadata included in the at least one object;

[0008] Based on the access log data and the operation behavior type, a metadata backup instruction corresponding to the at least one object is generated; the metadata backup instruction is used to instruct a target metadata backup system independent of the object storage system to perform a data update.

[0009] In one embodiment, identifying the type of operation behavior for the metadata included in the at least one object based on the request information field in the access log data includes:

[0010] When the request information field represents a creation request, the operation behavior type is identified as a creation type;

[0011] When the request information field represents a modification request, the operation behavior type is identified as a modification type;

[0012] When the request information field represents a deletion request, the operation behavior type is identified as a deletion type.

[0013] In one embodiment, generating a metadata backup instruction corresponding to the at least one object based on the access log data and the operation behavior type includes:

[0014] When the operation behavior type is a creation type or a modification type, extract the object identifier of the at least one object and the metadata change data associated with the object identifier from the access log data; the metadata change data includes a set of metadata fields that need to be written or updated;

[0015] The metadata backup instruction is generated based on the object identifier, the metadata change data, and the operation behavior type.

[0016] In one embodiment, generating a metadata backup instruction corresponding to the at least one object based on the access log data and the operation behavior type includes:

[0017] If the operation behavior type is deletion, the object identifier of the at least one of them shall be extracted from the access log data;

[0018] The metadata backup instruction is generated based on the object identifier and the operation behavior type.

[0019] In one embodiment, obtaining the access log data of the object storage system from the access interface of the object storage system includes:

[0020] Obtain the raw log data of the object storage system from the access interface;

[0021] Based on preset filtering rules, log data triggered by external requests are determined from the original log data to obtain candidate log data;

[0022] Based on the status fields included in the candidate log data, log data indicating successful requests are determined from the candidate log data to obtain the access log data.

[0023] In one embodiment, the method further includes:

[0024] In response to a change event notification message, the object identifier of the changed object in the change event notification message is obtained; the change event notification message is an event notification triggered by the object storage system when the metadata of an object changes.

[0025] Read metadata content that matches the changed object identifier from the object storage system;

[0026] The metadata content is input into a preset feature extraction service for feature extraction to obtain feature metadata;

[0027] Based on the object identifier, the feature metadata is updated to the target metadata backup system.

[0028] Secondly, this application also provides a metadata retrieval method, which includes:

[0029] Receive a data retrieval request from an access interface of an object storage system; the data retrieval request is generated by the access interface when it recognizes that the original query it receives is a retrieval type;

[0030] The target metadata backup system, which uses the metadata backup method provided in the first aspect of this application, processes the data retrieval request to obtain the retrieval results.

[0031] Thirdly, this application also provides a metadata backup device, comprising:

[0032] The acquisition module is used to acquire access log data of the object storage system from the access interface of the object storage system; the access log data is used to record the operation behavior of at least one object in the object storage system;

[0033] The identification module is used to identify the type of operation behavior for the metadata included in the at least one object based on the fields in the access log data that characterize the nature of the operation behavior;

[0034] The generation module is used to generate a metadata backup instruction corresponding to the at least one object based on the access log data and the operation behavior type; the metadata backup instruction is used to instruct a target metadata backup system independent of the object storage system to perform a data update.

[0035] Fourthly, this application also provides a metadata retrieval device, comprising:

[0036] A receiving module is used to receive data retrieval requests from the access interface of the object storage system; the data retrieval request is generated by the access interface when it recognizes that the original query statement it receives is a retrieval type;

[0037] The retrieval module is used to process the data retrieval request of the target metadata backup system backed up using the metadata backup method provided in the first aspect of this application, and obtain retrieval results.

[0038] Fifthly, this application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the metadata backup method provided in the first aspect of this application or the metadata retrieval method provided in the second aspect of this application.

[0039] Sixthly, this application also provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the metadata backup method provided in the first aspect of this application or the metadata retrieval method provided in the second aspect of this application.

[0040] In a seventh aspect, this application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the metadata backup method provided in the first aspect of this application or the metadata retrieval method provided in the second aspect of this application.

[0041] The aforementioned metadata backup and retrieval methods, apparatus, computer equipment, and storage media obtain access log data from the object storage system's access interface, identify the operation behavior type for metadata of at least one object based on the request information field in the access log data, and generate a metadata backup instruction corresponding to at least one object based on the access log data and the operation behavior type. This application, by collecting and parsing the access log data of the object storage system, can achieve non-intrusive heterogeneous backup of object storage metadata, decoupling the metadata backup process from the core business of the object storage system. This avoids performance fluctuations or network latency of the target metadata backup system from blocking or interfering with the original object storage service, thereby effectively reducing the coupling between systems while achieving high-availability backup and synchronization of metadata in heterogeneous systems, and enabling backups both within and across regions. Attached Figure Description

[0042] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the description of the embodiments of this application or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0043] Figure 1 This is a diagram illustrating the application environment of a metadata backup method in one embodiment.

[0044] Figure 2 This is a flowchart illustrating a metadata backup method in one embodiment;

[0045] Figure 3 This is a schematic diagram of the process for obtaining access log data in one embodiment;

[0046] Figure 4 This is a flowchart illustrating the process of identifying operation behavior types in one embodiment;

[0047] Figure 5 This is a flowchart illustrating the process of generating metadata backup instructions in one embodiment;

[0048] Figure 6 A flowchart illustrating the process of generating metadata backup instructions in another embodiment;

[0049] Figure 7 This is a flowchart illustrating the metadata backup method in another embodiment;

[0050] Figure 8 This is a flowchart illustrating a metadata retrieval method in one embodiment;

[0051] Figure 9 This is a schematic diagram illustrating a method for performing metadata backup in a log analysis system according to one embodiment;

[0052] Figure 10 This is a schematic diagram illustrating the metadata retrieval method performed by the log analysis system in one embodiment;

[0053] Figure 11 This is a structural block diagram of a metadata backup device in one embodiment;

[0054] Figure 12 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0055] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0056] It should be noted that the terms "first," "second," etc., used in this application can be used to describe various elements, but these elements are not limited by these terms. These terms are only used to distinguish the first element from the second element. The terms "comprising" and "having," and any variations thereof, used in this application, are intended to cover non-exclusive inclusion. The term "multiple" used in this application refers to two or more. The term "and / or" used in this application refers to one of the embodiments, or any combination of multiple embodiments.

[0057] First, the relevant terms or concepts in the embodiments of this application will be explained:

[0058] Object-Based Storage (OPS) is a distributed storage system that, unlike traditional file systems and block storage, does not use a hierarchical folder structure to organize data. Instead, it stores all objects in a flat address space. Each object consists of three parts: metadata, data, and a unique identifier. Object storage is gaining increasing market favor due to its flat, easily manageable structure, high scalability, and high availability.

[0059] ETL (Extract, Transform, Load) involves obtaining raw data from a data source, then performing preliminary cleaning, formatting, aggregation, and other transformations to make it conform to the requirements of the target system, and finally loading the transformed data into the target system.

[0060] The metadata backup method provided in this application embodiment can be applied to, for example... Figure 1 In the application environment shown, computer device 102 communicates with the access interface 104 of the object storage system to be backed up via a network. The target metadata backup system can be integrated into computer device 102 or located in the cloud or on another network server. Computer device 102 can obtain access log data of the object storage system through the access interface 104, identify the operation behavior type of metadata included in at least one object based on the access log data, generate metadata backup instructions corresponding to at least one object based on the access log data and the operation behavior type, and use the metadata backup instructions to update the target metadata backup system, which is independent of the object storage system.

[0061] The computer device 102 can be a terminal or a server. The terminal can be, but is not limited to, various personal computers, laptops, smartphones, and tablets. The server can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides cloud computing services.

[0062] In one exemplary embodiment, such as Figure 2 As shown, a metadata backup method is provided, which can be applied to... Figure 1 The following steps are used as an example of computer equipment, including steps 202 to 206. Wherein:

[0063] Step 202: Obtain the access log data of the object storage system from the access interface of the object storage system.

[0064] The access log data is used to record the operation behavior of at least one object in the object storage system.

[0065] In this embodiment, the access interface refers to the data interaction entry point for the object storage system to provide services to the outside world. It may include a service gateway or front-end control node that supports standard network protocols (such as HTTP / HTTPS, RESTful API). The access interface can be used to receive requests from clients for uploading, downloading, deleting, and modifying attributes of objects. Access log data can characterize the pipeline information generated by the object storage system in the process of processing the above requests. Each log entry in the access log data may include key fields such as the time the request was initiated, the request source address, the request method, the unique identifier of the object being operated on, the status code of the request processing result, and related header information.

[0066] For example, a computer device establishes a communication connection with an object storage system via a network access interface and continuously or periodically collects access log data from that interface. During this process, the computer device does not directly interfere with the core read / write path of the object storage system, but rather obtains the log stream from the access interface in a non-intrusive manner, either by bypassing or through post-processing. The access log data obtained by the computer device includes data on all operational behaviors related to at least one object in the object storage system (such as unstructured data like images, videos, and documents), such as modifications to metadata or overwriting of entity data.

[0067] Step 204: Based on the request information field in the access log data, identify the operation behavior type for the metadata included in at least one object.

[0068] The request information field can refer to a key data field in the access log data used to characterize the intent or operational semantics of the request instruction. It can include an opcode, action identifier, or a specific protocol instruction string. The operation behavior type can refer to a logical category based on the impact of the request information field on the metadata state, such as a write class indicating that metadata needs to be persisted or a cleanup class indicating that metadata needs to be removed.

[0069] For example, the computer device parses each log entry in the access log data and extracts the request information field. The computer device can then match this field with preset behavior mapping rules to determine the type of operation on metadata recorded in each log entry.

[0070] Step 206: Based on the access log data and operation behavior type, generate at least one metadata backup instruction corresponding to an object.

[0071] The metadata backup command is used to instruct a target metadata backup system, which is independent of the object storage system, to perform data updates.

[0072] In this embodiment, the metadata backup instruction may include the operational intent and data load required to synchronize changes in the object storage system to the target metadata backup system. The target metadata backup system refers to a heterogeneous storage component that is logically or physically independent of the original object storage cluster, and it may have independent data indexing and retrieval capabilities.

[0073] For example, a computer device parses access log data based on the type of operation to extract key parameters required to construct metadata backup instructions. The computer device can map the type of operation to an action instruction recognizable by the target metadata backup system, and encapsulate the object identifier and related attribute information parsed from the access log data that matches the action into the instruction. For instance, if the operation type indicates a state change in metadata, the computer device can generate a synchronization instruction containing the object identifier and the latest state information; if the operation type indicates the end of the metadata lifecycle, the computer device can generate a cleanup instruction containing the object identifier. These metadata backup instructions are then sent to the target metadata backup system, driving it to perform corresponding data write, update, or remove operations, thereby completing the backup and synchronization of metadata in a heterogeneous environment.

[0074] In the aforementioned metadata backup method, access log data of the object storage system is obtained from the access interface of the object storage system. Based on the request information field in the access log data, the operation behavior type for the metadata of at least one object is identified. Based on the access log data and the operation behavior type, a metadata backup instruction corresponding to at least one object is generated. This embodiment of the application, by collecting and parsing the access log data of the object storage system, can achieve non-intrusive heterogeneous backup of object storage metadata. It decouples the metadata backup process from the core business of the object storage system, avoiding performance fluctuations or network latency of the target metadata backup system from blocking or interfering with the original object storage service. This effectively reduces the coupling between systems while achieving high-availability backup and synchronization of metadata in heterogeneous systems, and can perform backups both within and across regions.

[0075] In practical applications, the log data obtained from the access interface of an object storage system may contain multiple logs unrelated to metadata operations. These could include failed requests due to network fluctuations, internal heartbeat detection requests, or internal system maintenance logs. These invalid logs not only consume storage and computing resources but may also interfere with subsequent backup logic. Therefore, to improve the accuracy of data processing, in some embodiments, the computer device can perform cleaning and filtering on the raw log data, removing invalid or meaningless records to obtain access log data that accurately reflects the operations performed on object metadata.

[0076] In one exemplary embodiment, such as Figure 3 As shown, step 202 includes steps 302 to 306. Wherein:

[0077] Step 302: Obtain the raw log data of the object storage system from the access interface.

[0078] Raw log data refers to the initial record stream generated by the service gateway or front-end node of the object storage system when it receives any request. It contains all interaction information without filtering. Raw log data not only records successful requests, but also includes failed requests, internal health check requests, and cross-node internal communication requests. It is the data source for subsequent data cleaning and value extraction.

[0079] For example, a computer device can deploy and run a log collection component, such as an ETL component. This ETL component establishes a connection with the access interface of an object storage system and collects raw log data from the access interface by listening to file streams, polling API interfaces, or receiving system log pushes. The ETL component can also load the data into a massive database with retrieval and analysis capabilities.

[0080] Step 304: Based on preset filtering rules, determine the log data triggered by external requests from the original log data to obtain candidate log data.

[0081] The preset filtering rules refer to logical judgment conditions used to distinguish between business traffic and system management traffic, usually based on at least one of the following request identification fields: client IP address, user agent, API access endpoint, or request identity identifier. Candidate log data refers to a collection of logs related to business logic initiated by external clients (such as user terminals or business servers) after initial screening, excluding noisy data generated by non-business operations such as internal system heartbeat detection and inter-cluster data balancing.

[0082] For example, the computer device parses the request identifier field in the raw log data one by one based on preset filtering rules. For instance, the computer device identifies whether the request source IP address belongs to a predefined internal management network segment, or checks whether the user agent field contains an identifier for an internal inspection tool. If the log data matches the characteristics of an external request, it is marked and retained as candidate log data; if it is identified as internal system behavior, it is discarded directly.

[0083] Step 306: Based on the status fields included in the candidate log data, determine the log data of successful requests from the candidate log data to obtain the access log data.

[0084] The status field can refer to a code identifier in the log data that indicates the result of the request processing, which can include standard HTTP protocol status codes. Log data indicating a successful request can refer to a status field that indicates the operation has been correctly executed and taken effect in the object storage system.

[0085] For example, the computer device traverses the candidate log data, checking the status field of each log entry. The system performs a matching judgment based on a preset set of success status codes. When the status field of a candidate log entry belongs to the set of success status codes, the candidate log entry is confirmed as access log data; for logs whose status field indicates failure, the computer device discards them.

[0086] In this embodiment, by using ETL components to collect raw log data and combining preset filtering rules and status field verification mechanisms to implement multi-level cleaning, it is possible to accurately filter out internal system noise and invalid requests that fail to execute from the massive and messy initial record stream, ensuring that only external business operations that are actually effective are included in the backup scope. This reduces system resource consumption while greatly improving the data quality and consistency of heterogeneous metadata backup.

[0087] It is understandable that the access log data may also contain log data unrelated to changes in object metadata. In some embodiments, after obtaining the access log data, the computer device identifies the type of operation behavior targeting metadata through the request information field in the access log data, thereby eliminating log data unrelated to object metadata change operations.

[0088] For example, if the request information field indicates that the request corresponding to the log is a query request or a read request, then if the operation behavior type for the metadata is identified as query type or read type, the access log data that indicates a query request or read request can be removed.

[0089] In one exemplary embodiment, such as Figure 4 As shown, step 204 includes steps 402 to 406. Wherein:

[0090] Step 402: If the request information field represents a creation request, identify the operation behavior type as a creation type.

[0091] A creation request can refer to an operation instruction initiated by a client to an object storage system, aimed at generating a new object and its corresponding metadata.

[0092] For example, in a standard RESTful protocol log, the creation request can be identified by a specific HTTP method field (such as PUT, and for a non-existent key). The computer device can parse the request information field in the access log data, and when the value of that field represents upload creation semantics (e.g., a PUT method is detected and the log context indicates a new resource), the computer device recognizes the operation behavior type as a creation type.

[0093] Step 404: If the request information field represents a modification request, identify the operation behavior type as a modification type.

[0094] Modification requests can refer to changes made to the properties of existing objects in the object storage system (such as storage type, access control list, and custom tags).

[0095] For example, a computer device examines a request information field and, if it finds that the value of the field is POST (e.g., POSTObject is used to modify metadata) or detects an overwrite operation on an existing object, it identifies the operation behavior type as modification type.

[0096] Step 406: If the request information field represents a deletion request, identify the operation behavior type as deletion type.

[0097] In this context, a deletion request can refer to an operation instruction initiated by the client that aims to remove an object and its associated metadata from the object storage system.

[0098] For example, a computer device parses a request information field, and when it detects that the value of the field is DELETE (e.g., a DELETE Object operation), it identifies the operation behavior type as a deletion type.

[0099] In this embodiment, by parsing the request information field to accurately distinguish specific operation types such as creation, modification and deletion, it can provide a precise logical basis for generating targeted metadata backup instructions, thereby ensuring that the target metadata backup system can correctly execute the corresponding data synchronization actions and efficiently maintain the consistency between the native object storage system and the target metadata backup system.

[0100] It is understandable that computer devices can identify specific changes to metadata by recognizing request information fields, thereby guiding subsequent metadata backup operations.

[0101] In one exemplary embodiment, such as Figure 5 As shown, step 206 includes steps 502 and 504. Wherein:

[0102] Step 502: If the operation behavior type is creation or modification, extract at least one object identifier and the metadata change data associated with the object identifier from the access log data.

[0103] Metadata change data includes a set of metadata fields that need to be written to or updated.

[0104] In this embodiment, the object identifier refers to the unique addressing identifier of the object within the storage system and is the core field for constructing metadata instructions. Metadata change data refers to the set of metadata attributes parsed from access log data that need to be synchronized to the target backup repository, such as the object's ETag checksum, latest modification time, storage category, and client-defined metadata. These fields collectively define the latest state of the object after its creation or modification.

[0105] For example, for access log data with operation types of creation or modification, a computer device can use a batch processing framework (such as Flink, Spark, etc.) to batch process multiple log entries in the access log data. The object identifier of the operated object and all changed or write-required metadata attribute values ​​can be extracted from the request header, response body, or specific log fields. For instance, regarding the access log of an object upload request, the computer device can extract the object identifier, file size, entity tag (ETag), and storage type to obtain the metadata change data corresponding to that log entry.

[0106] Step 504: Generate metadata backup instructions based on object identifier, metadata change data, and operation behavior type.

[0107] The metadata backup instruction can be a JSON document containing fields for operation types such as insert, update, and delete, or a specific SQL statement.

[0108] For example, the computer device uses the object identifier extracted in step 502 as the primary key of the instruction, the metadata change data as the payload of the instruction, and the creation type or modification type as the action type of the instruction to generate a metadata backup instruction. After the instruction is generated, it can be sent to the target metadata backup system to achieve incremental or full data synchronization.

[0109] In this embodiment, by accurately extracting the object identifier and complete metadata change data for the creation or modification operation, and encapsulating them into synchronization instructions executable by the target metadata backup system, it can be ensured that when the object's lifecycle changes, the target metadata backup system can accurately synchronize the latest state of the object, thus guaranteeing the integrity and timeliness of heterogeneous backup data.

[0110] In one exemplary embodiment, such as Figure 6 As shown, step 206 includes steps 602 and 604. Wherein:

[0111] Step 602: If the operation behavior type is deletion, extract at least one object identifier from the access log data.

[0112] Among them, deletion type can refer to the operation of ending the metadata lifecycle, the result of which is that the corresponding metadata record in the target metadata backup system should be removed or marked as invalid.

[0113] For example, when the operation type is deletion, the computer device parses the access log data and extracts the object identifier of the object being operated on.

[0114] Step 604: Generate metadata backup instructions based on object identifier and operation behavior type.

[0115] For example, the computer device uses the object identifier extracted in step 602 as the lookup key for the instruction and the deletion type as the action type of the instruction to generate a metadata backup instruction. After the instruction is sent to the target metadata backup system, it will trigger the system to remove or logically delete the corresponding metadata records, thereby ensuring the accuracy of the metadata records in the backup library.

[0116] In this embodiment, by extracting only the necessary object identifiers and generating cleanup instructions for the deletion operation, unnecessary metadata extraction and transmission overhead can be avoided, achieving efficient and rapid cleanup of the metadata records of deleted objects, thereby ensuring that the data of the target metadata backup system is strictly consistent with the current state of the source object storage.

[0117] In practical applications, while the basic metadata of object storage (such as unique identifiers, update times, entity tags, storage categories, etc.) can meet basic management needs, in order to achieve higher value mining of massive amounts of data, especially in the context of the increasingly popular application of artificial intelligence (AI), it is possible to conduct deeper analysis of unstructured data content.

[0118] In some embodiments, computer devices can utilize the event notification mechanism provided by the object storage system to monitor key operations such as adding, deleting, and modifying objects, and trigger callback requests to a preset feature extraction service module (e.g., for image content recognition, video content analysis, etc.). Through a post-processing, asynchronous callback approach, the extracted structured feature values ​​are updated to the target metadata backup system, thereby greatly expanding the dimensions and application scenarios of metadata and providing powerful capabilities and methods for intelligent retrieval and value analysis based on massive amounts of data.

[0119] In one exemplary embodiment, such as Figure 7As shown, the metadata backup method further includes steps 702 to 708, wherein:

[0120] Step 702: In response to the change event notification message, obtain the change object identifier from the change event notification message.

[0121] The change event notification message is an event notification triggered by the object storage system when the metadata of an object changes. The changed object identifier is the key information contained in this message, used to uniquely identify the object that has been changed.

[0122] For example, a computer device can continuously listen for event notification message queues or callback interfaces from an object storage system. When a change event notification message is received, the computer device parses the message, extracts the change object identifier indicating the changed object, and uses this identifier as a credential for locating object metadata and entity data in subsequent processing.

[0123] Step 704: Read the metadata content that matches the changed object identifier from the object storage system.

[0124] Metadata content refers to the original metadata set stored in the object storage system that is associated with the changed object identifier.

[0125] For example, a computer device uses the acquired modified object identifier to initiate a query request to an object storage system, the query request being used to read the metadata content of the object.

[0126] In other embodiments, if the feature extraction service needs to analyze object entity data (such as images or video files), the computer device can also obtain the path or temporary access credentials to access the entity data by reading the metadata content, so as to obtain the object entity data corresponding to the metadata content.

[0127] Step 706: Input the metadata content into the preset feature extraction service to extract features and obtain feature metadata.

[0128] For example, the computer device inputs the read metadata content into a preset feature extraction service, which may perform tasks such as image recognition, text content analysis, or video content structuring. After processing, the feature extraction service outputs structured feature metadata, such as pornography or violence detection results, AI-recognized tags, and a list of text keywords.

[0129] In other embodiments, the computer device may also input the metadata content and the object entity data obtained using the metadata content into a preset feature extraction service for feature extraction to enrich the feature metadata.

[0130] Step 708: Update the feature metadata to the target metadata backup system based on the object identifier.

[0131] For example, the computer device uses the changed object identifier as index information, encapsulates the changed object identifier and feature metadata into an update instruction, which is then sent to the target metadata backup system to perform an incremental update operation.

[0132] In this embodiment, by utilizing the change event notification mechanism of the object storage system, asynchronous and delayed extraction and updating of unstructured data content features can be achieved. This effectively expands the metadata dimension of the target metadata backup system, providing a data foundation for intelligent retrieval, classification, and value mining of massive amounts of unstructured data, and the entire process does not affect the data writing performance of the object storage system.

[0133] The second aspect of this application also provides a metadata retrieval method, such as... Figure 8 As shown, the method includes:

[0134] Step 802: Receive a data retrieval request from the access interface of the object storage system.

[0135] The data retrieval request is generated by the access interface when it recognizes that the original query it receives is a retrieval type.

[0136] In this embodiment, the data retrieval request is a standardized request generated by the access interface when it recognizes that the original query statement it receives (e.g., an API request or SQL query statement initiated by a client) belongs to the retrieval analysis type. Due to its flat structure and design limitations, native object storage systems typically only support precise queries based on object unique identifiers, and traversal queries are usually sorted only according to the ASCII code of the object unique identifier, resulting in poor retrieval capabilities and a lack of statistical analysis capabilities.

[0137] For example, the client sends a raw query statement to the access interface of the native object storage system. After receiving the raw query statement, the access interface parses and escapes it. If the intent of the raw query statement is a retrieval and analysis type, such as range query, arbitrary field sorting, fuzzy query, or involves statistical methods such as grouping and summation, average, etc., the access interface can generate a data retrieval request based on the raw query statement, indicate the required query conditions and analysis operations, and route it to the computer device that is communicatively connected to the access interface. The computer device receives the data retrieval request.

[0138] Step 804: The target metadata backup system, backed up using the above metadata backup method, processes the data retrieval request to obtain the retrieval results.

[0139] The target metadata backup system is a heterogeneous storage system independent of the object storage cluster. The target metadata backup system in this application embodiment can be built using a high-performance database with powerful indexing and query analysis capabilities, and synchronized through the metadata backup method provided in the first aspect of this application embodiment.

[0140] For example, a computer device routes a received data retrieval request to a target metadata backup system. This computer device can be deployed on the same server or in the same processing cluster as the target metadata backup system to efficiently perform the retrieval task. The target metadata backup system leverages its query analysis capabilities to efficiently execute complex queries. For instance, the target metadata backup system can perform grouping and summing based on the storage category of objects, sorting queries by arbitrary fields according to entity tags, or performing fuzzy queries based on feature metadata added based on event notifications. After completing the query and statistical analysis, the target metadata backup system returns the matching metadata records or aggregated statistical data as retrieval results to the access interface of the native object storage system.

[0141] In this embodiment, by identifying and routing complex retrieval and analysis requests at the access interface and handing them over to a high-performance target metadata backup system for execution, the limitations of native object storage systems in complex queries can be effectively overcome. This enables functions such as arbitrary field sorting and multidimensional statistical analysis, greatly improving the retrieval speed and analysis capabilities of massive metadata and providing strong support for users' data value mining.

[0142] The following section uses the application of metadata backup and metadata retrieval methods to a log analysis system as examples to further illustrate this application.

[0143] In some embodiments, please refer to Figure 9 and Figure 10 A metadata backup method is provided, which includes:

[0144] Step A: The log analysis system uses the log collection component to collect raw log data from the object storage gateway.

[0145] Object storage typically uses standard RESTful protocols, where all object identifiers are treated as resources in requests, and object attributes as sub-resources. All CRUD operations on resources or sub-resources are reflected in the requests. Therefore, the access logs of an object storage gateway can record all lifecycle operations. Basic metadata for object storage can include object identifiers, update times, ETags, storage types (storageClass), and user-defined data, all of which can be reflected in the access logs. The raw log data of the object storage gateway can include operation logs for the entire object lifecycle.

[0146] For example, a log analysis system can use ETL components to collect and filter raw log data from an object storage gateway, and load the raw log data into a log database, which can refer to a massive database with certain retrieval and analysis capabilities.

[0147] Step B: The log analysis system cleans the raw log data to obtain the target's access log data.

[0148] For example, the raw log data loaded into the database contains many invalid logs. The log analysis system can have a built-in log processing module to remove operation logs that are unrelated to changes in object metadata, such as failed requests and query requests, from the raw log data.

[0149] Step C: The log analysis system combines the access log data and the type of operation behavior on metadata to generate a metadata backup instruction; this metadata backup instruction is used to instruct the target metadata backup system, which is independent of the object storage system, to perform data updates.

[0150] For example, the amount of logs is greatly reduced after cleaning. The log analysis system then uses a batch processing computing framework to collect the changed metadata information in the access log data, and generates change instructions based on the specific operation type of the metadata change, driving the synchronous update of object metadata in the target metadata backup system.

[0151] Step D: The log analysis system receives a change event notification message including the changed object identifier, reads the metadata content matching the changed object identifier from the object storage system, inputs the metadata content into the preset feature extraction service for feature extraction, obtains feature metadata, and updates the feature metadata to the target metadata backup system.

[0152] In this embodiment, the most basic metadata of object storage may include unique identifiers (keys), update times, etags, storage classes, etc. However, to generate significant value from massive amounts of data, especially with the increasing popularity of AI, it is necessary to extract more feature values ​​from structured or unstructured data. Therefore, the log analysis system can also utilize the event notification capabilities of object storage to listen for object creation, deletion, and modification operations to trigger callbacks. The callback requests are then directed to the corresponding feature value extraction module (e.g., image and video content moderation, license plate recognition, etc.), and the corresponding feature values ​​are updated in the metadata database. This post-callback approach can greatly expand the metadata content, providing the capability and means to generate value from massive amounts of data.

[0153] In other embodiments, since native object storage traversal is typically sorted by the ASCII code of the object key value, it does not support other sorting queries, resulting in limited retrieval capabilities and a lack of statistical analysis capabilities. Databases, on the other hand, often possess powerful retrieval and analysis capabilities, such as supporting DSL or SQL query statements. Therefore, this embodiment can route object storage retrieval and analysis requests at the object storage gateway to the target metadata backup system generated by the aforementioned metadata backup method. After statement parsing and escaping, the target metadata backup system can provide retrieval and analysis results. For example, range queries, arbitrary field sorting queries, fuzzy queries, and statistical methods such as grouped summation, average, median, and variance can enhance the data management and analysis capabilities of the storage system, empowering users.

[0154] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages in other steps. It is understood that the steps in different embodiments can be freely combined as needed, and all non-contradictory solutions formed by such combinations are within the scope of protection of this application.

[0155] Based on the same inventive concept, this application also provides a metadata backup device for implementing the metadata backup method described above. The solution provided by this device is similar to the implementation described in the above method; therefore, the specific limitations in one or more metadata backup device embodiments provided below can be found in the limitations of the metadata backup method described above, and will not be repeated here.

[0156] In one exemplary embodiment, such as Figure 11 As shown, a metadata backup device is provided, including: an acquisition module 1102, an identification module 1104, and a generation module 1106, wherein:

[0157] The acquisition module 1102 is used to acquire access log data of the object storage system from the access interface of the object storage system; the access log data is used to record the operation behavior of at least one object in the object storage system.

[0158] The identification module 1104 is used to identify the type of operation behavior for the metadata included in the at least one object based on the fields in the access log data that characterize the nature of the operation behavior.

[0159] The generation module 1106 is used to generate a metadata backup instruction corresponding to the at least one object based on the access log data and the operation behavior type; the metadata backup instruction is used to instruct a target metadata backup system independent of the object storage system to perform a data update.

[0160] In one embodiment, the metadata backup device further includes a feature module, which is configured to: in response to a change event notification message, obtain a changed object identifier from the change event notification message; the change event notification message is an event notification triggered by the object storage system when the metadata of an object changes; read metadata content matching the changed object identifier from the object storage system; input the metadata content into a preset feature extraction service for feature extraction to obtain feature metadata; and update the feature metadata to the target metadata backup system based on the object identifier.

[0161] Each module in the aforementioned metadata backup device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can invoke and execute the operations corresponding to each module.

[0162] In one exemplary embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 12 As shown, this computer device includes a processor, memory, input / output (I / O) interfaces, and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and databases. The internal memory provides the environment for the operating system and computer programs stored in the non-volatile storage media to run. The I / O interfaces are used for exchanging information between the processor and external devices. The communication interface is used for communicating with external terminals via a network connection. When the computer program is executed by the processor, it implements a metadata backup method or a metadata retrieval method.

[0163] Those skilled in the art will understand that Figure 12The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0164] In one exemplary embodiment, a computer device is provided, including a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the steps of the metadata backup method provided in the first aspect of the present application or the metadata retrieval method provided in the second aspect of the present application.

[0165] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored, wherein when the computer program is executed by a processor, the steps of the metadata backup method provided in the first aspect of the embodiments of this application or the metadata retrieval method provided in the second aspect of the embodiments of this application are implemented.

[0166] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps of the metadata backup method provided in the first aspect of the embodiments of this application or the metadata retrieval method provided in the second aspect of the embodiments of this application.

[0167] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.

[0168] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, artificial intelligence (AI) processors, etc., and are not limited to these.

[0169] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0170] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A metadata backup method characterized by comprising: The method comprises: obtaining access log data of the object storage system from an access interface of the object storage system; the access log data is used to record operation behaviors on at least one object in the object storage system; based on the request information field in the access log data, identifying the operation behavior type of the metadata included in the at least one object; based on the access log data and the operation behavior type, generating metadata backup instructions corresponding to the at least one object; the metadata backup instructions are used to instruct a target metadata backup system independent of the object storage system to perform data update.

2. The method of claim 1, wherein, The method comprises: in the case that the request information field represents a creation request, identifying the operation behavior type as a creation type; in the case that the request information field represents a modification request, identifying the operation behavior type as a modification type; in the case that the request information field represents a deletion request, identifying the operation behavior type as a deletion type.

3. The method of claim 2, wherein, The method comprises: in the case that the operation behavior type is a creation type or a modification type, extracting the object identifier of the at least one object and the metadata change data associated with the object identifier from the access log data; the metadata change data includes a set of metadata fields that need to be written or updated; based on the object identifier, the metadata change data and the operation behavior type, generating the metadata backup instructions.

4. The method of claim 2, wherein, The method comprises: in the case that the operation behavior type is a deletion type, extracting the object identifier of the at least one object from the access log data; based on the object identifier and the operation behavior type, generating the metadata backup instructions.

5. The method of claim 1, wherein, The method comprises: obtaining original log data of the object storage system from the access interface; based on a preset filtering rule, determining log data triggered by external requests from the original log data to obtain candidate log data; based on the state field included in the candidate log data, determining log data of successful requests from the candidate log data to obtain the access log data.

6. The method according to any one of claims 1 to 5, characterized in that, The method further comprises: in response to a change event notification message, obtaining a change object identifier in the change event notification message; the change event notification message is an event notification triggered by the object storage system when the metadata of an object changes; reading metadata content matching the change object identifier from the object storage system; inputting the metadata content into a preset feature extraction service for feature extraction to obtain feature metadata; based on the object identifier, updating the feature metadata to the target metadata backup system.

7. A metadata retrieval method characterized by comprising: The method comprises: receiving a data retrieval request from an access interface of an object storage system; the data retrieval request is generated by the access interface in a case that the access interface identifies that an original query statement received thereby is of a retrieval type; a target metadata backup system backed up by the metadata backup method of any one of claims 1 to 6, processes the data retrieval request to obtain a retrieval result.

8. A metadata backup apparatus characterized by comprising: The apparatus comprises: an obtaining module, configured to obtain access log data of an object storage system from an access interface of the object storage system; the access log data is used to record an operation behavior on at least one object in the object storage system; an identifying module, configured to identify, based on a field representing a nature of the operation behavior in the access log data, a type of the operation behavior on metadata included in the at least one object; a generating module, configured to generate, based on the access log data and the type of the operation behavior, a metadata backup instruction corresponding to the at least one object; the metadata backup instruction is used to instruct a target metadata backup system independent of the object storage system to perform data updating. 9.A computer device, comprising a memory and a processor, wherein the memory stores a computer program, and the computer device is configured to perform the method according to any one of claims 1-8 when the computer program is executed by the processor. The processor executes the computer program to implement the steps of the method of any one of claims 1 to 7.

10. A computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement the steps of the method of any one of claims 1 to 7.

Images