Object list acquisition method and device, equipment, medium and program product

Abstract

The invention relates to an object list obtaining method, device and equipment, a medium and a program product, and the method comprises the steps that an export instruction of a target terminal is received, and the export instruction comprises a target moment; in response to the export instruction, determining a file state of a target file stored in the target node at the target moment, and performing export operation on the target file according to the file state; and sending the exported target file to a target terminal, so that the target terminal analyzes the target file to obtain a target object list. By adopting the method, the obtained target object list can be matched with the obtaining time, and the time point consistency of the target object list is improved.

Description

Methods, apparatus, devices, media, and program products for obtaining the object inventory Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a method, apparatus, device, medium, and program product for obtaining an object list. Background Technology

[0002] With the development of data processing technology, storage systems often need to filter and export object lists based on specific conditions to support upper-level business such as data analysis and migration backup.

[0003] In traditional implementations, the object list is usually obtained by scanning the entire metadata. However, this scanning method makes it difficult to guarantee that the obtained object list has snapshot consistency. That is, the obtained object list may not be consistent data at the same point in time, which affects the correctness of subsequent business processing. Summary of the Invention

[0004] Therefore, it is necessary to provide a method, apparatus, device, medium, and program product for obtaining an object list to address the above-mentioned technical problems, so that the obtained object list matches the timing of the acquisition.

[0005] Firstly, this application provides a method for obtaining an object list, applied to a target node of a processing device, the method comprising:

[0006] Receive the export command from the target terminal, which includes the target time.

[0007] In response to the export command, determine the file status of the target file stored on the target node at the target time, and perform the export operation on the target file according to the file status;

[0008] The exported target file is sent to the target terminal so that the target terminal can parse the target file and obtain a list of target objects.

[0009] In this embodiment, by receiving an export command from the target terminal, and the export command including a target time, a clear snapshot time point is established for the overall export operation. This avoids data state chaos caused by ambiguous time points or excessively long scanning cycles in subsequent export operations, providing a basis for obtaining a time-consistent object list. By responding to the export command, the file state of the target file stored on the target node at the target time is determined, and the export operation is performed on the target file based on the file state. This allows for export operations on the target file at the target time in conjunction with the file state, avoiding snapshot inconsistencies caused by continuous changes in metadata. Furthermore, by sending the exported target file to the target terminal, the target terminal parses the target file to obtain a target object list matching the acquisition timing.

[0010] In one embodiment, performing an export operation on the target file based on the file status includes: if the file status at the target time indicates that the target file is being written by the target writer, then sending an error message to the target writer so that the target writer responds to the error message, cancels the writing to the target file, and continues writing to other files on the target node; and performing an export operation on the target file.

[0011] In this embodiment, by sending an error message to the target writer when the file status at the target time indicates that the target file is being written to, concurrent write operations on the target file can be actively interrupted. This avoids inconsistencies in file content due to continuous data changes during the export process, thus ensuring data consistency. Simultaneously, the continuity of the write service itself can be maintained by continuing to write to other files on the target node.

[0012] In one embodiment, performing an export operation on a target file based on the file status includes: if the file status at the target time indicates that the target file is being deleted, then performing delayed deletion processing on the target file; performing an export operation on the target file; and deleting the target file stored in the target node after the export operation is completed.

[0013] In this embodiment, by delaying the deletion of the target file when its status indicates that the file is being deleted at the target time, the physical deletion operation is temporarily suspended, ensuring that the file content is logically preserved. This provides a time window for subsequent export operations and avoids data loss in the target object list due to the deletion of the target file. Simultaneously, after the export operation is completed, the target file stored in the target node is deleted. This ensures the smooth execution of the export operation while simultaneously deleting the target file, effectively coordinating data export requirements with space reclamation.

[0014] In one embodiment, the method further includes: receiving a cancel export instruction from the target terminal; the cancel export instruction is generated when the target terminal does not receive response information for the export instruction from each target node in the processing device within a preset time period; and terminating the export operation on the target file in response to the cancel export instruction.

[0015] In this embodiment, the export operation on the target file is terminated in response to the cancel export command. The cancel export command is generated when the target terminal does not receive response information from each target node in the processing device for the export command within a preset time period. Therefore, the export operation can be terminated for target nodes that do not respond within the preset time period, thereby timely stopping subsequent data processing and resource occupation that may be meaningless, which is beneficial to improving the overall resource utilization efficiency and responsiveness of the system.

[0016] In one embodiment, receiving the export instruction from the target terminal includes: responding to the preparation instruction from the target terminal by sending confirmation preparation information back to the target terminal; receiving the export instruction from the target terminal; the export instruction is obtained when the target terminal receives confirmation preparation information from each target node in the processing device.

[0017] In this embodiment, by introducing a preparation command, a state synchronization channel can be established between the target terminal and the target node, enabling the querying of the target node's preparation status and ensuring the smooth execution of subsequent export operations. Since the export command is obtained after the target terminal receives confirmation of preparation information from each target node in the processing device, it provides a foundation for globally consistent export tasks in distributed multi-target node scenarios, avoiding invalid attempts or erroneous results due to inconsistent node states.

[0018] Secondly, this application also provides a method for obtaining an object list, applied to a target terminal, the method comprising:

[0019] Send an export command to the target node of the processing device. The export command includes the target time.

[0020] Receive the target file sent by the target node. The target file is obtained by the target node responding to the export command, determining the file status of the target file stored by the target node at the target time, and performing an export operation on the target file according to the file status.

[0021] The target file is parsed to obtain a list of target objects.

[0022] In this embodiment, by sending an export command to the target node of the processing device, and the export command including the target time, a clear snapshot time point is established for the overall export operation. This avoids data state chaos caused by ambiguous time points or excessively long scanning cycles in subsequent export operations, providing a basis for obtaining a time-consistent object list. By receiving the target file sent by the target node, the target node determines the file state of the target file stored at the target time in response to the export command, and performs an export operation on the target file according to the file state. This allows the target node to combine the file state to perform an export operation on the target file at the target time, avoiding snapshot inconsistencies caused by continuous changes in metadata. Furthermore, by parsing the target file, a target object list matching the acquisition timing is obtained.

[0023] In one embodiment, the method further includes: if no response information for the export instruction is received from each target node in the processing device within a preset time period, sending a cancel export instruction to the target node that has not provided a response, so that the corresponding target node terminates the export operation on the target file.

[0024] In this embodiment, if no response information for the export instruction is received from each target node in the processing device within a preset time period, a cancel export instruction is sent to the target node that has not responded. This allows the export operation to be terminated for target nodes that have not responded within the preset time period, thereby stopping potentially meaningless subsequent data processing and resource consumption in a timely manner, which is beneficial to improving the overall resource utilization efficiency and responsiveness of the system.

[0025] In one embodiment, sending an export instruction to a target node of the processing device includes: sending a preparation instruction to each target node of the processing device so that the corresponding target node can provide confirmation of preparation information; and sending an export instruction to each target node upon receiving confirmation of preparation information from each target node.

[0026] In this embodiment of the application, by introducing a preparation instruction and sending an export instruction to each target node upon receiving confirmation of preparation information from each target node, a foundation is provided for a globally consistent export task in a distributed multi-target node scenario, avoiding invalid attempts or erroneous results due to inconsistent node states.

[0027] In one embodiment, parsing the target file to obtain a target object list includes: parsing the target file corresponding to each target node to obtain an initial metadata list; the initial metadata list includes data information of at least one metadata, the data information including the data identifier and file version information of the corresponding metadata; removing invalid metadata from the initial metadata list based on the data identifier and file version information to obtain a target metadata list; and determining the target object list based on the target metadata list.

[0028] In this embodiment of the application, by removing invalid metadata from the initial metadata list based on data identifiers and file version information, a target metadata list is obtained, thereby avoiding the inclusion of invalid or outdated object information in the final target object list and improving the data quality, accuracy and reliability of the target object list.

[0029] In one embodiment, invalid metadata is removed from the initial metadata list based on the data identifier and file version information, including: obtaining multiple candidate metadata corresponding to the same data identifier in the initial metadata list; and removing invalid metadata from the multiple candidate metadata, wherein invalid metadata refers to other candidate metadata other than the latest file version information among the multiple candidate metadata.

[0030] By obtaining multiple candidate metadata entries with the same data identifier from the initial metadata list, duplicate metadata records can be identified, providing a data foundation for subsequent filtering and extraction of invalid metadata. By removing invalid metadata from the multiple candidate metadata entries (which are other candidate metadata besides the latest file version information), version management and merging of multiple update records for the same object are achieved during the construction of the target object list, ensuring the uniqueness and timeliness of each target object entry in the list.

[0031] Fourthly, this application also provides an object list acquisition device, configured at a target node of a processing device, the device comprising:

[0032] The first receiving module is used to receive the export instruction from the target terminal, which includes the target time.

[0033] The first processing module is used to respond to the export command, determine the file status of the target file stored on the target node at the target time, and perform the export operation on the target file according to the file status.

[0034] The first sending module is used to send the exported target file to the target terminal so that the target terminal can parse the target file and obtain a list of target objects.

[0035] Fifthly, this application also provides an apparatus for obtaining an object list, configured on a target terminal, the apparatus comprising:

[0036] The second sending module is used to send an export command to the target node of the processing device. The export command includes the target time.

[0037] The second receiving module is used to receive the target file sent by the target node. The target file is obtained by the target node responding to the export instruction, determining the file status of the target file stored by the target node at the target time, and performing an export operation on the target file according to the file status.

[0038] The parsing module is used to parse the target file and obtain a list of target objects.

[0039] Sixthly, this application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the methods of the first or second aspect described above in various possible implementations.

[0040] In a seventh aspect, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the methods of the first or second aspect described above in various possible implementations.

[0041] Eighthly, this application also provides a computer program product, including a computer program that, when executed by a processor, implements the methods of the first or second aspect described above in various possible implementations. Attached Figure Description

[0042] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the description of the embodiments of this application or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0043] Figure 1 is an application environment diagram of the object list acquisition method in one embodiment;

[0044] Figure 2A is a flowchart illustrating a method for obtaining an object list in one embodiment;

[0045] Figure 2B is a schematic diagram of the structure of distributed metadata in one embodiment;

[0046] Figure 2C is a schematic diagram of a shared storage architecture in one embodiment;

[0047] Figure 3 is a flowchart illustrating the process of terminating the target file export step in one embodiment;

[0048] Figure 4 is a flowchart illustrating the method for obtaining the object list in another embodiment;

[0049] Figure 5A is a flowchart illustrating the parsing steps of a target file in one embodiment;

[0050] Figure 5B is a schematic diagram of the process of determining the target object list in one embodiment;

[0051] Figure 5C is a schematic diagram of the process of determining the target object list in another embodiment;

[0052] Figure 6 is a sequence diagram of a method for obtaining an object list in one embodiment;

[0053] Figure 7 is a structural block diagram of an object list acquisition device in one embodiment;

[0054] Figure 8 is a structural block diagram of an object list acquisition device in another embodiment;

[0055] Figure 9 is an internal structure diagram of a computer device in one embodiment. Detailed Implementation

[0056] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0057] The method for obtaining the object list provided in this application embodiment can be applied to the application environment shown in Figure 1. The target terminal 102 can communicate with at least one target node 104 in the processing device via a network.

[0058] For example, the target terminal 102 can be, but is not limited to, various personal computers, laptops, smartphones, tablets, and IoT devices. The target terminal 102 can be an operation and maintenance terminal.

[0059] For example, the processing device may be, but is not limited to, a terminal device or a server. The server may be a standalone physical server, a server cluster or distributed system consisting of multiple physical servers, or a cloud server providing cloud computing services. The processing device may include at least one target node. Optionally, the processing device may be equipped with a key-value (kv) database. The data in the kv database is stored on different target nodes.

[0060] Understandably, traditional object list acquisition methods typically involve a full scan of the metadata stored on the target node. However, this scanning method struggles to guarantee snapshot consistency, meaning the acquired object list may not reflect consistent data from the same point in time, impacting the correctness of subsequent business processing. Therefore, this application provides a method for acquiring an object list. The target terminal 102 sends an export command to the target node 104 of the processing device, the export command including a target time. In response to the export command, the target node 104 determines the file status of the target file stored on the target node at the target time, performs an export operation on the target file based on the file status, and sends the exported target file to the target terminal 102. The target terminal 102 parses the target file to obtain the target object list.

[0061] In the above steps, by introducing an export command that includes a target time, a clear snapshot time point is established for the overall export operation. This avoids data state chaos caused by ambiguous time points or excessively long scan cycles in subsequent export operations, providing a foundation for obtaining a time-consistent object list. By responding to the export command, the file state of the target file stored on the target node at the target time is determined, and the export operation is performed on the target file based on the file state. This allows for exporting the target file at the target time while considering the file state, avoiding snapshot inconsistencies caused by continuous changes in metadata. Furthermore, by sending the exported target file to the target terminal, the target terminal parses the target file to obtain a target object list matching the acquisition timing.

[0062] In an exemplary embodiment, as shown in FIG2A, a method for obtaining an object list is provided. Taking the application of this method to target node 104 in FIG1 as an example, the method includes:

[0063] S210. Receive the export instruction from the target terminal. The export instruction includes the target time.

[0064] The target time can be understood as the snapshot time point used to synchronously export the list of target objects for at least one target node. By introducing the target time point, a clear snapshot time point is established for the overall export operation, avoiding data state chaos caused by ambiguous time points or excessively long scanning cycles in subsequent export operations, and providing a foundation for obtaining a time-consistent list of objects.

[0065] Optionally, the target time can be a pre-set export time, or it can be determined based on the sending time of the export command and a preset delay time. For example, the sending time and the preset delay time can be added together to obtain the target time.

[0066] In an optional embodiment, in response to a preparation instruction from the target terminal, confirmation preparation information can be fed back to the target terminal; and an export instruction can be received from the target terminal; the export instruction is obtained when the target terminal receives confirmation preparation information fed back by each target node in the processing device.

[0067] The preparation instruction, also known as the preparation export instruction, can be understood as an instruction sent by the target terminal to inquire about the preparation status of the target node.

[0068] The confirmation message can be ACK (Acknowledgement).

[0069] Optionally, in response to the preparation command from the target terminal, if the target terminal is in an idle state at the current time and / or at the target time, the system may send back confirmation information to the target terminal.

[0070] Optionally, if the target terminal does not receive confirmation of preparation information from each target node in the processing device within the target time period, it can resend the preparation command to each target node. Conversely, if the target terminal receives confirmation of preparation information from each target node in the processing device, it can send an export command to each target terminal. The target time period can be set by technical personnel based on needs or experience, or determined through extensive experimentation; this application does not impose any limitations on it.

[0071] Optionally, the preparation and export instructions can be sent from the target terminal to each target node via broadcast or other means. This application does not impose any restrictions on the specific method of sending the instructions.

[0072] Understandably, by introducing a preparation command, a state synchronization channel can be established between the target terminal and the target node, enabling the querying of the target node's preparation status and ensuring the smooth execution of subsequent export operations. Since the export command is obtained after the target terminal receives confirmation of preparation information from each target node in the processing device, it provides a foundation for globally consistent export tasks in distributed multi-target node scenarios, avoiding invalid attempts or erroneous results due to inconsistent node states.

[0073] S220. In response to the export command, determine the file status of the target file stored on the target node at the target time, and perform an export operation on the target file according to the file status.

[0074] The file status of the target file can include at least one of the following: write status, delete status, and inactive status. Write status can be understood as the target file being written to at the target time; delete status can be understood as the target file being deleted at the target time; inactive status can be understood as the target file not undergoing any data processing operations at the target time.

[0075] For example, the target file may include a binary file of an LSM (log-structured merge-tree) stored in the target node. The LSM is a sequentially written, hierarchical merge data structure that can be persistently stored using a key-value database. The binary file may include multiple binlogs (binary logs) and file description information. This application does not impose any limitations on the specific file type or content of the target file.

[0076] The export operation on the target file can be understood as copying and exporting the target file at the target time, that is, it is used to copy the snapshot data of the target file at the target time and export it.

[0077] In an optional embodiment, in response to an export command, an export task can be initiated at a target time and response information can be fed back to the target terminal. The export task includes determining the file status of the target file stored on the target node at the target time and performing an export operation on the target file according to the file status.

[0078] For example, performing an export operation on a target file based on its file status can be understood as recording a snapshot event and then exporting the target file. The snapshot event can include the export time and the corresponding list of target files at that time. The list of target files at the export time can be determined by combining the file status, thereby performing the export operation on the target files in the target file list. The target file list includes at least one target file.

[0079] For ease of understanding, the following examples illustrate three scenarios: write state, delete state, and quiescent state. It should be noted that this should not be interpreted as a limitation on the specific export steps.

[0080] A1. Write Status

[0081] In an optional embodiment, if the file status indicates that the target file is being written by the target writer at the target time, a delayed write information is sent to the target writer to make the target writer pause writing to the target file; after performing an export operation on the target file, a continue write information is sent to the target writer to make the target writer continue writing to the target file.

[0082] In another optional embodiment, if at the target time the file status indicates that the target file is being written by the target writer, an error message is sent to the target writer so that the target writer responds to the error message, cancels the writing of the target file, and continues writing to other files on the target node; and an export operation is performed on the target file.

[0083] The error message can be a pre-defined error code. This application does not impose any limitations on the specific content of the error message.

[0084] Other files may include at least one of the following: temporary files pre-defined in the target node, and new files created after the target time. This application does not impose any restrictions on the creation time of other files.

[0085] For example, a list of target files at a target time can be obtained, and the target files in the target file list can be exported.

[0086] Understandably, by sending an error message to the target writer when the file status at the target time indicates that the target file is being written to, concurrent write operations on the target file can be proactively interrupted. This avoids inconsistencies in file content due to continuous data changes during the export process, thus ensuring data consistency. Simultaneously, by continuing to write to other files on the target node, the continuity of the write service itself can be maintained.

[0087] A2, Deleted Status

[0088] In an optional embodiment, if the file status at the target time indicates that the target file is being deleted, then the target file is delayed for deletion; an export operation is performed on the target file; after the export operation is completed, the target file stored in the target node is deleted.

[0089] Optionally, delayed deletion of target files can be achieved by marking them as to be deleted. Correspondingly, target files marked with delayed deletion can be deleted after the export operation is completed.

[0090] In one implementation, the target file can be marked for delayed deletion so that the target deletion end delays the deletion of the target file; after the export operation is completed, a completion message is sent to the target deletion end so that the target deletion end can delete the target file with the deletion mark.

[0091] In another implementation, the target file can be marked for delayed deletion so that the target file can be deleted later; after the export operation is completed, the target node will delete the target file with the delayed deletion mark.

[0092] For example, a list of target files at a target time can be obtained, and the target files in the target file list can be exported.

[0093] Understandably, by delaying the deletion of a target file when its status indicates it is being deleted at the target time, the physical deletion operation is temporarily suspended, ensuring the file content is logically preserved. This provides a time window for subsequent export operations and avoids data loss in the target object list due to the file's deletion. Simultaneously, after the export operation is completed, the target file stored in the target node is deleted. This ensures the export operation executes smoothly while simultaneously deleting the target file, effectively coordinating data export needs with space reclamation.

[0094] A3. Still state

[0095] In an optional embodiment, if the file status representation has not undergone data operations at the target time, the target file at the target time is obtained; and an export operation is performed on the target file at the target time.

[0096] This includes operations that did not involve data manipulation, such as writing or deleting data from the target file, which could affect export consistency.

[0097] For example, a list of target files at a target time can be obtained, and the target files in the target file list can be exported.

[0098] S230. Send the exported target file to the target terminal so that the target terminal can parse the target file and obtain the target object list.

[0099] In an optional embodiment, the target terminal can parse the target files corresponding to each target node to obtain an initial metadata list; the initial metadata list includes data information of at least one metadata, the data information including the data identifier and file version information of the corresponding metadata; based on the data identifier and file version information, invalid metadata in the initial metadata list is removed to obtain a target metadata list; based on the target metadata list, a target object list is determined.

[0100] Optional. The target terminal can obtain multiple candidate metadata with the same data identifier from the initial metadata list; invalid metadata is removed from the multiple candidate metadata, which is other candidate metadata besides the latest file version information.

[0101] In an optional embodiment, referring to the foregoing, response information can be sent back to the target terminal in response to the export command. Accordingly, upon receiving the response information from all target nodes, the target terminal can parse the target files sent by each target node to obtain a list of target objects.

[0102] In an optional embodiment, the processing device described above may run an object storage system. This object storage system uses an LSM engine for metadata storage. The database managed by the LSM engine distributes data across different target nodes through consistent hashing, and read / write operations are performed on the corresponding target nodes. Each target node can correspond to one LSM instance. The reliability of metadata for a single LSM instance can be achieved using multiple replicas, i.e., according to a shared storage architecture. Therefore, globally consistent export operations require consistent processing across multiple target nodes.

[0103] Referring to Figure 2B, which illustrates the structure of distributed metadata, "hash(key)%n" is a distributed data sharding method used to map keys to a fixed number of target nodes (i.e., virtual nodes), so that each target node can correspond to an LSM instance.

[0104] Figure 2C illustrates a shared storage architecture. It shows that this architecture can include both star-row writes and chained writes. It's important to note that the shared storage architecture is stateless on the LSM engine side and does not handle data synchronization; this is handled by the underlying distributed storage. Both LSM and the underlying distributed storage use append-only writes, which facilitates subsequent export and replication.

[0105] Taking the star topology in Figure 2C as an example, under normal circumstances, each I / O operation (Input / Output, i.e., write operation) of the LSM instance needs to wait for all three replicas to complete before returning. When starting an export operation, a snapshot event can be recorded. This snapshot event records the time when the export task is started (i.e., the target time) and the list of SST files at that time to identify which files need to be copied. For the SST file being written at that time, an error code indicating that the write should be closed needs to be returned to the target writer, so that the target writer actively switches to the new file to continue writing. Since both the distributed storage and the LSM engine use append-only writes, and compaction only involves file deletion, the file can be not actually deleted after compaction; it can only be specially marked, and then actually deleted after the data copy is complete. In this way, the problem of concurrent operations between foreground writes and background exports is solved.

[0106] The method for obtaining the object list described above establishes a clear snapshot time point for the overall export operation by receiving an export command from the target terminal, which includes the target time. This avoids data state chaos caused by ambiguous time points or excessively long scanning cycles in subsequent export operations, providing a foundation for obtaining a time-consistent object list. By responding to the export command, the method determines the file state of the target files stored on the target node at the target time and performs an export operation on the target files based on the file state. This allows for exporting target files at the target time while considering the file state, avoiding snapshot inconsistencies caused by continuously changing metadata. Finally, by sending the exported target files to the target terminal, the target terminal parses the target files to obtain a target object list that matches the acquisition timing.

[0107] Based on the technical solutions of the above embodiments, this application also provides an optional embodiment, in which a step of terminating the export of the target file is added.

[0108] Referring to Figure 3, the flowchart for terminating the target file export step is shown, including:

[0109] S310, Receive the cancellation export command from the target terminal; the cancellation export command is generated when the target terminal does not receive response information for the export command from each target node in the processing device within a preset time period.

[0110] S320. In response to the cancel export command, terminate the export operation on the target file.

[0111] Referring to the foregoing, in response to an export command, an export task can be initiated at the target time and response information can be sent back to the target terminal. The export task includes determining the file status of the target files stored on the target node at the target time and performing an export operation on the target files according to the file status. Correspondingly, for the target terminal, upon receiving response information from each target node in the processing device regarding the export command within a preset time period, the target terminal can parse the target files received from each target node to obtain a list of target objects.

[0112] Conversely, if the target terminal does not receive any response information for the export command from each target node in the processing device within a preset time period, it can send a cancel export command to each target node or the target node that has not responded, thereby notifying the target node to cancel the current task, that is, to terminate the export operation on the target file.

[0113] For example, the target terminal can send a cancel export command to each target node or a target node that has not responded, and then repeat the above method for obtaining the object list, that is, resend the export command to the target nodes. The target terminal can send the cancel export command to each target node or a target node that has not responded via broadcast.

[0114] In the above steps, the export operation on the target file is terminated in response to the cancel export command. The cancel export command is generated when the target terminal does not receive response information from the target nodes in the processing device for the export command within a preset time period. Therefore, the export operation can be terminated for target nodes that do not respond within the preset time period, thereby stopping subsequent data processing and resource occupation that may no longer be meaningful in a timely manner, which is conducive to improving the overall resource utilization efficiency and responsiveness of the system.

[0115] Based on the technical solutions of the above embodiments, as shown in Figure 4, another method for obtaining the object list is provided. Taking the application of this method to the target terminal 102 in Figure 1 as an example, it includes:

[0116] S410. Send an export command to the target node of the processing device. The export command includes the target time.

[0117] The details regarding the export instructions and target time have already been explained above and will not be repeated here.

[0118] In an optional embodiment, the number of target nodes is at least one; accordingly, a preparation instruction can be sent to each target node of the processing device to enable the corresponding target node to provide confirmation of preparation information; upon receiving confirmation of preparation information from each target node, an export instruction is sent to each target node.

[0119] Understandably, by introducing a preparation command and sending an export command to each target node upon receiving confirmation of preparation information from each target node, a foundation is provided for globally consistent export tasks in a distributed multi-target node scenario, avoiding invalid attempts or erroneous results due to inconsistent node states.

[0120] S420: Receive the target file sent by the target node. The target file is obtained by the target node in response to the export instruction, determining the file status of the target file stored by the target node at the target time, and performing an export operation on the target file according to the file status.

[0121] In an optional embodiment, after sending an export command to the target node of the processing device, a response message sent by the target node can be received; the response message is the response message returned by the target node after responding to the export command and starting the export task.

[0122] Referring to the foregoing, the export task includes determining the file status of the target file stored on the target node at the target time, and performing an export operation on the target file based on the file status. The export task has been described in detail above, and the export methods for three different file statuses—write status, delete status, and quiescent status—have been listed, so they will not be repeated here.

[0123] In an optional embodiment, if no response information for the export command is received from each target node in the processing device within a preset time period, a cancel export command can be sent to each target node or the target node that has not provided a response, so that the corresponding target node terminates the export operation on the target file.

[0124] Understandably, by sending a cancel export command to the target nodes that have not responded to the export command within a preset time period, the export operation can be terminated for target nodes that have not responded within the preset time period. This can promptly stop subsequent data processing and resource consumption that may no longer be meaningful, thereby improving the overall resource utilization efficiency and responsiveness of the system.

[0125] S430. Parse the target file to obtain a list of target objects.

[0126] In an optional embodiment, if response information for the export command is received from each target node in the processing device within a preset time period, the target files received from each target node can be parsed to obtain a list of target objects.

[0127] The method for obtaining the object list described above establishes a clear snapshot time point for the overall export operation by sending an export command to the target node of the processing device. This export command includes the target time, preventing data state confusion caused by ambiguous time points or excessively long scan cycles in subsequent export operations, and providing a foundation for obtaining a time-consistent object list. By receiving the target file sent by the target node, which responds to the export command, determines the file state of the target file stored on the target node at the target time, and performs an export operation on the target file based on the file state, the target node can perform an export operation on the target file at the target time, avoiding snapshot inconsistencies caused by continuous changes in metadata. Furthermore, by parsing the target file, a target object list matching the acquisition timing is obtained.

[0128] Based on the technical solutions of the above embodiments, this application also provides an optional embodiment in which the parsing steps of the target file are refined.

[0129] Referring to Figure 5A, which shows a flowchart of the target file parsing steps, the process includes:

[0130] S510. Parse the target files corresponding to each target node to obtain an initial metadata list; the initial metadata list includes data information of at least one metadata, including the data identifier and file version information of the corresponding metadata.

[0131] For example, the target files corresponding to each target node can be parsed concurrently to obtain an initial metadata list. The target files may include sst (Sorted String Table) files.

[0132] S520. Based on the data identifier and file version information, remove invalid metadata from the initial metadata list to obtain the target metadata list.

[0133] Invalid metadata may include at least one of the following: duplicate metadata, or metadata whose file version information is old version information.

[0134] The file version information may include at least one of the following: version number and timestamp.

[0135] Optionally, multiple candidate metadata with the same data identifier can be obtained from the initial metadata list; invalid metadata is removed from the multiple candidate metadata, which are other candidate metadata besides the latest file version information.

[0136] For example, if multiple SST files contain data identifier keys with the same name, the latest valid data identifier can be identified by version number or timestamp, and the latest version can be used as the parsing result. To address the issue of a large number and size of SST files, concurrent parsing can be used, and a hash aggregation method can be employed to summarize the data identifier keys.

[0137] S530. Determine the target object list based on the target metadata list.

[0138] Referring to Figure 5B, this is a schematic diagram of the process of determining the target object list in one embodiment.

[0139] Figure 5B illustrates, for example, that the process of determining the target object list can mainly include the following three steps:

[0140] 1. Perform concurrent parsing of the sst files of multiple target nodes.

[0141] 2. After parsing, perform hash deduplication; where "hash(key)%n" is a distributed data sharding method used to map keys to a fixed number of target nodes (i.e., virtual nodes); after hash deduplication, multiple data lists objlist are obtained, which are also metadata lists.

[0142] 3. Merge the multiple deduplicated data lists objlist to obtain the merged data list objlist.

[0143] The merged data list objlist, also known as the target metadata list, refers to the full metadata information at the target time.

[0144] In an optional embodiment, the target metadata list can be filtered based on preset tasks to determine the target object list.

[0145] For example, for lifecycle tasks, the background service can filter the timestamp field in objlist, compare it with the lifecycle rules configured in the corresponding bucket strategy, and output a list of target objects to be deleted.

[0146] For example, for a tiered storage task, the timestamp field in objlist is compared with the specified rule time in the tiered storage to filter out the corresponding list of target objects to be converted.

[0147] Referring to Figure 5C, which illustrates the process of determining the target object list in another embodiment, Figure 5C shows the process of exporting a binary log from a kv database to obtain an sst file. Based on preset tasks, the sst file is parsed to obtain a target object list that conforms to the corresponding preset tasks. Preset tasks may include at least one. For example, the first preset task Item1 may be a lifecycle task, the second preset task Item2 may be a remote replication task, the third preset task Item3 may be a hierarchical storage task, and the fourth preset task Item4 may be other tasks. This application does not limit the specific task type or the specific number of preset tasks.

[0148] The above-described target file parsing steps, by removing invalid metadata from the initial metadata list based on data identifiers and file version information, yield a target metadata list. This avoids including invalid or outdated object information in the final target object list, improving the data quality, accuracy, and reliability of the target object list. By obtaining multiple candidate metadata records corresponding to the same data identifier from the initial metadata list, duplicate metadata records can be identified, providing a data foundation for subsequent filtering and extraction of invalid metadata. By removing invalid metadata from multiple candidate metadata records—specifically, invalid metadata that is not related to the latest file version information—version management and merging of multiple update records for the same object are achieved during the construction of the target object list, ensuring the uniqueness and timeliness of each target object entry in the target object list.

[0149] Based on the technical solutions of the above embodiments, this application also provides an optional embodiment, in which the method for obtaining the object list is described in detail.

[0150] Referring to the sequence diagram of the method for obtaining the object list shown in Figure 6, it includes:

[0151] S601, The target terminal sends a preparation command to the target node.

[0152] S602. The target node responds to the preparation command by sending confirmation information to the target terminal.

[0153] S603. Upon receiving confirmation and preparation information from the target node, the target terminal sends an export command to the target node.

[0154] The target node must have at least one node.

[0155] S604. The target node responds to the export command and starts the export task. The export task includes determining the file status of the target file stored on the target node at the target time and performing the export operation on the target file according to the file status.

[0156] For example, if at the target time the file status indicates that the target file is being written by the target writer, an error message is sent to the target writer so that the target writer responds to the error message, cancels the writing to the target file, and continues writing to other files on the target node; and an export operation is performed on the target file.

[0157] For example, if the file status at the target time indicates that the target file is being deleted, then the target file is delayed for deletion; an export operation is performed on the target file; after the export operation is completed, the target file stored in the target node is deleted.

[0158] S605. After starting the export task, the target node sends a response message to the target terminal.

[0159] For example, if the target terminal does not receive any response information for the export command from each target node in the processing device within a preset time period, it sends a cancel export command to the target node; the target node responds to the cancel export command and terminates the export operation on the target file.

[0160] S606, The target node sends the target file to the target terminal.

[0161] S607. The target terminal parses and processes the target files corresponding to each target node to obtain an initial metadata list. The initial metadata list includes data information of at least one metadata, including the data identifier and file version information of the corresponding metadata.

[0162] S608. The target terminal obtains multiple candidate metadata with the same data identifier from the initial metadata list.

[0163] S609. The target terminal removes invalid metadata from multiple candidate metadata to obtain a target metadata list. Invalid metadata refers to candidate metadata other than the latest file version information among the multiple candidate metadata.

[0164] S610. The target terminal determines the target object list based on the target metadata list.

[0165] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0166] Based on the same inventive concept, this application also provides an object list acquisition device for implementing the object list acquisition method described above. The solution provided by this device is similar to the implementation described in the above method; therefore, the specific limitations of one or more object list acquisition device embodiments provided below can be found in the limitations of the object list acquisition method described above, and will not be repeated here.

[0167] In an exemplary embodiment, as shown in FIG7, an object list acquisition device is provided, including: a first receiving module 710, a first processing module 720, and a first sending module 730, wherein:

[0168] The first receiving module 710 is used to receive the export instruction from the target terminal, the export instruction including the target time;

[0169] The first processing module 720 is used to respond to the export command, determine the file status of the target file stored in the target node at the target time, and perform an export operation on the target file according to the file status.

[0170] The first sending module 730 is used to send the exported target file to the target terminal so that the target terminal can parse the target file and obtain a list of target objects.

[0171] In an exemplary embodiment, the first processing module 720 includes: a first sending unit, configured to send an error message to the target writing end if the file status indicates that the target file is being written by the target writing end at the target time, so that the target writing end responds to the error message, cancels the writing of the target file, and continues writing to other files of the target node; and a first exporting unit, configured to perform an export operation on the target file.

[0172] In an exemplary embodiment, the first processing module 720 includes: a first processing unit, configured to perform delayed deletion processing on the target file if the file status indicates that the target file is being deleted at the target time; a second export unit, configured to perform an export operation on the target file; and a second processing unit, configured to perform deletion processing on the target file stored in the target node after the export operation is completed.

[0173] In an exemplary embodiment, the above apparatus further includes: a third receiving module, configured to receive a cancel export instruction from the target terminal; the cancel export instruction is generated when the target terminal does not receive response information for the export instruction from each target node in the processing device within a preset time period; and a second processing module, configured to terminate the export operation on the target file in response to the cancel export instruction.

[0174] In an exemplary embodiment, the first receiving module 710 includes: a feedback unit, configured to respond to a preparation instruction from the target terminal by feeding back confirmation preparation information to the target terminal; and a receiving unit, configured to receive an export instruction from the target terminal; the export instruction is obtained when the target terminal receives confirmation preparation information fed back by each target node in the processing device.

[0175] In an exemplary embodiment, as shown in FIG8, another object list acquisition device is provided, including: a second sending module 810, a second receiving module 820, and a parsing module 830, wherein:

[0176] The second sending module 810 is used to send an export command to the target node of the processing device. The export command includes the target time.

[0177] The second receiving module 820 is used to receive the target file sent by the target node. The target file is obtained by the target node responding to the export instruction, determining the file status of the target file stored by the target node at the target time, and performing an export operation on the target file according to the file status.

[0178] Parsing module 830 is used to parse the target file and obtain a list of target objects.

[0179] In an exemplary embodiment, the above-mentioned apparatus further includes: a third sending module, configured to send a cancel export instruction to the target node that has not provided a response to the export instruction if no response information for the export instruction is received from each target node in the processing device within a preset time period, so that the corresponding target node terminates the export operation on the target file.

[0180] In an exemplary embodiment, the second sending module 810 includes: a second sending unit, configured to send a preparation instruction to each target node of the processing device, so that the corresponding target node provides feedback confirmation preparation information; and a third sending unit, configured to send an export instruction to each target node upon receiving confirmation preparation information from each target node.

[0181] In an exemplary embodiment, the parsing module 830 includes: a parsing unit, configured to parse the target files corresponding to each target node to obtain an initial metadata list; the initial metadata list includes data information of at least one metadata, the data information including the data identifier and file version information of the corresponding metadata; a removal unit, configured to remove invalid metadata from the initial metadata list according to the data identifier and file version information to obtain a target metadata list; and a determination unit, configured to determine a target object list according to the target metadata list.

[0182] In an exemplary embodiment, the elimination unit includes: an acquisition subunit, used to acquire multiple candidate metadata elimination subunits corresponding to the same data identifier in the initial metadata list, used to eliminate invalid metadata from the multiple candidate metadata, wherein invalid metadata refers to other candidate metadata other than the latest file version information in the multiple candidate metadata.

[0183] Each module in the above-mentioned object list acquisition method device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device in hardware form, or stored in the memory of a computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0184] In an exemplary embodiment, a computer device is provided, which may be a server, and its internal structure diagram is shown in Figure 9. The computer device includes a processor, memory, input / output interfaces (I / O), and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is connected to the system bus via the I / O interfaces. The processor of the computer device provides computing and control capabilities. The memory of the computer device includes non-volatile storage media and internal memory. The non-volatile storage media stores an operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The database of the computer device stores object files. The I / O interfaces of the computer device are used for exchanging information between the processor and external devices. The communication interface of the computer device is used for communicating with external terminals via a network connection. When the computer program is executed by the processor, it implements a method for obtaining an object list.

[0185] Those skilled in the art will understand that the structure shown in Figure 9 is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or may combine certain components, or may have different component arrangements.

[0186] In one embodiment, a computer device is also provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above method embodiments.

[0187] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon that, when executed by a processor, implements the steps in the above method embodiments.

[0188] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above method embodiments.

[0189] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, artificial intelligence (AI) processors, etc., and are not limited to these.

[0190] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0191] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A method for obtaining an object list, characterized in that, The method, applied to a target node of a processing device, includes: receiving an export instruction from a target terminal, the export instruction including a target time; responding to the export instruction, determining the file status of a target file stored on the target node at the target time, and performing an export operation on the target file according to the file status; and sending the exported target file to the target terminal so that the target terminal can parse the target file to obtain a list of target objects.

2. The method according to claim 1, characterized in that, The step of performing an export operation on the target file based on the file status includes: if, at the target time, the file status indicates that the target file is being written by the target writing end, then sending an error message to the target writing end, so that the target writing end responds to the error message, cancels the writing of the target file, and continues writing to other files of the target node; and performing an export operation on the target file.

3. The method according to claim 1, characterized in that, The step of performing an export operation on the target file based on the file status includes: if the file status indicates that the target file is being deleted at the target time, then performing delayed deletion processing on the target file; performing an export operation on the target file; and deleting the target file stored in the target node after the export operation is completed.

4. The method according to any one of claims 1-3, characterized in that, The method further includes: receiving a cancel export instruction from the target terminal; the cancel export instruction is generated when the target terminal does not receive response information for the export instruction from each target node in the processing device within a preset time period; and terminating the export operation on the target file in response to the cancel export instruction.

5. The method according to any one of claims 1-3, characterized in that, The step of receiving the export instruction from the target terminal includes: responding to the preparation instruction from the target terminal by sending confirmation preparation information back to the target terminal; and receiving the export instruction from the target terminal; wherein the export instruction is obtained when the target terminal receives confirmation preparation information from each target node in the processing device.

6. A method for obtaining an object list, characterized in that, Applied to a target terminal, the method includes: sending an export instruction to a target node of a processing device, the export instruction including a target time; receiving a target file sent by the target node, the target file being obtained by the target node in response to the export instruction, determining the file status of the target file stored by the target node at the target time, and performing an export operation on the target file according to the file status; and parsing the target file to obtain a list of target objects.

7. The method according to claim 6, characterized in that, The method further includes: if no response information for the export instruction is received from each target node in the processing device within a preset time period, sending a cancel export instruction to the target node that did not provide the response information, so that the corresponding target node terminates the export operation on the target file.

8. The method according to claim 6, characterized in that, Sending export instructions to the target nodes of the processing device includes: sending preparation instructions to each target node of the processing device so that the corresponding target node can provide confirmation of preparation information; and sending export instructions to each target node upon receiving confirmation of preparation information from each target node.

9. The method according to any one of claims 6-8, characterized in that, The step of parsing the target file to obtain the target object list includes: parsing the target file corresponding to each target node to obtain an initial metadata list; the initial metadata list includes data information of at least one metadata, the data information including the data identifier and file version information of the corresponding metadata; removing invalid metadata from the initial metadata list according to the data identifier and file version information to obtain a target metadata list; and determining the target object list according to the target metadata list.

10. The method according to claim 9, characterized in that, The step of removing invalid metadata from the initial metadata list based on the data identifier and file version information includes: obtaining multiple candidate metadata corresponding to the same data identifier in the initial metadata list; removing invalid metadata from the multiple candidate metadata, wherein the invalid metadata is other candidate metadata other than the latest file version information in the multiple candidate metadata.

11. An apparatus for obtaining an object list, characterized in that, The device, configured on a target node of a processing device, includes: a first receiving module for receiving an export instruction from a target terminal, the export instruction including a target time; a first processing module for responding to the export instruction, determining the file status of a target file stored on the target node at the target time, and performing an export operation on the target file according to the file status; and a first sending module for sending the exported target file to the target terminal, so that the target terminal can parse the target file to obtain a list of target objects.

12. An apparatus for obtaining an object list, characterized in that, Configured on a target terminal, the device includes: a second sending module for sending an export instruction to a target node of a processing device, the export instruction including a target time; a second receiving module for receiving a target file sent by the target node, the target file being obtained by the target node in response to the export instruction, determining the file status of the target file stored by the target node at the target time, and performing an export operation on the target file according to the file status; and a parsing module for parsing the target file to obtain a list of target objects.

13. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 10.

14. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 10.

15. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 10.

Images