Data interaction method and device, computer equipment and storage medium

Abstract

The invention relates to a data interaction method and device, computer equipment, a storage medium and a computer program product. The method comprises the steps that in response to an interaction request for target data in a storage cluster, under the condition that the metadata center is abnormal, identification information of the target data carried in the interaction request is utilized, matching target merged file identification information corresponding to the identification information from backup data corresponding to the metadata center; determining metadata of the target data by utilizing record information corresponding to the merged file identification information; wherein the record information comprises identification information of data stored in the merged file and metadata of the stored data; and carrying out interaction operation on the target data in the storage cluster according to the metadata of the target data. By adopting the method, the storage cluster can still keep quick access and interaction with the target data even under the condition that the metadata center is invalid.

Description

Technical Field

[0001] The present application relates to the field of big data technology, and in particular to a data interaction method, apparatus, computer equipment, storage medium and computer program product. Background Art

[0002] With the development of big data technology, the concept of metadata has been widely used. Metadata mainly records index information such as the size, upload time, and location of the data. Therefore, the target data can be accurately found through metadata, thereby achieving efficient interaction with the target data. At present, in order to facilitate the storage and use of metadata, data storage gateways usually manage all metadata through a relational database.

[0003] However, if the relational database storing metadata fails and becomes unavailable, the metadata of the entire system will be inaccessible, which greatly affects the retrieval and use of data. In the prior art, metadata is usually backed up so that the backed-up metadata can be used for timely retrieval and use when relational database fails. However, the metadata backup process consumes a lot of storage resources and time, and the integrity of the backup cannot be guaranteed, so it cannot meet actual needs. Summary of the invention

[0004] Based on this, it is necessary to provide a data interaction method, apparatus, computer device, computer-readable storage medium and computer program product to address the above technical issues.

[0005] In a first aspect, the present application provides a data interaction method. The method comprises:

[0006] In response to an interactive request for target data in the storage cluster, when an abnormality exists in the metadata center, using identification information of the target data carried in the interactive request, matching target merged file identification information corresponding to the identification information from backup data corresponding to the metadata center;

[0007] Determine the metadata of the target data by using the record information corresponding to the identification information of the merged file; wherein the record information includes the identification information of the data stored in the merged file and the metadata of the stored data;

[0008] An interactive operation is performed on the target data in the storage cluster according to the metadata of the target data.

[0009] In one of the embodiments, when an abnormality exists in the metadata center, using the identification information of the target data carried in the interaction request to match the target merged file identification information corresponding to the identification information from the backup data corresponding to the metadata center includes:

[0010] In the case where there is an abnormality in the metadata center, determining the interaction type of the interaction request;

[0011] In the case where the interaction type is an incremental read / write request, switching the interaction address of the interaction request to a preset target address, and performing a corresponding interaction operation;

[0012] In the case where the interaction type is an existing read / write request, the identification information of the target data carried in the interaction request is used to match the target merged file identification information corresponding to the identification information from the backup data corresponding to the metadata center.

[0013] In one embodiment, the method further comprises:

[0014] In the case that the merged file identification information is not matched, each data pool in each storage cluster is traversed in turn to find the data file corresponding to the target data, and a corresponding interactive operation is performed on the target data.

[0015] In one embodiment, the target data includes a single data or multiple data, and traversing each data pool in each storage cluster in turn to find the data file corresponding to the target data includes:

[0016] Performing full permutation processing on each data pool in each storage cluster to reach multiple queues, and establishing a traversal thread for each queue;

[0017] The data files corresponding to the target data are searched by using the traversal threads.

[0018] In one embodiment, generating backup data corresponding to the metadata center includes:

[0019] Regularly obtaining the association relationship between each data and the merged file from the metadata center;

[0020] The association relationship is stored in preset static metadata.

[0021] In one embodiment, generating the merged file includes:

[0022] Get the data files to be merged;

[0023] Storing the data in the data file into a target merged file, and establishing record information of the target merged file;

[0024] When the amount of data stored in the merged file reaches a preset threshold, the merged file is stored in a preset storage cluster.

[0025] In a second aspect, the present application also provides a data interaction device. The device comprises:

[0026] A merge file matching module is used to respond to an interactive request for target data in the storage cluster, and in the case that there is an abnormality in the metadata center, use the identification information of the target data carried in the interactive request to match the target merge file identification information corresponding to the identification information from the backup data corresponding to the metadata center;

[0027] A metadata determination module, used to determine the metadata of the target data using the record information corresponding to the merged file identification information; wherein the record information includes the identification information of the data stored in the merged file and the metadata of the stored data;

[0028] The interactive operation module is used to perform interactive operations on the target data in the storage cluster according to the metadata of the target data.

[0029] In one embodiment, the merged file matching module includes:

[0030] An interaction type determination submodule, used to determine the interaction type of the interaction request when an abnormality exists in the metadata center;

[0031] An interactive operation submodule, used for switching the interactive address of the interactive request to a preset target address and performing corresponding interactive operations when the interactive type is an incremental read / write request;

[0032] The merged file matching submodule is used to match the target merged file identification information corresponding to the identification information from the backup data corresponding to the metadata center using the identification information of the target data carried in the interaction request when the interaction type is an existing read and write request.

[0033] In one embodiment, the interactive operation module is further used to:

[0034] In the case that the merged file identification information is not matched, each data pool in each storage cluster is traversed in turn to find the data file corresponding to the target data, and a corresponding interactive operation is performed on the target data.

[0035] In one embodiment, the target data includes a single or multiple data, and the interactive operation module includes:

[0036] The thread establishment submodule is used to perform full permutation processing on each data pool in each storage cluster to achieve multiple queues, and establish a traversal thread for each queue;

[0037] The file search submodule is used to use each of the traversal threads to search for the data file corresponding to each of the target data.

[0038] In one embodiment, the device further comprises:

[0039] An association relationship acquisition module, used for periodically acquiring the association relationship between each data and the merged file from the metadata center;

[0040] The association relationship storage module is used to store the association relationship in preset static metadata.

[0041] In one embodiment, the device further comprises:

[0042] A data file module, used to obtain the data files to be merged;

[0043] A data file merging module, used for storing the data in the data file into a target merged file and establishing record information of the target merged file;

[0044] The merged file storage module is used to store the merged file in a preset storage cluster when the amount of data stored in the merged file reaches a preset threshold.

[0045] In a third aspect, the present application further provides a computer device, wherein the computer device comprises a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the data interaction method as described in any one of the embodiments of the present disclosure is implemented.

[0046] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the data interaction method as described in any one of the embodiments of the present disclosure is implemented.

[0047] In a fifth aspect, the present application further provides a computer program product, wherein the computer program product comprises a computer program, and when the computer program is executed by a processor, the data interaction method as described in any one of the embodiments of the present disclosure is implemented.

[0048] The above-mentioned data interaction method, apparatus, computer equipment, storage medium and computer program product, when an abnormality occurs in the metadata center, matches the merged file identifier corresponding to the interaction request, and uses the record information corresponding to the merged file identifier to determine the metadata of the target data, and performs interactive operations on the target data in the storage cluster based on the metadata. In the event of an abnormality in the metadata center, the target data identification information carried in the interaction request is used to match the identification information of the target merged file from the backup data and based on the record information of the merged file identifier, the metadata of the target data can be effectively restored and identified. This method ensures that even if the metadata center fails, the storage cluster can still maintain rapid access to and interaction with the target data, improves the reliability and data recovery capabilities of the system, and ensures business continuity and data integrity. At the same time, by using the record information to recover the metadata, the risk of data loss is reduced, providing a safer guarantee for storage management. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] FIG1 is a schematic diagram of a flow chart of a data interaction method in one embodiment;

[0050] FIG2 is a schematic diagram of a process of using record information to construct metadata and read target data in one embodiment;

[0051] FIG3 is a schematic diagram of a process flow of processing different interaction types in one embodiment;

[0052] FIG4 is a schematic diagram of a process of traversing various data pools in one embodiment;

[0053] FIG5 is a schematic diagram of a process for establishing a full arrangement in one embodiment;

[0054] FIG6 is a schematic diagram of a full arrangement result in one embodiment;

[0055] FIG7 is a schematic diagram of a process of backing up data in one embodiment;

[0056] FIG8 is a schematic diagram of a process of creating a merged file in one embodiment;

[0057] FIG9 is a schematic diagram of a process of creating and persisting a merged file in one embodiment;

[0058] FIG10 is a structural block diagram of a data interaction device in one embodiment;

[0059] FIG11 is a structural block diagram of a data interaction device implemented in one embodiment;

[0060] FIG. 12 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0061] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0062] In one embodiment, as shown in FIG1 , a data interaction method is provided. This embodiment is illustrated by applying the method to a terminal. It is understandable that the method can also be applied to a server, and can also be applied to a system including a terminal and a server, and is implemented through the interaction between the terminal and the server. In this embodiment, the method includes the following steps:

[0063] Step S100, in response to an interactive request for target data in a storage cluster, in the event that an abnormality exists in a metadata center, using identification information of the target data carried in the interactive request, matching target merged file identification information corresponding to the identification information from backup data corresponding to the metadata center.

[0064] In an exemplary embodiment, the metadata center may include managing all metadata through a relational database, and the relational database is the metadata center, etc.

[0065] In an exemplary embodiment, the backup data corresponding to the metadata center may include the association relationship between the target data and the merged file identifier, etc.; in actual use, in order to occupy less memory, it may only include the association relationship between the data and the merged file identifier, and the association relationship may be stored in static metadata, etc.

[0066] In an exemplary embodiment, the merged file may be obtained by merging data files; in actual use, data may be obtained from the data files and stored in the merged file.

[0067] Step S200, determining the metadata of the target data using the record information corresponding to the merged file identification information; wherein the record information includes the identification information of the data stored in the merged file and the metadata of the stored data.

[0068] In an exemplary embodiment, the record information may include an identifier of the merged data file, an identifier of the merged file, an offset and a length of the data file in the merged file, etc.; it can be understood that the target data can be found from the merged file through the offset and the length; wherein the offset and the length are metadata of the stored data, etc.

[0069] In an exemplary embodiment, the method of determining metadata using record information can be implemented through a flowchart as shown in FIG. 2 , which specifically includes step S901, rebuilding the index metadata center based on the name of the data file in the record information (the file before the target data is merged), the name of the merged file, the offset of the data file in the merged file, and the length information of the data file; and step S902, the object storage gateway performs normal reading of the merged target data based on the rebuilt index metadata.

[0070] Step S300: performing an interactive operation on the target data in the storage cluster according to the metadata of the target data.

[0071] In the above data interaction method, in the event of an abnormality in the metadata center, the merged file identifier corresponding to the interaction request is matched, and the metadata of the target data is determined by using the record information corresponding to the merged file identifier, and the target data in the storage cluster is interactively operated based on the metadata. In the event of an abnormality in the metadata center, the target data identification information carried in the interaction request is used to match the identification information of the target merged file from the backup data and based on the record information of the merged file identifier, the metadata of the target data can be effectively restored and identified. This method ensures that even in the event of a failure of the metadata center, the storage cluster can still maintain rapid access to and interaction with the target data, thereby improving the reliability and data recovery capabilities of the system and ensuring the business continuity and data integrity of the user. At the same time, by using the record information to recover the metadata, the risk of data loss is reduced, providing a safer guarantee for storage management.

[0072] In one embodiment, as shown in FIG3 , when the metadata center is abnormal, using the identification information of the target data carried in the interaction request, matching the target merged file identification information corresponding to the identification information from the backup data corresponding to the metadata center includes:

[0073] Step S101: when there is an abnormality in the metadata center, determine the interaction type of the interaction request.

[0074] Step S102, when the interaction type is an incremental read / write request, the interaction address of the interaction request is switched to a preset target address, and a corresponding interaction operation is performed.

[0075] In an exemplary embodiment, when the interaction type is an incremental read / write request, it can be considered that there is no need to find the target data, only the target data needs to be increased, etc. Therefore, in this case, the interaction address of the interaction request can be switched to a preset healthy resource pool, etc.

[0076] Step S103, when the interaction type is an existing read / write request, using the identification information of the target data carried in the interaction request, matching the target merged file identification information corresponding to the identification information from the backup data corresponding to the metadata center.

[0077] In an exemplary embodiment, the existing read and write request may include the need to find the target data and perform corresponding interactive operations on the target data. Therefore, the target data must be found in advance, so the target merge file identification information corresponding to the identification information is matched with the corresponding backup data from the metadata center, etc.

[0078] In this embodiment, in the event of an abnormality in the metadata center, the interaction strategy can be flexibly adjusted by identifying the interaction type of the interaction request. For incremental read and write requests, switch to the preset target address and perform corresponding interactive operations to ensure the system's immediate responsiveness and business continuity. For existing read and write requests, the target merge file identification information in the backup data can be used to accurately locate and restore the target data, thereby improving the reliability of data access. This flexible response method not only optimizes the data interaction process and improves the robustness of the system, but also effectively reduces the risk of potential data access interruptions caused by metadata center failures, ensuring the smoothness of data interaction.

[0079] In one embodiment, the method further comprises:

[0080] In the case that the merged file identification information is not matched, each data pool in each storage cluster is traversed in turn to find the data file corresponding to the target data, and a corresponding interactive operation is performed on the target data.

[0081] In an exemplary embodiment, the target data is stored in a storage cluster. In the absence of metadata and the target data has no corresponding merged file identification information, the storage clusters are traversed to search for the target data in sequence. It can be understood that the target data can be quickly found through metadata, and the target data can also be found by traversing each data pool in each storage cluster.

[0082] In this embodiment, when the target merge file identification information cannot be matched, the data pools in each storage cluster are traversed in turn to find the data files corresponding to the target data, which can ensure the comprehensiveness and reliability of data access. This method reduces the risk of not being able to find the target data due to missing or faulty metadata by systematically scanning all possible data sources. In addition, this strategy ensures that even in a complex storage environment, the required interactive operations can be obtained, maintaining data availability and business continuity, thereby improving the stability of the overall system.

[0083] In one embodiment, as shown in FIG. 4 , the target data includes a single data or multiple data, and the step of sequentially traversing each data pool in each storage cluster to search for a data file corresponding to the target data includes:

[0084] Step S401 , performing full permutation processing on each data pool in each storage cluster to obtain multiple queues, and establishing a traversal thread for each queue.

[0085] In an exemplary embodiment, there may be multiple storage clusters, each storage cluster may correspond to a single or multiple data pools, the target data is stored in the data pool, and the data pool is fully sorted, wherein each sorting result is equivalent to a traversal path, etc.; the generation method of the full arrangement (traversal path) may adopt the following two methods, namely recursion and dynamic programming, wherein dynamic programming may include a strategy of space-for-time by saving calculation results to avoid repeated calculations, which reduces the time complexity compared to the recursive algorithm. The algorithm for generating the traversal path may include first selecting a data pool, then inserting each position of the data pool into another data pool, and so on, until all data pools are traversed. For example, there are two storage clusters, C1 and C2; cluster C1 includes two data pools, P1 and P2; C2 includes a data pool, P1; at this time, there are three data pools, namely C1P1, C1P2, and C2P1; the process of generating a traversal path can be implemented as shown in FIG5, step S801, determine whether all data pools participate in the arrangement, if yes, proceed to step S804; if not, proceed to step S802; step S802, select a data pool and insert it into each position of the existing arrangement; step S803, generate a new polling path; step S804, output all polling paths; specifically, the starting polling path has only one pool. That is, C1P1. In the second round of insertion, C1P2 is inserted before and after C1P1, which becomes two combinations of (C1P1, C1P2) and (C1P2, C1P1). The third insertion is to insert C2P1 before, in the middle and after (C1P1, C1P2), and obtain (C2P1, C1P1, C1P2), (C1P1, C2P1, C1P2), (C1P1, C1P2, C2P1), and so on to generate all polling paths, etc.; the full arrangement result is obtained as shown in Figure 6.

[0086] Step S402: using each of the traversal threads to search for a data file corresponding to each of the target data.

[0087] In an exemplary embodiment, a thread is established for each queue, and when the data pool needs to be traversed, the traversal of the target data can be evenly distributed to different threads. It can be understood that this method can ensure that the amount of tasks assigned to each thread is balanced, and all threads correspond to the full sorting of the data pool, so the amount of tasks for all data pools can be balanced, avoiding multiple traversals of a data pool and causing the rate to be too slow.

[0088] In this embodiment, the data pools in each storage cluster are fully sorted and multiple traversal threads are established, which can significantly improve the load balancing effect of the system. This method reasonably distributes the tasks of each thread through parallel processing, thereby avoiding the bottleneck caused by a single thread in the traversal process. Thread load balancing ensures that each thread processes different data pools evenly, improving the search efficiency. At the same time, since all threads correspond to the full sorting of all data pools, the balanced load of each thread will further ensure the load balancing of each data pool, reducing the performance degradation caused by overload of a certain data pool. This efficient parallel processing strategy not only speeds up the search speed of target data, but also improves the responsiveness and reliability of the overall system.

[0089] In one embodiment, as shown in FIG. 7 , generating backup data corresponding to the metadata center includes:

[0090] Step S111 : periodically obtaining the association relationship between each data and the merged file from the metadata center.

[0091] Step S112: storing the association relationship in preset static metadata.

[0092] In an exemplary embodiment, the data stored in the static metadata may include data that does not usually change with the dynamic changes of the data. Static metadata describes the characteristics of the data, such as data type, data format, data relationship, constraint conditions, etc., but does not store the data itself. Static metadata is usually stored in a database or file system to improve the efficiency of data management and query, etc.

[0093] In this embodiment, the association relationship between each data and the merged file is regularly obtained from the metadata center and stored in the preset static metadata, which can greatly enhance the transparency and management efficiency of the system for the data structure. By maintaining the latest association relationship, this strategy ensures that the system can quickly identify and access data, thereby reducing the delay in data retrieval. The use of static metadata can also provide reliable backup and access paths when the metadata center fails, ensuring the continuity and consistency of the data. In addition, regularly updating the association relationship can help maintain the flexibility of the system during data operations and changes, optimize the storage management process, and improve the security and reliability of data operations.

[0094] In one embodiment, as shown in FIG8 , generating the merged file includes:

[0095] Step S121, obtaining the data files to be merged.

[0096] In an exemplary embodiment, the data files to be merged may include write-amplified data files, which may include a number of times the amount of physical data actually written being the amount of written data, that is, there is only a small amount of valid data and a large number of placeholders, so that the data file occupies a large amount of space but stores only a small amount of data.

[0097] Step S122, storing the data in the data file into a target merged file, and establishing record information of the target merged file.

[0098] In an exemplary embodiment, the record information may include log information of the merged file, which may specifically include adding the data file name, the name of the merged file, and the offset and length of the data file to the log information when merging the files.

[0099] Step S123: When the amount of data stored in the merged file reaches a preset threshold, the merged file is stored in a preset storage cluster.

[0100] In an exemplary embodiment, the amount of data stored in the merged file reaches a preset threshold value, which may include that the amount of data stored in the merged file reaches the maximum storage capacity of the merged file. In another exemplary embodiment, by receiving data of the next data file, if the remaining space in the merged file is insufficient to store the data of the data file, it can be considered that the amount of data stored in the merged file reaches the preset threshold value, and the merged file can be stored in a preset storage cluster, etc.

[0101] In an exemplary embodiment, the generation of the merged file may adopt the flowchart shown in FIG9 , which may specifically include:

[0102] Step S911, uploading data files;

[0103] Step S912, determining whether the length of the merged file after merging reaches a threshold; if so, proceeding to step S915, if not, proceeding to step S913;

[0104] Step S913, merging the data files into a merged file in the memory, and returning information such as the data file name, the merged file name, the data file offset, and the length of the data file;

[0105] Step S914, the access layer component records the returned message into the log;

[0106] Step S915, persisting the merged file in the memory to the underlying storage cluster.

[0107] In this embodiment, by obtaining the data files to be merged and effectively storing the data therein into the target merged file, and establishing the record information of the target merged file, this process can systematically manage the data integration process. When the amount of data stored in the merged file reaches a preset threshold, it is stored in a preset storage cluster, which can significantly improve storage efficiency and resource utilization. This method not only ensures the orderly and structured storage of data, but also reduces the performance loss caused by frequent writing, thereby improving the response speed and expansion capability of the overall system. In addition, centralized storage after reaching the threshold helps to optimize data archiving and backup strategies, ensure data security and availability, and support efficient data retrieval and management.

[0108] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0109] Based on the same inventive concept, the embodiment of the present application also provides a data interaction device for implementing the data interaction method involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in one or more data interaction device embodiments provided below can refer to the limitations on the data interaction method above, and will not be repeated here.

[0110] In one embodiment, as shown in FIG. 10 , a data interaction device 100 is provided, comprising: a merged file matching module 101, a metadata determination module 102 and an interaction operation module 103, wherein:

[0111] A merge file matching module is used to respond to an interactive request for target data in the storage cluster, and in the case that there is an abnormality in the metadata center, use the identification information of the target data carried in the interactive request to match the target merge file identification information corresponding to the identification information from the backup data corresponding to the metadata center;

[0112] A metadata determination module, used to determine the metadata of the target data using the record information corresponding to the merged file identification information; wherein the record information includes the identification information of the data stored in the merged file and the metadata of the stored data;

[0113] The interactive operation module is used to perform interactive operations on the target data in the storage cluster according to the metadata of the target data.

[0114] In one embodiment, the merged file matching module includes:

[0115] An interaction type determination submodule, used to determine the interaction type of the interaction request when an abnormality exists in the metadata center;

[0116] An interactive operation submodule, used for switching the interactive address of the interactive request to a preset target address and performing corresponding interactive operations when the interactive type is an incremental read / write request;

[0117] The merged file matching submodule is used to match the target merged file identification information corresponding to the identification information from the backup data corresponding to the metadata center using the identification information of the target data carried in the interaction request when the interaction type is an existing read and write request.

[0118] In one embodiment, the interactive operation module is further used to:

[0119] In the case that the merged file identification information is not matched, each data pool in each storage cluster is traversed in turn to find the data file corresponding to the target data, and a corresponding interactive operation is performed on the target data.

[0120] In one embodiment, the target data includes a single or multiple data, and the interactive operation module includes:

[0121] The thread establishment submodule is used to perform full permutation processing on each data pool in each storage cluster to achieve multiple queues, and establish a traversal thread for each queue;

[0122] The file search submodule is used to use each of the traversal threads to search for the data file corresponding to each of the target data.

[0123] In one embodiment, the device further comprises:

[0124] An association relationship acquisition module, used for periodically acquiring the association relationship between each data and the merged file from the metadata center;

[0125] The association relationship storage module is used to store the association relationship in preset static metadata.

[0126] In one embodiment, the device further comprises:

[0127] A data file module, used to obtain the data files to be merged;

[0128] A data file merging module, used for storing the data in the data file into a target merged file and establishing record information of the target merged file;

[0129] The merged file storage module is used to store the merged file in a preset storage cluster when the amount of data stored in the merged file reaches a preset threshold.

[0130] In an exemplary embodiment, the data interaction device may be applied to an object storage gateway, and may be specifically applied to the schematic diagram shown in FIG. 11 , which specifically includes:

[0131] The object storage gateway cannot obtain metadata from the metadata center; the object storage gateway includes multiple storage clusters, each storage cluster includes multiple data pools; the static metadata such as users, buckets, clusters, etc. in the metadata center are backed up regularly; the object storage gateway can interact with the access layer components, and the access layer components can obtain the logs of the merged files; the logs of the merged files are formatted and archived, and the metadata of the data stored in the merged files can be further reconstructed and provided to the object storage gateway for use by the gateway, etc.

[0132] Each module in the above data interaction device can be implemented in whole or in part by software, hardware, or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to each module.

[0133] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be shown in FIG12. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. Among them, the processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store metadata. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a data interaction method is implemented.

[0134] Those skilled in the art will understand that the structure shown in FIG. 12 is merely a block diagram of a partial structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different arrangement of components.

[0135] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards of relevant countries and regions.

[0136] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited to this.

[0137] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0138] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A data interaction method, characterized in that: The method comprises: In response to an interactive request for target data in the storage cluster, when an abnormality exists in the metadata center, using identification information of the target data carried in the interactive request, matching target merged file identification information corresponding to the identification information from backup data corresponding to the metadata center; Determine the metadata of the target data by using the record information corresponding to the identification information of the merged file; wherein the record information includes the identification information of the data stored in the merged file and the metadata of the stored data; An interactive operation is performed on the target data in the storage cluster according to the metadata of the target data.

2. The method according to claim 1, characterized in that: In the case that the metadata center is abnormal, using the identification information of the target data carried in the interaction request to match the target merged file identification information corresponding to the identification information from the backup data corresponding to the metadata center, includes: In the case where there is an abnormality in the metadata center, determining the interaction type of the interaction request; In the case where the interaction type is an incremental read / write request, switching the interaction address of the interaction request to a preset target address, and performing a corresponding interaction operation; In the case where the interaction type is an existing read / write request, the identification information of the target data carried in the interaction request is used to match the target merged file identification information corresponding to the identification information from the backup data corresponding to the metadata center.

3. The method according to claim 1, characterized in that The method further comprises: In the case that the merged file identification information is not matched, each data pool in each storage cluster is traversed in turn to find the data file corresponding to the target data, and a corresponding interactive operation is performed on the target data.

4. The method according to claim 3, characterized in that The target data may be single or multiple, and the step of sequentially traversing each data pool in each storage cluster to search for a data file corresponding to the target data includes: Performing full permutation processing on each data pool in each storage cluster to reach multiple queues, and establishing a traversal thread for each queue; The data files corresponding to the target data are searched by using the traversal threads.

5. The method according to claim 1, characterized in that Generating backup data corresponding to the metadata center includes: Regularly obtaining the association relationship between each data and the merged file from the metadata center; The association relationship is stored in preset static metadata.

6. The method according to claim 1, characterized in that Generate the merged file, including: Get the data files to be merged; Storing the data in the data file into a target merged file, and establishing record information of the target merged file; When the amount of data stored in the merged file reaches a preset threshold, the merged file is stored in a preset storage cluster.

7. A data interaction device, characterized in that: The device comprises: A merge file matching module is used to respond to an interactive request for target data in the storage cluster, and in the case that there is an abnormality in the metadata center, use the identification information of the target data carried in the interactive request to match the target merge file identification information corresponding to the identification information from the backup data corresponding to the metadata center; A metadata determination module, used to determine the metadata of the target data using the record information corresponding to the merged file identification information; wherein the record information includes the identification information of the data stored in the merged file and the metadata of the stored data; The interactive operation module is used to perform interactive operations on the target data in the storage cluster according to the metadata of the target data.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images