Data management method and device, equipment, medium and program product

Abstract

The invention relates to the technical field of big data, and discloses a data management method and device, equipment, a medium and a program product. The method comprises the following steps: when a data reading request of a user is detected, obtaining a read lock; whether target stored metadata matched with the data reading request exists in a database or not is judged based on the read lock, if yes, the read lock is asynchronously called, and when it is determined that the stored metadata in the database are different from metadata in a memory, a write lock is obtained; performing metadata updating on the database according to the metadata in the memory based on the write lock, and releasing the write lock after updating is completed; and sending the target stored metadata to the user, and releasing the read lock. According to the scheme of the embodiment, the read-write lock is adopted for read-write control, and the data management efficiency can be improved.

Description

Technical Field

[0001] The present invention relates to the field of big data technology, and in particular to a data management method, device, equipment, medium and program product. Background Art

[0002] With the rapid development of information technology, enterprises and organizations have accumulated a large number of heterogeneous data sources, including relational databases, NoSQL databases, distributed file systems, and real-time streaming data. When integrating these external data sources, existing big data platforms face problems such as inconsistent data formats, complex data management, and low data query efficiency.

[0003] Currently, existing data management methods usually use dedicated data management tools to achieve data integration and data management. However, dedicated data management tools may be complex for non-technical users and may have limitations when processing real-time data streams and large-scale concurrent tasks. Summary of the invention

[0004] The present invention provides a data management method, apparatus, device, medium and program product, which can realize efficient integration and management of heterogeneous data sources, reduce data management complexity and improve data management efficiency.

[0005] According to one aspect of the present invention, there is provided a data management method, comprising:

[0006] When a user's data read request is detected, a read lock is acquired;

[0007] Based on the read lock, it is determined whether there is target metadata stored in the database that matches the data read request. If so, the read lock is asynchronously called to acquire a write lock when it is determined that the metadata stored in the database is different from the metadata in the memory;

[0008] Based on the write lock, updating the metadata of the database according to the metadata in the memory, and releasing the write lock after the update is completed;

[0009] The target stored metadata is sent to the user, and the read lock is released.

[0010] According to another aspect of the present invention, there is provided a data management device, comprising:

[0011] A data read request detection module is used to acquire a read lock when a user's data read request is detected;

[0012] A matching metadata judgment module is used to judge whether there is target stored metadata matching the data read request in the database based on the read lock, and if so, acquire a write lock by asynchronously calling the read lock and when it is determined that the metadata stored in the database is different from the metadata in the memory;

[0013] A metadata update module, configured to update the metadata of the database according to the metadata in the memory based on the write lock, and release the write lock after the update is completed;

[0014] The metadata sending module is used to send the target stored metadata to the user and release the read lock.

[0015] According to another aspect of the present invention, there is provided an electronic device, the electronic device comprising:

[0016] at least one processor; and

[0017] a memory communicatively connected to the at least one processor; wherein,

[0018] The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the data management method described in any embodiment of the present invention.

[0019] According to another aspect of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a computer program, and the computer program is used to enable a processor to implement the data management method described in any embodiment of the present invention when executed.

[0020] According to another aspect of the present invention, a computer program product is provided, comprising a computer program, wherein when the computer program is executed by a processor, the data management method according to any embodiment of the present invention is implemented.

[0021] The technical solution of the embodiment of the present invention is as follows: when a user's data read request is detected, a read lock is acquired; based on the read lock, it is determined whether there is target stored metadata matching the data read request in the database; if so, a write lock is acquired by asynchronously calling the read lock when it is determined that the metadata stored in the database is different from the metadata in the memory; based on the write lock, the database is updated with metadata according to the metadata in the memory, and the write lock is released after the update is completed; the target stored metadata is sent to the user, and the read lock is released; and data management efficiency can be improved by using the read-write lock for read-write control.

[0022] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0024] Figure 1 is a flow chart of a data management method provided according to Embodiment 1 of the present invention;

[0025] Figure 2 is a flow chart of data integration and management provided according to the first embodiment of the present invention;

[0026] Figure 3 A data lineage management process is provided according to the first embodiment of the present invention;

[0027] Figure 4 is a structural diagram of a data management device provided according to Embodiment 2 of the present invention;

[0028] Figure 5 It is a structural schematic diagram of an electronic device for implementing the data management method according to an embodiment of the present invention. DETAILED DESCRIPTION

[0029] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0030] It should be noted that the terms "first", "second", "target", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0031] Embodiment 1

[0032] Figure 1 A flowchart of a data management method is provided for the first embodiment of the present invention. This embodiment is applicable to the case of data synchronization and data management of different data sources. The method can be executed by a data management device, which can be implemented in the form of hardware and / or software. Typically, the data management device can be configured in an electronic device, such as a computer device or a server. Figure 1 As shown, the method includes:

[0033] S110: When a data read request from a user is detected, a read lock is acquired.

[0034] The data read request may be a query request for metadata stored in the database, and may include the database name, table name, etc. that need to be read.

[0035] It should be noted that when the system starts, the corresponding functional modules can be initialized. Specifically, the data synchronization instance can be initialized to prepare to load metadata and other resources. For example, a DataSynchronization object can be created to initialize the data structures related to database management, table management, and column management.

[0036] In this embodiment, first, access to different types of external data sources can be achieved, where the data source can include relational databases (such as MySQL, PostgreSQL, etc.), NoSQL databases (such as MongoDB, Cassandra, etc.), distributed file systems (such as HDFS, etc.), and real-time streaming data (such as Kafka, etc.). In a specific example, the relational database MySQL can be connected, and the data source path and access permissions can be configured to achieve access to data stored in MySQL.

[0037] Then, you can load metadata from external data sources, including information about databases, tables, and columns into memory. For example, you can call a pre-established DataSynchronization object to load metadata from a specified data source (such as MySQL, PostgreSQL, ClickHouse, and OceanBase, etc.), and you can load snapshots and operation logs of metadata through EditLog and Image, and parse them in memory to restore metadata objects such as databases, tables, and columns.

[0038] In this embodiment, the permission configuration can also be synchronized from an external data source to ensure the security of the system by loading user, role and permission information. For example, the PrivilegeManager object can be used to load permission data and load user and role information from an external data source. Secondly, it also supports operations such as adding, deleting and modifying permission information based on the user's permission configuration operations.

[0039] Secondly, this embodiment can adopt a read-write lock mechanism, and when reading or writing metadata, it is necessary to apply for a corresponding read lock or write lock. For example, a ReentrantReadWriteLock object can be initialized to ensure that a read-write lock is used when reading and writing metadata to prevent data competition and inconsistency, and ensure the concurrency safety of metadata operations.

[0040] In a specific example, when data synchronization is performed, metadata, EditLog, and Image are first loaded from different types of external data sources into the memory through the cache of the local device at intervals of a first set period. Then, the memory performs data classification, data lineage analysis, and data table aggregation and splitting on the received metadata, and the managed data can be further stored in a Massively Parallel Processing (MPP) database at intervals of a second set period.

[0041] Specifically, when a user needs to read metadata in an MPP database, the user can enter query conditions in the database management page and initiate a read; when the server detects a data read request, it needs to apply for a read lock in advance for the data read operation.

[0042] S120. Determine whether there is target stored metadata matching the data read request in the database based on the read lock. If so, acquire the write lock by asynchronously calling the read lock and when it is determined that the metadata stored in the database is different from the metadata in the memory.

[0043] Specifically, after successfully acquiring the read lock, the target stored metadata matching the data read request can be searched in the database based on the read lock. If it is determined that the target stored metadata is successfully found, the read lock is asynchronously called to compare all the metadata in the memory with the metadata stored in the database, which can be regarded as an immediate refresh of the metadata in the database. If it is determined that all the metadata in the memory is the same as the metadata stored in the database, the target stored metadata can be directly sent to the user. If it is determined that they are not the same, the write lock can be called back.

[0044] S130. Based on the write lock, update the metadata of the database according to the metadata in the memory, and release the write lock after the update is completed.

[0045] Specifically, after successfully acquiring the write lock, the metadata of the database can be immediately updated using the metadata in the memory based on the write lock, for example, incremental update, full update, etc., and the write lock can be released after the update is completed.

[0046] S140: Send the target stored metadata to the user and release the read lock.

[0047] In this embodiment, after completing the metadata update of the database, the target stored metadata may be sent to the user and the read lock may be released, thereby completing the data reading.

[0048] The technical solution of the embodiment of the present invention is as follows: when a user's data read request is detected, a read lock is acquired; based on the read lock, it is determined whether there is target stored metadata matching the data read request in the database; if so, a write lock is acquired by asynchronously calling the read lock when it is determined that the metadata stored in the database is different from the metadata in the memory; based on the write lock, the database is updated with metadata according to the metadata in the memory, and the write lock is released after the update is completed; the target stored metadata is sent to the user, and the read lock is released; and data management efficiency can be improved by using the read-write lock for read-write control.

[0049] In an optional implementation of this embodiment, after determining whether there is target stored metadata matching the data read request in the database based on the read lock, the following method may also be included:

[0050] If there is no target stored metadata matching the data read request in the database, blocking the read lock and acquiring the write lock;

[0051] If it is determined based on the write lock that the metadata in the cache is updated, the updated metadata is obtained, and the updated metadata is asynchronously written to the memory and the database;

[0052] After the update metadata is written, the write lock is released, and the target stored metadata matching the data read request is acquired from the database based on the read lock;

[0053] The target stored metadata is sent to the user, and the read lock is released.

[0054] In another case, if no target metadata matching the data read request is currently found in the database, the read lock can be blocked, that is, its action is suspended, and the write lock is called. After successfully acquiring the write lock, the metadata in the cache is parsed based on the write lock to determine whether there is a metadata update. For example, the time when the database was last synchronized with the data and the time when the cache received the latest metadata from different data sources can be determined. If the time when the latest metadata was received is later than the time when the database was last synchronized with the data, it can be considered that there is a metadata update.

[0055] If it is determined that the metadata in the cache has been updated, the unsynchronized metadata in the cache can be read based on the write lock, and the parsing method can be called to parse the EditLog and Image in the cache to obtain the unsynchronized parsed data, and the obtained unsynchronized metadata and parsed data can be used as the updated metadata. Then, the updated metadata can be written to the memory based on the write lock, and asynchronously written to the database at the same time. Finally, after the update metadata is written, the pause of the read lock can be released, and the target metadata stored in the database can be searched again based on the read lock. If the target metadata is successfully found, the target metadata stored can be fed back to the user and the read lock can be released. If the data is still not found successfully, the error message that the data does not exist can be fed back to the user.

[0056] Optionally, after blocking the read lock and acquiring the write lock, the following steps may also be included:

[0057] If it is determined based on the write lock that the metadata in the cache has not been updated, a data non-existence error message is generated and the data non-existence error message is sent to the user.

[0058] In this embodiment, if it is detected that the metadata in the cached data has not been updated, that is, there is no unsynchronized metadata, then an error message indicating that the data does not exist may be directly fed back to the user.

[0059] In another optional implementation of this embodiment, the technical solution of this embodiment may further include:

[0060] When a user's data change request is detected, a write lock is acquired, and based on the write lock, a data change operation is performed on the metadata in the memory according to the data change request to acquire the changed metadata;

[0061] Generate a change operation record corresponding to the data change operation, add the change operation record to a file system operation record file, and release the write lock;

[0062] Generate an operation log entry corresponding to the data change operation, and write the operation log entry into the file system operation record file;

[0063] A read lock is acquired, and a snapshot file corresponding to the changed metadata is generated based on the read lock, and the read lock is released.

[0064] In this embodiment, metadata changes need to be persistently stored on disk to ensure that the system can restore all changes when it is restarted or recovers from a fault. Specifically, when a data change request is detected, such as creating or modifying a database, table, or column, a write lock is first applied to ensure concurrency safety, and the metadata change operation corresponding to the data change request is performed in memory based on the write lock to obtain the changed metadata; then, the change operation record can be added to the file system operation record file, such as EditLog, and the write lock is released to allow other operations to continue.

[0065] Secondly, the data change operation can be formatted as an operation log entry and appended to the EditLog. In addition, the log can be periodically refreshed from memory to disk to ensure data persistence. At the same time, a snapshot file corresponding to the change metadata can be created. Specifically, in order to ensure consistency, a read lock is first applied for, and the change metadata is saved as a snapshot file based on the read lock, and then the read lock is released to allow other operations to continue.

[0066] In a specific implementation of this embodiment, the process of data integration and management can be as follows: Figure 2 As shown. Specifically, first, metadata is collected from multiple data sources into the cache. The metadata may include host port, user / password, driver information, jar package, database / table / column, EditLog, and Image, etc. Then, the metadata is loaded from the cache into the memory. The data management in the memory may include library management, table management, and column management. Finally, the managed metadata is loaded from the memory into the MPP database. The read and write operations on metadata are based on read-write locks.

[0067] Optionally, after releasing the read lock, the following steps may also be included:

[0068] When a metadata recovery request is detected, the latest snapshot file and the corresponding generation time point are obtained;

[0069] The operation log entries after the generation time point are obtained, and metadata recovery is implemented according to the latest snapshot file and the operation log entries after the generation time point.

[0070] In this embodiment, when the system is restarted or fails, the metadata can be restored through the snapshot and operation log. Specifically, first load the latest snapshot file and all operation log entries after the snapshot file; then, start playing back from the operation log entries after the snapshot file, and apply all change operations to the metadata corresponding to the snapshot file to achieve metadata recovery.

[0071] Optionally, this embodiment may also use a TransactionManager object to recover an unfinished transaction, read a persistent transaction log, recover an unfinished transaction state, and handle possible transaction conflicts and recovery failures.

[0072] In this embodiment, when the system starts, various configurations can be initialized. After the system is started, the front-end service can be started to listen to user requests and route the user requests to the corresponding module for processing. When a user requests a query, after passing the corresponding permission management layer, the user's query will be forwarded to the query optimization layer through the user interface layer. At this time, the system will asynchronously call the MetadataSynchronizer object of metadata synchronization to synchronize metadata in the cluster. After ensuring that all nodes are synchronized, the query plan is generated using the latest metadata.

[0073] The advantage of the above settings is that it can achieve quasi-real-time metadata synchronization, integrate external data sources, efficiently process and persist user metadata updates, ensure data consistency and query optimization, and achieve efficient data synchronization while occupying a small amount of memory, significantly improving data processing and utilization efficiency.

[0074] Optionally, the technical solution of this embodiment may further include:

[0075] Parsing the metadata stored in the database to identify the dependency relationships between data entities, and establishing a data lineage relationship diagram based on the dependency relationships between the data entities;

[0076] According to the request for viewing the data lineage relationship, the data lineage relationship diagram is displayed on the front-end page.

[0077] In this embodiment, data management mainly includes data classification and data lineage, which can be implemented by corresponding functional modules. Data classification facilitates the organization, management and discovery of data, while data lineage ensures the transparency of data flow, tracks the path of data from source to target, and helps users understand data flow and dependencies.

[0078] Specifically, when managing data lineage, first, data extraction and data change are performed. For example, data extraction can be connected to the data source through the DatahubIngestion object, metadata is extracted, and then a MetadataChangeEvent (MCE) object is generated. The metadata change event is sent to the Metadata Service by the MCE in an asynchronous manner. Then, metadata storage and processing are performed. For example, metadata storage can be asynchronously receiving MCE and storing metadata, building an EntityService object to receive MCE, and storing metadata in the database through the AspectDao object.

[0079] Furthermore, the data lineage relationship is constructed, for example, a LineageService object is constructed to parse metadata, identify dependencies between data entities, and then a LineageRelationship object is created and stored in the database, thereby storing the data lineage relationship. Finally, data query and display are performed; for example, first the user queries the data lineage information through the GraphQL API, and the LineageResolver object processes the query request and returns the lineage data; then the front end displays the data lineage relationship diagram through the LineageView component, and uses the LineageGraph object to build a visual chart.

[0080] The design concepts of LineageService include: scalability, that is, supporting multiple data sources and complex blood relationships; real-time, that is, timely updating of data blood relationship information to ensure the accuracy of data dependencies; consistency, that is, ensuring the consistency of metadata and blood relationship information through transaction management and locking mechanisms; efficient query, that is, optimizing query performance and supporting complex blood relationship queries.

[0081] In a specific example, the data lineage management process can be as follows Figure 3 As shown in the figure, first, heterogeneous data, such as tables, columns, and data of different categories (important data, core data, and general data), are parsed into two tables, order and customers, according to the set dimensions; then, these two tables are aggregated into the result table through business; finally, the result table is split into multiple tables according to the corresponding caliber. The flow relationship of each field in this entire process is sent to the Metadata Service by MCE in an asynchronous manner, and then a data lineage relationship diagram can be constructed based on the flow relationship.

[0082] Secondly, when managing data classification, first define the classification model. For example, extract classification information from various data sources through the DatahubIngestion class and generate ClassificationEvent. At the same time, define the classification data model through the Classification class, including the name, description and sub-classification list of the classification. Then provide interfaces for creating, modifying and querying data classifications to handle the storage and query of classification information. For example, the Classification Service class is responsible for handling the creation, modification and query requests of classifications, and the classification information is persisted to the database by calling ClassificationDao. Finally, the data classification tree is displayed on the front end so that users can browse and manage classification information. For example, the classification tree can be displayed on the front end through the Classification TreeComponent, and users can browse and manage (for example, add, delete, modify, etc.) classification information through a graphical interface. Data classification and data lineage functions are implemented through multiple modules and classes. Each module and class has its own responsibilities, and together ensures the effective management and display of data classification and data lineage.

[0083] In this embodiment, the data synchronization implementation function is pre-packaged, and the user only needs to configure the driving information of the corresponding data source to meet the user's data synchronization needs, and it is at a quasi-real-time level, which lowers the user's usage threshold and can meet most of the application scenarios of current enterprises; and the implementation of data lineage and data classification meets the data problems that urgently need to be solved in current enterprises, such as low data quality, data duplication and redundancy, and chaotic data association relationships, and meets the diversified analysis needs in the big data environment.

[0084] In a specific implementation, first, by connecting to the relational database MySQL, configuring the data source path and access permissions, access to the data stored in MySQL is achieved; then, the managed data is loaded into the MPP-level database using the read-write lock separation method, and the corresponding tables and data models are created to ensure efficient access to the data; at the same time, through functions such as data classification and lineage configuration, the loaded data is layered and graded, and the data is traced to ensure efficient data management and traceability; finally, the MPP-level database is used to perform aggregate queries and real-time analysis on large-scale data to support complex business scenarios and real-time data processing requirements.

[0085] The technical solution of this embodiment simplifies the data integration process and improves the data integration efficiency by directly reading the metadata of the corresponding data source and asynchronously synchronizing the metadata. It ensures the integrity, security and traceability of the data and further improves the efficiency of data management by providing functions such as data classification and data lineage. It can efficiently process complex queries and real-time analysis of large-scale data by using an MPP-level high-performance query database, support SQL queries, and perform cross-table join queries, aggregate queries and real-time analysis, which significantly improves the performance of data queries and analysis. Through functions such as data classification and data lineage, it can normalize the data during the data link process to ensure the consistency and integrity of the data in the entire data link and improve the data quality. By synchronizing the metadata by separating the read-write lock, the process of tool deployment and installation can be reduced, and the packaged package can be directly referenced to achieve plug-and-play, thereby realizing a simple solution for data integration, simplifying the system architecture and reducing the operation and maintenance costs. By providing standardized access and management tools, the need for customized development is reduced and the development and operation and maintenance efficiency is significantly improved.

[0086] In this embodiment, by combining data lineage management, data classification implementation, and metadata asynchronous read and write synchronization, an efficient big data external data source integration and management method is provided, which significantly improves data integration efficiency, data management capabilities and query performance, ensures data quality, simplifies system architecture, improves development and operation and maintenance efficiency, and has significant technical and economic benefits.

[0087] Embodiment 2

[0088] Figure 4 This is a schematic diagram of the structure of a data management device provided in Embodiment 2 of the present invention. Figure 4 As shown, the device includes: a data reading request detection module 210, a matching metadata judgment module 220, a metadata update module 230 and a metadata sending module 240; wherein,

[0089] A data read request detection module 210, configured to acquire a read lock when a data read request from a user is detected;

[0090] The matching metadata judgment module 220 is used to judge whether there is target stored metadata matching the data read request in the database based on the read lock, and if so, acquire the write lock by asynchronously calling the read lock and when it is determined that the metadata stored in the database is different from the metadata in the memory;

[0091] The metadata update module 230 is used to update the metadata of the database according to the metadata in the memory based on the write lock, and release the write lock after the update is completed;

[0092] The metadata sending module 240 is used to send the target stored metadata to the user and release the read lock.

[0093] The technical solution of the embodiment of the present invention is as follows: when a user's data read request is detected, a read lock is acquired; based on the read lock, it is determined whether there is target stored metadata matching the data read request in the database; if so, a write lock is acquired by asynchronously calling the read lock when it is determined that the metadata stored in the database is different from the metadata in the memory; based on the write lock, the database is updated with metadata according to the metadata in the memory, and the write lock is released after the update is completed; the target stored metadata is sent to the user, and the read lock is released; and data management efficiency can be improved by using the read-write lock for read-write control.

[0094] Optionally, the data management device further includes:

[0095] A write lock acquisition module, configured to block the read lock and acquire the write lock if there is no target stored metadata matching the data read request in the database;

[0096] A data writing module, configured to obtain the updated metadata if it is determined that the metadata in the cache is updated based on the write lock, and asynchronously write the updated metadata to the memory and the database;

[0097] A write lock release module, used to release the write lock after the update metadata is written, and obtain the target stored metadata matching the data read request from the database based on the read lock;

[0098] The read lock release module is used to send the target stored metadata to the user and release the read lock.

[0099] Optionally, the data management device further includes:

[0100] The error information sending module is used to generate data non-existence error information if it is determined that the metadata in the cache has not been updated based on the write lock, and send the data non-existence error information to the user.

[0101] Optionally, the data management device further includes:

[0102] A data change module is used to obtain a write lock when a data change request from a user is detected, and based on the write lock, perform a data change operation on the metadata in the memory according to the data change request to obtain the changed metadata;

[0103] A change operation record generation module, used to generate a change operation record corresponding to the data change operation, add the change operation record to the file system operation record file, and release the write lock;

[0104] An operation log entry generating module, used to generate an operation log entry corresponding to the data change operation, and write the operation log entry into the file system operation record file;

[0105] The snapshot file generation module is used to obtain a read lock, generate a snapshot file corresponding to the change metadata based on the read lock, and release the read lock.

[0106] Optionally, the data management device further includes:

[0107] The latest snapshot file acquisition module is used to acquire the latest snapshot file and the corresponding generation time point when a metadata recovery request is detected;

[0108] The metadata recovery module is used to obtain the operation log entries after the generation time point, and implement metadata recovery according to the latest snapshot file and the operation log entries after the generation time point.

[0109] Optionally, the data management device further includes:

[0110] A dependency identification module is used to parse the metadata stored in the database, identify the dependency relationships between data entities, and establish a data lineage relationship diagram based on the dependency relationships between the data entities;

[0111] The data lineage relationship diagram display module is used to display the data lineage relationship diagram on the front-end page according to the request for viewing the data lineage relationship.

[0112] The data management device provided in the embodiment of the present invention can execute the data management method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0113] In the technical solution of the present disclosure, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0114] Embodiment 3

[0115] Figure 5 A schematic diagram of the structure of an electronic device 30 that can be used to implement an embodiment of the present invention is shown. The electronic device 30 is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or claimed herein.

[0116] like Figure 5As shown, the electronic device 30 includes at least one processor 31, and a memory connected to the at least one processor 31, such as a read-only memory (ROM) 32, a random access memory (RAM) 33, etc., wherein the memory stores a computer program that can be executed by at least one processor, and the processor 31 can perform various appropriate actions and processes according to the computer program stored in the read-only memory 32 or the computer program loaded from the storage unit 38 to the random access memory 33. In the RAM 33, various programs and data required for the operation of the electronic device 30 can also be stored. The processor 31, the ROM 32 and the RAM 33 are connected to each other through a bus 34. The input / output (I / O) interface 35 is also connected to the bus 34.

[0117] A number of components in the electronic device 30 are connected to the I / O interface 35, including: an input unit 36, such as a keyboard, a mouse, etc.; an output unit 37, such as various types of displays, speakers, etc.; a storage unit 38, such as a disk, an optical disk, etc.; and a communication unit 39, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 39 allows the electronic device 30 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0118] The processor 31 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the processor 31 include, but are not limited to, a central processing unit, a graphics processing unit, various special artificial intelligence computing chips, various processors running machine learning model algorithms, digital signal processors, and any appropriate processors, controllers, microcontrollers, etc. The processor 31 executes the various methods and processes described above, such as data management methods.

[0119] In some embodiments, the data management method may be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 38. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 30 via the ROM 32 and / or the communication unit 39. When the computer program is loaded into the RAM 33 and executed by the processor 31, one or more steps of the data management method described above may be performed. Alternatively, in other embodiments, the processor 31 may be configured to perform the data management method in any other appropriate manner (e.g., by means of firmware).

[0120] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays, application specific integrated circuits, application specific standard products, systems on a chip, load programmable logic devices, computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0121] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the computer program is executed by the processor, the functions / operations specified in the flow chart and / or block diagram are implemented. The computer program may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0122] In the context of the present invention, a computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, device, or equipment. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or equipment, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory, a read-only memory, an erasable programmable read-only memory, an optical fiber, a portable compact disk read-only memory, an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0123] To provide interaction with a user, the systems and techniques described herein may be implemented on an electronic device 30 having: a display device (e.g., a cathode ray tube or a liquid crystal display) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or a trackball) through which the user can provide input to the electronic device 30. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).

[0124] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: local area networks, wide area networks, blockchain networks, and the Internet.

[0125] This embodiment may also include a computer program product, which includes a computer program. When the computer program is executed by a processor, it implements the data management method provided by any embodiment of the present invention.

[0126] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution of the present invention can be achieved, and this document does not limit this.

[0127] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A data management method, characterized in that: include: When a user's data read request is detected, a read lock is acquired; Based on the read lock, it is determined whether there is target metadata stored in the database that matches the data read request. If so, the read lock is asynchronously called to acquire a write lock when it is determined that the metadata stored in the database is different from the metadata in the memory; Based on the write lock, updating the metadata of the database according to the metadata in the memory, and releasing the write lock after the update is completed; The target stored metadata is sent to the user, and the read lock is released.

2. The method according to claim 1, characterized in that After determining whether there is target stored metadata matching the data read request in the database based on the read lock, the method further includes: If there is no target stored metadata matching the data read request in the database, blocking the read lock and acquiring the write lock; If it is determined based on the write lock that the metadata in the cache is updated, the updated metadata is obtained, and the updated metadata is asynchronously written to the memory and the database; After the update metadata is written, the write lock is released, and the target stored metadata matching the data read request is acquired from the database based on the read lock; The target stored metadata is sent to the user, and the read lock is released.

3. The method according to claim 2, characterized in that After blocking the read lock and acquiring the write lock, it also includes: If it is determined based on the write lock that the metadata in the cache has not been updated, a data non-existence error message is generated and the data non-existence error message is sent to the user.

4. The method according to claim 1, characterized in that: Also includes: When a user's data change request is detected, a write lock is acquired, and based on the write lock, a data change operation is performed on the metadata in the memory according to the data change request to acquire the changed metadata; Generate a change operation record corresponding to the data change operation, add the change operation record to a file system operation record file, and release the write lock; Generate an operation log entry corresponding to the data change operation, and write the operation log entry into the file system operation record file; A read lock is acquired, and a snapshot file corresponding to the changed metadata is generated based on the read lock, and the read lock is released.

5. The method according to claim 4, characterized in that After releasing the read lock, the method further includes: When a metadata recovery request is detected, the latest snapshot file and the corresponding generation time point are obtained; The operation log entries after the generation time point are obtained, and metadata recovery is implemented according to the latest snapshot file and the operation log entries after the generation time point.

6. The method according to claim 1, characterized in that Also includes: Parsing the metadata stored in the database to identify the dependency relationships between data entities, and establishing a data lineage relationship diagram based on the dependency relationships between the data entities; According to the request for viewing the data lineage relationship, the data lineage relationship diagram is displayed on the front-end page.

7. A data management device, characterized in that: include: A data read request detection module is used to acquire a read lock when a user's data read request is detected; A matching metadata judgment module is used to judge whether there is target stored metadata matching the data read request in the database based on the read lock, and if so, acquire a write lock by asynchronously calling the read lock and when it is determined that the metadata stored in the database is different from the metadata in the memory; A metadata update module, configured to update the metadata of the database according to the metadata in the memory based on the write lock, and release the write lock after the update is completed; The metadata sending module is used to send the target stored metadata to the user and release the read lock.

8. An electronic device, characterized in that: The electronic device comprises: at least one processor, and a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the data management method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and the computer program is used to enable a processor to implement the data management method according to any one of claims 1 to 6 when executed.

10. A computer program product, characterized in that The invention comprises a computer program, which implements the data management method according to any one of claims 1 to 6 when being executed by a processor.

Images