Data synchronization method and device, equipment, storage medium and program product
By using metadata analysis and caching mechanisms in the data synchronization scheme, the problem of adapting multiple data formats in the prior art is solved, efficient, accurate and real-time data synchronization is achieved, and the cost of access party transformation is reduced.
Patent Information
- Application Number
- CN202410961394.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-17
- Publication Date
- 2024-11-12
AI Technical Summary
The existing data synchronization solution is low in efficiency, accuracy and real-time when adapting to multiple data formats, resulting in large workloads for access parties and inconsistent intervals in data synchronization, which makes it impossible to adapt well to various data formats.
By receiving resource synchronization events sent by the product side, the resource raw data is parsed with metadata, stored in the database, and asynchronously processed using the cache mechanism and message queue to ensure the accuracy and real-timeness of data synchronization.
It improves the adaptability of multiple data formats, improves the efficiency, accuracy and real-time of data synchronization, and reduces data read latency and development adaptation costs.
Smart Images

Figure CN118939726A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of computer technology, and in particular to a data synchronization method, device, equipment, storage medium and program product. Background Art
[0002] In the field of cloud computing, a general cloud platform has many cloud products. Users can perform various operations by logging into the console. The resource management service provides unified viewing and management capabilities for resources. Each cloud product needs to be connected to the resource management service, and relevant data must be consistent in different consoles.
[0003] The existing data synchronization solution uses a scheduled resource synchronization method to ensure the synchronization between the resource status of the management platform and the actual status of cloud resources, reduce the calling frequency of the cloud platform API interface, save bandwidth usage, and introduce NoSql database to save large amounts of resource data, thereby improving data storage and reading performance.
[0004] The scheduled synchronization method in the existing data synchronization solution cannot adapt well to various data formats. For example, the access party has a large amount of transformation workload, frequent re-adaptation to changes, inconsistent intervals for data synchronization, and cannot adapt well to various data formats. This leads to technical problems such as low efficiency, accuracy, and real-time performance of data synchronization. Summary of the invention
[0005] The embodiments of the present application provide a data synchronization method, apparatus, device, storage medium and program product to achieve the technical effect of improving the adaptability of multiple data formats and improving the efficiency, accuracy and real-time performance of data synchronization.
[0006] In a first aspect, an embodiment of the present application provides a data synchronization method, which is applied to a resource management side. The method includes:
[0007] Receive resource synchronization events sent by the product side. Resource synchronization events include metadata and resource original data. Use metadata to parse resource original data to obtain resource-related data.
[0008] The resource association data, metadata and resource original data are stored in a database to obtain a storage result, the storage result is stored in a third database to obtain cache content, and the storage result and cache content are sent to the product side;
[0009] Receive the change results of the cache content on the product side, determine the current cache value of the change result based on the change result, and write the current cache value into the database for data synchronization.
[0010] In a possible implementation, resource association data, metadata, and resource original data are stored in a database to obtain a storage result, including:
[0011] Using the first database to store the resource-related data and metadata, obtaining a first storage result;
[0012] Using a second database to store the original resource data to obtain a second storage result, wherein the database includes the first database and the second database;
[0013] A storage result is obtained according to the first storage result and the second storage result.
[0014] In a possible implementation, determining a current cache value of the change result according to the change result, and writing the current cache value into a database for data synchronization includes:
[0015] Asynchronously process the change results to obtain the cache key;
[0016] According to the cache key, the current cache value corresponding to the change result is obtained, and the current cache value is written to the database for data synchronization.
[0017] In a possible implementation, asynchronously processing the change result to obtain a cache key includes:
[0018] Use the message queue to process the change results and obtain a data synchronization message containing the cache key;
[0019] Parse the data synchronization message to obtain the cache key.
[0020] In a second aspect, an embodiment of the present application provides a data synchronization method, which is applied to a product side. The method includes:
[0021] Determine the requirements for the resource center, label, and resource group of the resource to be synchronized, and configure the resource attribute path corresponding to the resource center, label, and resource group of the resource to be synchronized according to the requirements;
[0022] Determine the metadata and resource original data of the resource to be synchronized, where the metadata includes: resource attribute path, generate resource synchronization events of the resource to be synchronized according to the metadata and resource original data, and send the resource synchronization events to the resource management side, where the metadata is the description data of the resource to be synchronized, and the resource original data is used to indicate the attributes of the resource to be synchronized;
[0023] Receive the storage results and cache contents sent by the resource management side, query the storage results and cache contents, obtain the cache results, use the distributed lock to change the cache results, obtain the change results, and send the change results to the resource management side. The storage results and cache contents are obtained by the resource management side based on the resource synchronization event, and the cache content is the priority query cache.
[0024] In a possible implementation, querying the storage result and cache content to obtain the cache result includes:
[0025] Query the cache content and determine whether the cache content exists. If so, obtain the cache result. If not, traverse the stored results to obtain the query result, and use the query result as the cache content corresponding to the stored result.
[0026] In a possible implementation, configuring the resource attribute path corresponding to the resource center, label, and resource group of the resource to be synchronized as required includes:
[0027] When accessing the resource center, configure the first resource attribute path of the resource to be synchronized as required;
[0028] When accessing a tag, configure the second resource attribute path of the resource to be synchronized;
[0029] When accessing a resource group, configure the third resource attribute path of the resource to be synchronized;
[0030] The first resource attribute path, the second resource attribute path and the third resource attribute path are integrated to obtain a resource attribute path.
[0031] In a third aspect, an embodiment of the present application provides a data synchronization device, which is applied to a resource management side, and the device includes:
[0032] The receiving module is used to receive resource synchronization events sent by the product side. The resource synchronization events include metadata and resource original data. The metadata is used to parse the resource original data to obtain resource-related data.
[0033] A storage module is used to store resource-related data, metadata and resource original data in a database, obtain a storage result, store the storage result in a third database, obtain cache content, and send the storage result and cache content to the product side;
[0034] The first processing module is used to receive the change result of the cache content on the product side, determine the current cache value of the change result according to the change result, and write the current cache value into the database for data synchronization.
[0035] In a possible implementation manner, the storage module is further used for:
[0036] Using the first database to store the resource-related data and metadata, obtaining a first storage result;
[0037] Using a second database to store the original resource data to obtain a second storage result, wherein the database includes the first database and the second database;
[0038] A storage result is obtained according to the first storage result and the second storage result.
[0039] In a possible implementation manner, the first processing module is further configured to:
[0040] Asynchronously process the change results to obtain the cache key;
[0041] According to the cache key, the current cache value corresponding to the change result is obtained, and the current cache value is written to the database for data synchronization.
[0042] In a possible implementation manner, the first processing module is further configured to:
[0043] Use the message queue to process the change results and obtain a data synchronization message containing the cache key;
[0044] Parse the data synchronization message to obtain the cache key.
[0045] In a fourth aspect, an embodiment of the present application provides a data synchronization device, which is applied to a product side, and the device includes:
[0046] The configuration module is used to determine the requirements for the resource center, label and resource group of the resource to be synchronized to access the resource management side, and configure the resource attribute path corresponding to the resource center, label and resource group of the resource to be synchronized according to the requirements;
[0047] The second processing module is used to determine the metadata and resource original data of the resource to be synchronized, where the metadata includes: resource attribute path, generate resource synchronization events of the resource to be synchronized according to the metadata and resource original data, and send the resource synchronization events to the resource management side, where the metadata is the description data of the resource to be synchronized, and the resource original data is used to indicate the attributes of the resource to be synchronized;
[0048] The third processing module is used to receive the storage results and cache contents sent by the resource management side, query the storage results and cache contents to obtain the cache results, and use the distributed lock to change the cache results to obtain the change results, and send the change results to the resource management side. The storage results and cache contents are obtained by the resource management side based on the resource synchronization event, and the cache content is a priority query cache.
[0049] In a possible implementation, the configuration module is further configured to:
[0050] When accessing the resource center, configure the first resource attribute path of the resource to be synchronized as required;
[0051] When accessing a tag, configure the second resource attribute path of the resource to be synchronized;
[0052] When accessing a resource group, configure the third resource attribute path of the resource to be synchronized;
[0053] The first resource attribute path, the second resource attribute path and the third resource attribute path are integrated to obtain a resource attribute path.
[0054] In a possible implementation manner, the third processing module is further configured to:
[0055] Query the cache content and determine whether the cache content exists. If so, obtain the cache result. If not, traverse the stored results to obtain the query result, and use the query result as the cache content corresponding to the stored result.
[0056] In a fifth aspect, an embodiment of the present application provides a data synchronization device, including: a memory, a processor;
[0057] Memory stores computer-executable instructions;
[0058] The processor executes the computer-executable instructions stored in the memory, so that the processor executes the data synchronization method applied to any one of the first aspect or the second aspect above.
[0059] In a sixth aspect, an embodiment of the present application provides a computer-readable storage medium, in which computer execution instructions are stored. When the computer execution instructions are executed by a processor, they are used to implement the data synchronization method applied to any one of the first aspect or the second aspect above.
[0060] In a seventh aspect, an embodiment of the present application provides a computer program product, including a computer program, which, when executed by a processor, implements the data synchronization method applied in any one of the first aspect or the second aspect above.
[0061] The embodiments of the present application provide a data synchronization method, apparatus, device, storage medium and program product, which receives a resource synchronization event sent by a product side, wherein the resource synchronization event includes: metadata and resource original data, and uses the metadata to parse the resource original data to obtain resource-related data;
[0062] The resource-related data, metadata and resource original data are stored in a database to obtain storage results, the storage results are stored in a third database to obtain cached content, and the storage results and cached content are sent to the product side; the change results of the cached content on the product side are received, and the current cache value of the change results is determined according to the change results, and the current cache value is written into the database for data synchronization, ultimately achieving the technical effect of improving the adaptability of multiple data formats and improving the efficiency, accuracy and real-time performance of data synchronization. BRIEF DESCRIPTION OF THE DRAWINGS
[0063] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.
[0064] Figure 1 A schematic diagram of a data synchronization method provided in an embodiment of the present application Figure 1 ;
[0065] Figure 2 A schematic diagram of a data synchronization method provided in an embodiment of the present application Figure 2 ;
[0066] Figure 3 A schematic diagram of a data synchronization method provided in an embodiment of the present application Figure 3 ;
[0067] Figure 4 A schematic diagram of a data synchronization method provided in an embodiment of the present application Figure 4 ;
[0068] Figure 5 A schematic diagram of a data synchronization device provided in an embodiment of the present application Figure 1 ;
[0069] Figure 6 A schematic diagram of a data synchronization device provided in an embodiment of the present application Figure 2 ;
[0070] Figure 7 A schematic diagram of the structure of a data synchronization device provided in an embodiment of the present application.
[0071] The above drawings have shown clear embodiments of the present application, which will be described in more detail later. These drawings and text descriptions are not intended to limit the scope of the present application in any way, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION
[0072] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.
[0073] The existing data synchronization solution adopts a resource timed synchronization mode to ensure that the resource status of the management platform can be synchronized with the actual status of cloud resources, saving bandwidth usage, and introducing a NoSql database to store a large amount of resource data to improve data storage and reading performance.
[0074] However, the scheduled synchronization method needs to be re-adapted for data with large workload and frequent changes, and there are technical problems such as inconsistent intervals and inability to adapt well to various data formats.
[0075] A data synchronization method provided by an embodiment of the present application receives a resource synchronization event sent by a product side, and uses metadata definition and parsing mechanisms to parse resource original data of different structures and formats, thereby adapting to multiple sources and formats, and improving the efficiency and accuracy of data synchronization, thereby obtaining resource-related data; storing resource-related data, metadata, and resource original data in a database, obtaining a storage result, and using the storage result to obtain cache content, sending the storage result and cache content to the product side, receiving the change result of the cache content by the product side, determining the current cache value of the change result according to the change result, and writing the current cache value into the database to ensure data consistency, and at the same time using a cache mechanism to speed up data acquisition and change speed. Therefore, the method solves the technical problems of poor adaptability to multiple data formats, low efficiency, accuracy, and real-time performance of data synchronization.
[0076] The technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems are described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.
[0077] Figure 1 A schematic diagram of a data synchronization method provided in an embodiment of the present application Figure 1 ,like Figure 1 As shown, a data synchronization method provided in this embodiment is applied to a resource management side, and the method includes:
[0078] S101. Receive a resource synchronization event sent by a product side, where the resource synchronization event includes metadata and resource original data, and use the metadata to parse the resource original data to obtain resource-related data.
[0079] In this embodiment, for different resource original data, JsonPath-based metadata is used to parse the resource original data to obtain resource-related data, so as to achieve persistence of the resource-related data.
[0080] The resource-related data includes resources, tags, resource groups, resource-tag relationships, and resource-resource group relationships. In addition, JsonPath is an information extraction library that is a tool for extracting specified information from JSON documents based on expressions.
[0081] S102: Store resource-related data, metadata, and resource original data in a database to obtain a storage result, store the storage result in a third database to obtain cache content, and send the storage result and cache content to the product side.
[0082] In this embodiment, in order to support high concurrency in data synchronization processing, a cache mechanism is introduced to implement a cache-database two-layer data persistence solution.
[0083] Specifically, different types of data (resource-related data, metadata, and resource original data) are uniformly stored in the database to facilitate centralized management and maintenance, ensuring data integrity and consistency. Storing the storage results in a third database as cache content can speed up data acquisition on the product side, reduce data reading delays, and improve user experience. At the same time, the database provides a good data backup mechanism to ensure data security and recoverability.
[0084] S103: Receive the change result of the cache content from the product side, determine the current cache value of the change result according to the change result, and write the current cache value into the database for data synchronization.
[0085] In this embodiment, the change results of the cache content on the product side are received to ensure that the changes on the product side can be reflected in the database in a timely manner, and the real-time synchronization of data is maintained. Changes can be flexibly made according to the needs of the product side, and these changes can be solidified in the database in a timely manner for data synchronization, adapting to different business scenarios and changes.
[0086] A data synchronization method provided by an embodiment of the present application receives a resource synchronization event sent by a product side, and uses metadata to parse resource original data to obtain resource-related data; stores the resource-related data, metadata, and resource original data in a database to obtain a storage result, stores the storage result in a third database to obtain cache content, and sends the storage result and the cache content to the product side; receives a change result of the cache content on the product side, determines a current cache value of the change result according to the change result, and writes the current cache value into the database for data synchronization, thereby achieving a technical effect of improving the adaptability of multiple data formats and improving the efficiency, accuracy, and real-time performance of data synchronization.
[0087] Figure 2 A schematic diagram of a data synchronization method provided in an embodiment of the present application Figure 2 ,like Figure 2 As shown, in this embodiment Figure 1 Based on the embodiment, a data synchronization method applied to the resource management side is described in detail, and the method includes:
[0088] S201. Receive a resource synchronization event sent by a product side, where the resource synchronization event includes metadata and resource original data, and use the metadata to parse the resource original data to obtain resource-related data.
[0089] Step S201 is similar to the above step S101 and will not be described again.
[0090] S202. Use the first database to store resource-related data and metadata to obtain a first storage result, use the second database to store resource original data to obtain a second storage result, and obtain a storage result based on the first storage result and the second storage result.
[0091] In this embodiment, according to the characteristics and requirements of resource-related data, metadata, and resource original data, a suitable database configuration and storage strategy are selected to help improve the performance of data storage and reading.
[0092] The first database may be, for example, a relational database, and the second database may be, for example, a document database. The database includes the first database and the second database.
[0093] S203: Store the storage result in a third database, obtain cache content, and send the storage result and cache content to the product side.
[0094] In this embodiment, the third database refers to a cache database. Since the access speed to the cache database is faster than that to the database, a cache mechanism is introduced so that the cache content in the third database is a priority query cache. This avoids frequent query operations on the database and reduces the pressure on the database.
[0095] S204: Receive the change result of the cache content from the product side, process the change result using the message queue, obtain a data synchronization message including a cache key, parse the data synchronization message, and obtain the cache key.
[0096] In this embodiment, the changes to the cache results are received from the product side, and the changes to the cache contents are implemented. The message queue is used to process the change results in a production-consumption mode. First, a specific data synchronization message is formed at one end of the message queue. This message contains key information related to the cache, namely the cache key, which is transmitted through the message queue. At the other end of the message queue, the data synchronization information is received and parsed to obtain the cache key.
[0097] Among them, the message queue is a middleware technology used to transmit messages between different components or services, which can improve the system's concurrent processing capabilities and overall performance.
[0098] S205. Obtain the current cache value corresponding to the change result according to the cache key, and write the current cache value into the database for data synchronization.
[0099] In this embodiment, a specific cache key is used to find the current cache value associated with it, which has been changed, that is, the latest data state. Then the current cache value is written into the database, so that the data in the database is consistent with the data in the cache content, thereby achieving data synchronization and ensuring that the data in different storage locations have the same information content.
[0100] It should be noted that the database refers to the first database and the second database, and the cache content is stored in the third database.
[0101] A data synchronization method provided by an embodiment of the present application receives a resource synchronization event sent by a product side, uses metadata to parse the resource original data, realizes adaptation to multiple sources and formats, and improves the efficiency and accuracy of data synchronization, thereby obtaining resource-related data; uses different databases to store resource-related data, better manages and adapts to multiple data formats, stores the storage results in a third database to form a cache content of a priority query cache, improves the efficiency of data acquisition, and enhances the processing capability of different data formats; receives the change result of the cache content by the product side, uses a message queue to asynchronously process the change result, obtains a data synchronization message containing a cache key, parses the data synchronization message, obtains a cache key, and then searches in the cache content according to the cache key to obtain the current cache value, and writes the current cache value into the database for data synchronization, thereby improving the real-time and accuracy of data synchronization. Therefore, the method achieves the technical effect of improving the adaptability to multiple data formats and the efficiency, accuracy and real-time of data synchronization.
[0102] Figure 3 A schematic diagram of a data synchronization method provided in an embodiment of the present application Figure 3 ,like Figure 3 As shown, a data synchronization method provided in this embodiment is applied to a product side, and the method includes:
[0103] S301. Determine the requirements for the resource center, label, and resource group of the resource management side that the resources to be synchronized are accessed, and configure the resource attribute paths corresponding to the resource center, label, and resource group of the resources to be synchronized according to the requirements.
[0104] In this embodiment, according to different access targets and requirements, appropriate attribute paths for accurate access and identification are set for the resources to be synchronized to ensure that the resources can be accurately connected and associated with various parts of the resource management side in a prescribed and expected manner. Therefore, when data is synchronized between the product side and the resource management side, it is necessary to allow the resources to be synchronized to access the resource center, label and resource group of the resource management side, and configure the resource attribute paths corresponding to the resource center, label and resource group of the resources to be synchronized.
[0105] It should be noted that accessing the resource center is a prerequisite for accessing tags and resource groups.
[0106] S302: Determine the metadata and resource original data of the resource to be synchronized, where the metadata includes: resource attribute path, generate a resource synchronization event of the resource to be synchronized according to the metadata and resource original data, and send the resource synchronization event to the resource management side.
[0107] In this embodiment, before the resources to be synchronized on the product side are connected to the resource management side, it is necessary to first create metadata unique to the resource type in the unified configuration center, and fill in information such as product type, product identification, resource type, resource identification, and person in charge information. Among them, the resource type and resource identification are globally unique. At the same time, when accessing the resource center, label, and resource group, the configuration of the resource attribute path is configured in the metadata. The metadata is the description data of the resources to be synchronized, and it is also the basis for process control and data analysis.
[0108] Resource original data refers to the full resource information of the resources to be synchronized on the product side, which is used to indicate the attributes of the resources to be synchronized. The resource original data includes but is not limited to resource ID, resource master account ID, resource name, resource creation time, resource cloud region ID, resource meta-zone ID, resource group ID, and resource tag.
[0109] A resource synchronization event for the resource to be synchronized is generated based on the metadata and resource original data. In addition to the metadata, resource original data, and full information of the resource to be synchronized, the resource synchronization event also includes event ID, event time, event type, operator ID, resource original data, and resource identification attributes. Among them, event types are divided into three types: addition, change, and deletion.
[0110] It should be noted that the introduction of resource original data rather than mandatory resource information format is based on system architecture considerations. For cloud platform resource management services, resource original data is located upstream of IaaS, PaaS and SaaS products, and downstream of monitoring, access control, billing and other services. If the original data of resources is sent when each product is connected to the resource management side, and the resource management side is persisted, even if the upstream service changes later, the resource management side still provides data support, and only the upstream service needs to make corresponding adaptations. Second, it is based on system interaction considerations. In order to achieve data consistency between the product side and the resource management side, in some cases, the resource management side also needs to synchronize data with the product side. The data persisted locally on the product side varies in form. If the resource management side persists and uses the original data of resources sent by each product as a template to send, rather than sending it in a fixed format, it will save the development and adaptation costs of each product console.
[0111] S303, receiving the storage result and cache content sent by the resource management side, querying the storage result and cache content to obtain the cache result, and using the distributed lock to change the cache result to obtain the change result, and sending the change result to the resource management side.
[0112] In this embodiment, the storage results and cache contents sent by the resource management side are received, wherein the storage results and cache contents are obtained by the resource management side according to the resource synchronization event. The storage results and cache contents are queried to obtain the cache results. It should be noted that the cache contents are priority query caches. The cache results are changed, wherein the change processing includes the insertion of new data and the modification and deletion of cache results. And in order to solve the concurrency problem, a distributed lock is introduced to control the changes of the cache contents.
[0113] The steps of distributed lock are as follows: When multiple requests are changing specific data at the same time, a distributed lock is acquired. The request that obtains the lock can safely perform changes to the data. During this period, other requests are blocked because they cannot obtain the lock. When the request completes the operation and releases the lock, other waiting requests have the opportunity to obtain the lock and perform corresponding operations. This avoids data inconsistency or other abnormal situations caused by conflicting operations of multiple requests at the same time, ensuring the orderliness and correctness of data operations in a concurrent environment.
[0114] A data synchronization method provided in an embodiment of the present application configures the corresponding resource attribute path according to the demand by clarifying the requirements of the resource to be synchronized for the resource center, label and resource group in the resource management; determining the original data of the resource, and determining the metadata according to the resource attribute path, and then determining the resource synchronization event according to the metadata and the original data of the resource, and sending the resource synchronization event to the resource management side, which can trigger the synchronization action of the resource in time and ensure the real-time and accuracy of the synchronization. By receiving the storage results and cached content on the resource management side, querying to obtain the cached results, and then using distributed locks to make changes to the cached results, the consistency and security of the data are guaranteed, the problem of data confusion caused by concurrent operations is avoided, and the change results are sent to the resource management side. Therefore, this method achieves the technical effect of improving the adaptability to multiple data formats and the efficiency, accuracy and real-time performance of data synchronization.
[0115] Figure 4 A schematic diagram of a data synchronization method provided in an embodiment of the present application Figure 4 ,like Figure 4 As shown, in this embodiment Figure 3 Based on the embodiment, a data synchronization method applied to the product side is described in detail, and the method includes:
[0116] S401: Determine the requirements for the resource to be synchronized to access the resource center, label and resource group on the resource management side.
[0117] Step S401 is similar to the above step S301 and will not be described again.
[0118] S402. According to the requirements, when accessing the resource center, configure the first resource attribute path of the resource to be synchronized; when accessing the tag, configure the second resource attribute path of the resource to be synchronized; when accessing the resource group, configure the third resource attribute path of the resource to be synchronized.
[0119] In this embodiment, the access resource center needs to configure the first resource attribute path of the resource to be synchronized, which includes "path pointing to resource ID", "path pointing to resource name", "path pointing to resource primary account ID", and "path pointing to resource creation time". Because some products have the concept of cloud regions or cloud zones, in order to distinguish products of cloud regions or cloud zones, it is necessary to fill in "path pointing to cloud regions" and "path pointing to cloud zones" respectively.
[0120] When accessing a tag, a second resource attribute path of the resource to be synchronized is configured to achieve persistence of the tag, and the second resource attribute path includes a "path pointing to the resource tag".
[0121] When accessing a resource group, configure the third resource attribute path of the resource to be synchronized to achieve persistence of the resource group. The third resource attribute path includes the "path pointing to the resource group ID"
[0122] S403: Integrate the first resource attribute path, the second resource attribute path, and the third resource attribute path to obtain a resource attribute path.
[0123] In this embodiment, three resource attribute paths corresponding to different situations (e.g., access resource center, access tag, access resource group), namely, the first resource attribute path, the second resource attribute path, and the third resource attribute path are summarized. Through this integration operation, a comprehensive resource attribute path containing a variety of relevant information is formed. This integrated resource attribute path can more comprehensively describe the attribute characteristics and associations of the resources to be synchronized in different aspects, so that the resources can be accurately identified and processed in the subsequent resource synchronization process.
[0124] S404. Determine the metadata and resource original data of the resources to be synchronized. The metadata includes: resource attribute path. Generate a resource synchronization event for the resources to be synchronized based on the metadata and resource original data. Send the resource synchronization event to the resource management side, and receive the storage result and cache content sent by the resource management side.
[0125] Step S404 is similar to the above step S302 and will not be described again.
[0126] S405: Query the cache content to determine whether the cache content exists.
[0127] In this embodiment, the access speed of the cache is usually much faster than that of the database, which greatly improves the query response speed, reduces the waiting time, and improves the user experience. At the same time, it avoids frequent query operations on the database, reduces the pressure on the database, and can better cope with a large number of query requests in high-concurrency scenarios, ensuring that the system can still respond quickly under high load. Therefore, a cache mechanism is introduced to make the cache content a priority query cache. By querying the cache content, it is determined whether the cache content exists, which provides a basis for subsequent changes.
[0128] S406: If yes, obtain the cache result.
[0129] S407: If not, traverse the stored results to obtain the query results, use the query results as the cache content corresponding to the stored results, and re-execute step S405.
[0130] In this embodiment, if the data to be queried is not found in the cache content, the stored results are traversed and queried to obtain the query results, which are then saved as cache content.
[0131] S408: Use the distributed lock to modify the cache result, obtain the modified result, and send the modified result to the resource management side.
[0132] In this embodiment, it can be seen from the above embodiments that since the cache content is obtained based on the storage result, the cache content also includes resources, tags, resource groups, resource-tag relationships, resource-resource group relationships, metadata and resource original data. Furthermore, the cache result will also be the above data.
[0133] In a possible implementation, when the cache result is modified, if the cache result is a resource and resource original data, the resource and resource original data are added, updated, or deleted.
[0134] When the cache result is a resource-tag relationship, edit the tag. Editing tags will fully update the tags in the cache result, that is, unbinding tags that existed before but do not exist now, and binding tags that did not exist before but do exist now. It also supports the cache result being empty or having new tags that have not been created. The former means unbinding all tags, and the latter means creating first and then binding.
[0135] When the cached result is a resource-resource group relationship, the resource group is moved. The cached result includes the new resource group ID and the old resource group ID. If the cached result contains the old resource group ID, it means that the relationship is terminated. If not, no operation is performed. If the cached result contains the new resource group ID, it means that a new relationship is created. If not, a relationship between the resource and the default resource group is established.
[0136] A data synchronization method provided in an embodiment of the present application realizes refined management of resource access by configuring the resource attribute path of the resource to be synchronized to access the resource center, label and resource group, configures specific paths according to different access requirements, and improves the pertinence and accuracy of resource management. By determining the original data of the resource to be synchronized, and determining the metadata according to the resource attribute path, and generating the resource synchronization event of the resource to be synchronized according to the metadata and the original data of the resource, and sending the resource synchronization event to the resource management side, the accuracy and integrity of resource data synchronization are guaranteed. Priority is given to querying the cache content, and different strategies are adopted according to whether the cache content exists, which improves the efficiency of data synchronization, and then the distributed lock is used to ensure the security and stability of data changes and prevent data conflicts. Therefore, this method achieves the technical effect of improving the adaptability to multiple data formats and the efficiency, accuracy and real-time performance of data synchronization.
[0137] Figure 5 A schematic diagram of a data synchronization device provided in an embodiment of the present application Figure 1 ,like Figure 5 As shown, the data synchronization device provided in this embodiment is applied to the resource management side. The data synchronization device 50 includes:
[0138] The receiving module 501 is used to receive the resource synchronization event sent by the product side, the resource synchronization event includes: metadata, resource original data, and use the metadata to parse the resource original data to obtain resource related data;
[0139] The storage module 502 is used to store the resource association data, metadata and resource original data in the database, obtain the storage result, store the storage result in the third database, obtain the cache content, and send the storage result and the cache content to the product side;
[0140] The first processing module 503 is used to receive the change result of the cache content on the product side, determine the current cache value of the change result according to the change result, and write the current cache value into the database for data synchronization.
[0141] In a possible implementation, the storage module 502 is further configured to:
[0142] Using the first database to store the resource-related data and metadata, obtaining a first storage result;
[0143] Using a second database to store the original resource data to obtain a second storage result, wherein the database includes the first database and the second database;
[0144] A storage result is obtained according to the first storage result and the second storage result.
[0145] In a possible implementation manner, the first processing module 503 is further configured to:
[0146] Asynchronously process the change results to obtain the cache key;
[0147] According to the cache key, the current cache value corresponding to the change result is obtained, and the current cache value is written to the database for data synchronization.
[0148] In a possible implementation manner, the first processing module 503 is further configured to:
[0149] Use the message queue to process the change results and obtain a data synchronization message containing the cache key;
[0150] Parse the data synchronization message to obtain the cache key.
[0151] The data synchronization device provided in this embodiment can execute the method provided in the above method embodiment. Its implementation principle and technical effect are similar, and this embodiment will not be described in detail here.
[0152] Figure 6 A schematic diagram of a data synchronization device provided in an embodiment of the present application Figure 2 ,like Figure 6As shown, the data synchronization device provided in this embodiment is applied to the product side. The data synchronization device 60 includes:
[0153] Configuration module 601, used to determine the requirements of the resource to be synchronized accessing the resource center, label and resource group of the resource management side, and configure the resource attribute path corresponding to the resource to be synchronized accessing the resource center, label and resource group according to the requirements;
[0154] The second processing module 602 is used to determine the metadata and resource original data of the resource to be synchronized, where the metadata includes: resource attribute path, generate a resource synchronization event of the resource to be synchronized according to the metadata and resource original data, and send the resource synchronization event to the resource management side, where the metadata is the description data of the resource to be synchronized, and the resource original data is used to indicate the attribute of the resource to be synchronized;
[0155] The third processing module 603 is used to receive the storage results and cache contents sent by the resource management side, query the storage results and cache contents to obtain the cache results, and use the distributed lock to change the cache results to obtain the change results, and send the change results to the resource management side. The storage results and cache contents are obtained by the resource management side according to the resource synchronization event, and the cache content is a priority query cache.
[0156] In a possible implementation, the configuration module 601 is further configured to:
[0157] When accessing the resource center, configure the first resource attribute path of the resource to be synchronized as required;
[0158] When accessing a tag, configure the second resource attribute path of the resource to be synchronized;
[0159] When accessing a resource group, configure the third resource attribute path of the resource to be synchronized;
[0160] The first resource attribute path, the second resource attribute path and the third resource attribute path are integrated to obtain a resource attribute path.
[0161] In a possible implementation manner, the third processing module 603 is further configured to:
[0162] Query the cache content and determine whether the cache content exists. If so, obtain the cache result. If not, traverse the stored results to obtain the query result, and use the query result as the cache content corresponding to the stored result.
[0163] The data synchronization device provided in this embodiment can execute the method provided in the above method embodiment. Its implementation principle and technical effect are similar, and this embodiment will not be described in detail here.
[0164] Figure 7A schematic diagram of the structure of a data synchronization device provided in an embodiment of the present application. Figure 7 As shown, the data synchronization device 70 provided in this embodiment includes: at least one processor 701 and a memory 702. Optionally, the device 70 also includes a communication component 703. The processor 701, the memory 702 and the communication component 703 are connected via a bus 704.
[0165] In a specific implementation process, at least one processor 701 executes the computer-executable instructions stored in the memory 702, so that at least one processor 701 executes the above method.
[0166] The specific implementation process of the processor 701 can be found in the above method embodiment, and its implementation principle and technical effect are similar, so this embodiment will not be repeated here.
[0167] In the above embodiments, it should be understood that the processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), etc. A general-purpose processor may be a microprocessor or any conventional processor. The steps of the method disclosed in the invention may be directly implemented as being executed by a hardware processor, or may be executed by a combination of hardware and software modules in the processor.
[0168] The memory may include a high-speed memory (Random Access Memory, RAM), and may also include a non-volatile memory (Non-volatile Memory, NVM), such as at least one disk memory.
[0169] The bus may be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, the bus in the drawings of the embodiments of the present application is not limited to only one bus or one type of bus.
[0170] An embodiment of the present application also provides a computer program product, including a computer program, which implements the above method when executed by a processor.
[0171] An embodiment of the present application also provides a computer-readable storage medium, in which computer-executable instructions are stored. When a processor executes the computer-executable instructions, the above method is implemented.
[0172] The above-mentioned readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk. The readable storage medium can be any available medium that can be accessed by a general or special-purpose computer.
[0173] An exemplary readable storage medium is coupled to a processor so that the processor can read information from the readable storage medium and write information to the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can be located in an application specific integrated circuit (Application Specific Integrated Circuits, referred to as: ASIC). Of course, the processor and the readable storage medium can also exist in the device as discrete components.
[0174] The division of units is only a logical function division, and there may be other divisions in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interface, device or unit, which can be electrical, mechanical or other forms.
[0175] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.
[0176] If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the methods of each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.
[0177] Those skilled in the art can understand that all or part of the steps of implementing the above-mentioned method embodiments can be completed by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, the steps of the above-mentioned method embodiments are executed; and the aforementioned storage medium includes: ROM, RAM, disk or optical disk and other media that can store program codes.
[0178] The terms "first", "second", "third", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein, for example, can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, products or devices.
[0179] It should be noted that in the embodiments of the present application, words such as "exemplary" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary" or "for example" in the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a specific way.
[0180] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in the embodiments of the present application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data need to comply with relevant laws, regulations and standards of the relevant regions, and provide corresponding operation entrances for users to choose to authorize or refuse.
[0181] Finally, it should be noted that those skilled in the art will readily conceive of other embodiments of the present invention after considering the specification and practicing the invention disclosed herein. The present invention is intended to cover any variations, uses or adaptations of the present invention, which follow the general principles of the present invention and include common knowledge or customary technical means in the art not disclosed by the present invention, are not limited to the precise structure described above and shown in the drawings, and may be modified and changed in various ways without departing from the scope thereof. The scope of the present invention is limited only by the appended claims.
Claims
1. A data synchronization method, It is characterized in that Applied to resource management, including: Receiving a resource synchronization event sent by a product side, the resource synchronization event including metadata and resource original data, and parsing the resource original data using the metadata to obtain resource-related data; The resource association data, the metadata and the resource original data are stored in a database to obtain a storage result, the storage result is stored in a third database to obtain cache content, and the storage result and the cache content are sent to the product side; Receive the change result of the cache content from the product side, determine the current cache value of the change result according to the change result, and write the current cache value into the database for data synchronization.
2. The method according to claim 1, It is characterized in that The storing of the resource association data, the metadata and the resource original data in a database to obtain a storage result includes: Using a first database to store the resource-related data and the metadata, obtaining a first storage result; Using a second database to store the resource original data to obtain a second storage result, wherein the database includes a first database and a second database; The storage result is obtained according to the first storage result and the second storage result.
3. The method according to claim 1, It is characterized in that The step of determining a current cache value of the change result according to the change result, and writing the current cache value into the database for data synchronization includes: Asynchronously processing the change result to obtain a cache key; According to the cache key, the current cache value corresponding to the change result is obtained, and the current cache value is written into the database for data synchronization.
4. The method according to claim 3, It is characterized in that The asynchronously processing the change result to obtain a cache key includes: Use the message queue to process the change results and obtain a data synchronization message containing the cache key; The data synchronization message is parsed to obtain the cache key.
5. A data synchronization method, It is characterized in that Applied to the product side, including: Determine the requirements for the resources to be synchronized to access the resource center, label and resource group of the resource management side, and configure the resource attribute path corresponding to the resource to be synchronized to access the resource center, the label and the resource group according to the requirements; Determine the metadata and resource original data of the resource to be synchronized, wherein the metadata includes: a resource attribute path, generate a resource synchronization event of the resource to be synchronized according to the metadata and the resource original data, and send the resource synchronization event to the resource management side, wherein the metadata is description data of the resource to be synchronized, and the resource original data is used to indicate the attribute of the resource to be synchronized; Receive the storage result and cache content sent by the resource management side, query the storage result and the cache content to obtain the cache result, and use the distributed lock to change the cache result to obtain the change result, and send the change result to the resource management side. The storage result and the cache content are obtained by the resource management side according to the resource synchronization event, and the cache content is a priority query cache.
6. The method according to claim 5, It is characterized in that The stored result and the cached content are queried to obtain the cached result, including: The cache content is queried to determine whether the cache content exists. If so, a cache result is obtained. If not, the stored results are traversed to obtain a query result, and the query result is used as the cache content corresponding to the stored result.
7. The method according to claim 5, It is characterized in that The configuring, according to the requirement, that the resource to be synchronized access the resource center, the tag, and the resource attribute path corresponding to the resource group includes: According to the requirement, when accessing the resource center, configuring the first resource attribute path of the resource to be synchronized; When accessing the tag, configuring a second resource attribute path of the resource to be synchronized; When accessing a resource group, configuring a third resource attribute path of the resource to be synchronized; The first resource attribute path, the second resource attribute path and the third resource attribute path are integrated to obtain a resource attribute path.
8. A data synchronization device, It is characterized in that The device is applied to the resource management side, and includes: A receiving module, configured to receive a resource synchronization event sent by a product side, wherein the resource synchronization event includes metadata and resource original data, and parse the resource original data using the metadata to obtain resource associated data; A storage module, used for storing the resource association data, the metadata and the resource original data in a database, obtaining a storage result, storing the storage result in a third database, obtaining a cache content, and sending the storage result and the cache content to a product side; The first processing module is used to receive the change result of the cache content on the product side, determine the current cache value of the change result according to the change result, and write the current cache value into the database for data synchronization.
9. A data synchronization device, It is characterized in that The device is applied to the product side and includes: A configuration module, used to determine the requirements for the to-be-synchronized resource to access the resource center, label and resource group of the resource management side, and configure the resource attribute path corresponding to the to-be-synchronized resource to access the resource center, the label and the resource group according to the requirements; The second processing module is used to determine the metadata and resource original data of the resource to be synchronized, wherein the metadata includes: a resource attribute path, and generate a resource synchronization event of the resource to be synchronized according to the metadata and the resource original data, and send the resource synchronization event to the resource management side, wherein the metadata is the description data of the resource to be synchronized, and the resource original data is used to indicate the attribute of the resource to be synchronized; The third processing module is used to receive the storage results and cache contents sent by the resource management side, query the storage results and the cache contents to obtain the cache results, and use the distributed lock to change the cache results to obtain the change results, and send the change results to the resource management side. The storage results and the cache contents are obtained by the resource management side according to the resource synchronization event, and the cache content is a priority query cache.
10. A data synchronization device, It is characterized in that The device comprises: a memory and a processor; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory, so that the processor executes the data synchronization method according to any one of claims 1-4 or claims 5-7.
11. A computer-readable storage medium, It is characterized in that The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the data synchronization method as described in any one of claims 1-4 or claims 5-7.
12. A computer program product, comprising a computer program, which, when executed by a processor, implements the data synchronization method according to any one of claims 1 to 4 or claims 5 to 7.
Citation Information
Patent Citations
Method for sub-surface retrieval of MOOC course based on ElasticSearch
CN106776878A
Distributed data service processing method, system and device and storage medium
CN110633320A
Data synchronization processing method, system and device, electronic equipment and storage medium
CN117319416A
Enhancing HTTP caching by allowing content sharing of data blocks across resources identified by different uniform resource locators
US9378178B1
