Metadata processing method and system

Abstract

The invention provides a metadata processing method and system, and relates to the technical field of metadata, and the method comprises the steps: determining the delay effective time of metadata based on the duration consumed by a metadata consumption service in caching the metadata; the changed metadata, the identification information of the metadata, the update timestamp and the delay effective time are sent to a metadata consumption service, and the metadata consumption service can respond to an external business request based on the effective metadata of the latest version, the effective moment of the metadata is determined according to the update timestamp and the delay effective time of the metadata, so that in each metadata consumption service, the effective moments of the metadata of the same version are the same regardless of the time when the metadata information is received; therefore, the metadata of the same version can be used when business processing is carried out on each metadata consumption service, and the method can be applied to different business scenes such as e-commerce, risk control and asset management, so that the accuracy and reliability of business system implementation are ensured.

Description

Technical Field

[0001] This application relates to the field of metadata technology, and in particular to a metadata processing method and system. Background Technology

[0002] In today's rapidly developing information technology landscape, the data volume of core enterprise business systems (such as information systems built for core business processes like risk control, e-commerce, customer service, or asset management) is exploding, posing extremely serious challenges to data processing and transmission. To significantly improve the efficiency of system data access, caching technology has become widely used. Examples include price caching in e-commerce scenarios, risk rule caching in decentralized control scenarios, and asset status caching in asset management scenarios.

[0003] These business systems typically include multiple service nodes (such as front-end services, back-end interface services, and data storage services). Once the metadata required for business processing is cached across multiple service nodes, ensuring its consistency becomes a critical challenge. Especially in complex business processes, operations such as updating, deleting, and adding metadata can easily lead to inconsistencies in cached data across different service nodes. As a result, each service node may respond to requests in the business process based on different versions of metadata, negatively impacting the accuracy and reliability of the business system. For example, in a risk control scenario, if some front-end review nodes still use the cached old rules after risk rules are updated, they may be unable to effectively intercept high-risk orders.

[0004] Therefore, there is an urgent need to propose a metadata processing method that enables all service nodes to use the same metadata version when responding to requests, thereby ensuring the accuracy and reliability of business system implementation. Summary of the Invention

[0005] Therefore, it is necessary to provide a metadata processing method and system to address the aforementioned technical issues, enabling each service node to uniformly use the metadata when responding to requests, thereby ensuring the accuracy and reliability of the business system implementation.

[0006] Firstly, a metadata processing method is proposed. A metadata service acquires changed metadata and its identifier information. The metadata service generates an update timestamp for the metadata. The metadata service retrieves the metadata consumption service associated with the identifier information from a pre-stored metadata consumption record. A metadata request record records the correspondence between the metadata identifier information and the metadata consumption service that needs to use the metadata for business processing. The metadata service determines the delayed effective time of the metadata based on the time required for the metadata consumption service associated with the identifier information to cache the metadata. The metadata service locally caches the metadata information and sends a metadata change notification to the metadata consumption service, enabling the metadata consumption service to locally cache the metadata information in the metadata change notification. Upon receiving an external business request, the method determines the target metadata required to respond to the external business request based on the metadata information. The metadata information includes the metadata, the identifier information, the update timestamp, and the delayed effective time. The delayed effective time and the update timestamp are used to determine whether the metadata is effective.

[0007] In this embodiment of the application, the metadata service determines the delayed effective time of the metadata based on the time required for the metadata consumption service associated with the identification information to cache the metadata, including: obtaining the time required for each metadata consumption service associated with the identification information to cache the metadata, and filtering out the maximum time; and determining the maximum time as the delayed effective time of the metadata.

[0008] In this embodiment of the application, the method further includes: the metadata service receiving a first metadata acquisition request sent by the metadata consumption service, and acquiring target identification information in the first metadata acquisition request; the first metadata acquisition request is triggered by the metadata consumption service when it receives the external business request and does not cache the target metadata required to respond to the external business request locally; the target identification information is obtained by the metadata consumption service from the external business request; the metadata service searches for metadata information including the target identification information in local storage, and determines the latest version of effective metadata from the metadata in the metadata information; the metadata information including the latest version of effective metadata is sent to the metadata consumption service, so that the metadata consumption service caches the metadata information locally and uses the latest version of effective metadata as the target metadata.

[0009] In this embodiment, the metadata information further includes a version number; after the metadata service obtains the changed metadata and the identifier information of the metadata, it further includes: the metadata service receiving a second metadata acquisition request sent by the metadata consumption service, and acquiring the target version number and target identifier information contained in the second metadata acquisition request; the second metadata acquisition request is triggered by the metadata consumption service when it receives a call request sent by another metadata consumption service, and the target metadata required to respond to the call request is not cached locally; the target version number and the target identifier information are obtained by the metadata consumption service from the call request; the metadata service searches for metadata information including the target version number and the target identifier information in local storage, and sends the metadata information to the metadata consumption service, so that the metadata consumption service caches the metadata information locally and uses the metadata in the metadata information as the target metadata.

[0010] Secondly, a metadata processing method is proposed, comprising: a metadata consumption service receiving a metadata change notification sent by a metadata service; caching the metadata information in the metadata change notification locally; the metadata information including metadata, the metadata's identifier, an update timestamp, and a delayed effective time; the delayed effective time and the update timestamp being used to determine whether the metadata is effective; the delayed effective time being determined based on the time required for the metadata consumption service associated with the identifier to cache the metadata; the metadata consumption service associated with the identifier refers to a metadata consumption service that needs to use the metadata corresponding to the identifier for business processing; and when the metadata consumption service receives an external business request, determining the target metadata required to respond to the external business request based on the metadata information.

[0011] In this embodiment, when the metadata consumption service receives an external business request, it determines the target metadata required to respond to the external business request based on the metadata information. This includes: when the metadata consumption service receives the external business request, it parses target identification information from the external business request; the metadata consumption service determines whether the metadata corresponding to the target identification information is cached locally; if so, it determines the effective time of the metadata based on the update timestamp and delayed effective time corresponding to the metadata, and determines the latest version of the effective metadata from the metadata corresponding to the target identification information based on the effective time; the latest version of the effective metadata is used as the target metadata; if not, it determines that the target metadata required to respond to the external business request is not cached locally, and sends a first metadata retrieval request to the metadata service. The first metadata retrieval request includes the target identification information, so that the metadata service searches for metadata information including the target identification information in local storage, determines the latest version of the effective metadata from the metadata in the metadata information, and sends the metadata information including the latest version of the effective metadata to the metadata consumption service; the metadata consumption service caches the metadata information locally and uses the latest version of the effective metadata as the target metadata.

[0012] In this embodiment of the application, the metadata information also includes a version number, and the method further includes: in the process of responding to the external business request, if it is necessary to call other metadata consumption services, then add the target version number and target identifier information of the target metadata to the call request; pass the call request to the other metadata consumption services that need to be called, so that the other metadata consumption services use the metadata corresponding to the target version number and the target identifier information as the target metadata required to respond to the call request.

[0013] In this embodiment of the application, the method further includes: upon receiving a call request sent by another metadata consumption service, determining whether the local cache is used to cache metadata corresponding to the target version number and target identification information in the call request; if so, using the metadata corresponding to the target version number and target identification information as the target metadata required to respond to the call request; if not, determining that the local cache is not used to cache the target metadata required to respond to the call request, and sending a second metadata acquisition request to the metadata service, the second metadata acquisition request including the target version number and target identification information, so that the metadata service sends metadata information including the target version number and target identification information to the metadata consumption service; the metadata consumption service caches the metadata information locally and uses the metadata in the metadata information as the target metadata.

[0014] In this embodiment of the application, the method further includes: when the number of versions of the locally cached metadata exceeds a preset number of versions, sorting all update timestamps of the metadata in chronological order, deleting the version corresponding to the earliest update timestamp in the sequence, until the number of versions of the metadata does not exceed the preset number of versions.

[0015] Thirdly, a metadata processing system is proposed, the system comprising a first metadata processing device for providing metadata services and a second metadata processing device for providing metadata consumption services; the first metadata processing device is used to perform the steps of the metadata processing method described in any embodiment of the first aspect; the second metadata processing device is used to perform the steps of the metadata processing method described in any embodiment of the second aspect.

[0016] In summary, this application proposes a metadata processing method and system. In any of the schemes, when the metadata service obtains changed metadata, an update timestamp is generated for the metadata. Then, the metadata consumption service associated with the identifier information of the metadata is obtained from the pre-stored metadata consumption record. The metadata consumption record is used to record the correspondence between the identifier information of the metadata and the metadata consumption service that uses the metadata for business processing. Then, based on the time required for the metadata consumption service associated with the identifier information to cache the metadata, the delayed effective time of the metadata is determined. Finally, a metadata change notification is sent to the metadata consumption service, transmitting metadata information including the changed metadata, the identifier information of the metadata, the update timestamp, and the delayed effective time. At the same time, the metadata information is cached locally in the metadata service. The metadata consumption service will also cache the received metadata information locally. Then, when a response to an external business request is needed, it determines the metadata required to respond to that request based on the locally cached metadata. The update timestamp and delay time in the metadata information determine the effective time of the metadata. Therefore, regardless of when the metadata is received by each metadata consumption service, the effective time of the metadata is the same. This ensures that the same version of the effective metadata is applied when each metadata consumption service performs business processing, thereby ensuring the accuracy and reliability of the business system implementation. Furthermore, the effective time determined by the update timestamp and delay time in the metadata information ensures that the metadata takes effect after each metadata consumption service has completed caching. This avoids situations where a metadata consumption service has not yet received the latest version of the metadata but needs to use that version to respond to a request, resulting in frequent requests for metadata from the metadata service. Attached Figure Description

[0017] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is a flowchart illustrating a metadata processing method according to an exemplary embodiment of this application.

[0019] Figure 2 This is a flowchart illustrating a metadata processing method according to another exemplary embodiment of this application; Figure 3 This is a flowchart illustrating a metadata processing method according to another exemplary embodiment of this application; Figure 4 This is an application scenario diagram illustrating a metadata processing method according to another exemplary embodiment of this application; Figure 5 This is a flowchart illustrating a metadata processing method according to another exemplary embodiment of this application; Figure 6 This is a flowchart illustrating a metadata processing method according to another exemplary embodiment of this application; Figure 7 This is a block diagram illustrating a metadata processing system according to an exemplary embodiment of this application. Detailed Implementation

[0020] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. The embodiments described with reference to the accompanying drawings are exemplary and intended to explain this application, and should not be construed as limiting this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of this application.

[0021] Figure 1 This is a flowchart illustrating a metadata processing method according to an exemplary embodiment of this application.

[0022] For example, the metadata processing method proposed in this application can be applied to nodes in a distributed system architecture that provide metadata services. These nodes can be physical servers or virtual servers, etc. Alternatively, the steps of the above-described metadata processing method can be implemented by a device on the business system server that provides metadata services. Here, metadata service can be understood as a service for managing and providing metadata-related functions. For example, metadata service can be responsible for storing, organizing, retrieving, and maintaining metadata, providing metadata access interfaces to other systems and services, enabling these systems to easily obtain and use metadata information to support the implementation of various business functions and data processing flows.

[0023] like Figure 1 As shown, the metadata processing method proposed in this application may include the following steps: S101, the metadata service obtains the changed metadata and metadata identification information.

[0024] For example, the metadata service can respond to operations by business system operators or metadata change messages triggered by the business system itself, and obtain the changed metadata and metadata identification information.

[0025] Metadata is data used to describe data. For example, the metadata for a product may include category, attributes, and brand, etc.

[0026] The metadata identification information may include, but is not limited to, at least one of the following: object ID and entity ID.

[0027] An object ID is used to uniquely identify a specific metadata object, such as the ID used to identify the brand metadata mentioned above.

[0028] An entity ID is used to identify the entity described by the metadata. For example, the ID of the product mentioned above can serve as the entity ID for the product's brand metadata. When the product's metadata, such as its category, attributes, and brand, changes, the specific product whose metadata was modified can be identified through the product ID.

[0029] Here's an example of this step: Operations personnel modify the metadata of a promotional activity, such as the activity period, discount rules, and the range of products participating in the activity. The metadata service will then retrieve the changed metadata.

[0030] S102, the metadata service generates update timestamps for metadata.

[0031] For example, the metadata service can use the moment it receives a piece of metadata as the update timestamp. Alternatively, it can determine the update timestamp as the moment the metadata service completes the local caching of the metadata. It should be noted that this is merely an exemplary implementation of how to determine the update timestamp and is not intended to limit the scope of the implementation.

[0032] S103, the metadata service obtains the metadata consumption service associated with the identification information from the pre-stored metadata consumption record; the metadata consumption record is used to record the correspondence between the identification information of the metadata and the metadata consumption service that needs the metadata for business processing.

[0033] For example, metadata consumption service can be understood as a service that requires the use of metadata for business processing. For instance, in a customer acquisition scenario, metadata consumption service may include a user profiling service, which generates user profiles based on metadata (such as user tags). In a risk control scenario, metadata may include risk rules, blacklists, and threshold parameters, and metadata consumption service in this scenario may include a real-time review service; the real-time review service determines whether a transaction or operation is compliant in real time based on risk rules and blacklist metadata. It should be noted that this is merely an illustrative description of metadata and metadata consumption services and does not constitute a limitation on their content.

[0034] Metadata services can pre-store metadata consumption records in the form of a registry or directory, recording the identification information of the metadata and the metadata consumption services that need to use it. After obtaining the changed metadata and its identification information, the identification information can be used to search the aforementioned registry or directory to identify the metadata consumption service that needs to use the metadata as the metadata consumption service associated with that identification information.

[0035] In one example, the above metadata consumption record can be constructed in the following way: After initially acquiring a piece of metadata, the metadata service stores that metadata and its identifier. When the metadata consumption service needs to respond to an external business request but does not have the required metadata stored locally, it will request the metadata from the metadata service. Therefore, when responding to the request from the metadata consumption service and sending the metadata, the metadata service also records the metadata consumption service that requested the metadata. In this way, the metadata service can record the identifier information of each piece of metadata and the metadata consumption service that needs to use it.

[0036] It should be noted that metadata consumption records can also be generated in other ways. For example, by reviewing the configuration files or source code of various services involved in the business system, the reference or usage of various metadata can be determined, thereby identifying the metadata consumption service that needs to use a certain metadata, and thus pre-determining the identification information of the metadata and the correspondence between each metadata consumption service.

[0037] S104, The metadata service determines the delayed effective time of the metadata based on the time required for the metadata consumption service associated with the identification information to cache the metadata.

[0038] In this embodiment, the time required for the metadata consumption service to cache metadata can be understood as the total time required for the metadata stored in the metadata service to be transmitted and cached in the metadata consumption service. This total time can be the sum of the network transmission time between the metadata service and the metadata consumption service and the time required for the metadata consumption service to complete the local caching of the metadata.

[0039] In one example, the timestamp of sending metadata and the timestamp of the response message returned by the metadata consumption service can be recorded each time metadata is sent to the metadata consumption service. This response message is sent back to the metadata service by the metadata consumption service after completing the local caching of metadata.

[0040] The total time required to transmit and cache metadata stored in the metadata service to the metadata consumption service can be determined based on the timestamp of sending metadata and the timestamp of the response message returned by the metadata consumption service. The average of this total time can be used as the time mentioned in step S104 above.

[0041] In another example, the time required for each metadata consumption service to cache metadata can be obtained in other ways. For example, the time taken for the metadata consumption service to cache the acquired metadata locally can be obtained periodically from the metadata consumption service, and the network transmission time between the metadata service and the metadata consumption service can be obtained using a network performance testing tool. The sum of the two values ​​can then be determined as the time required for the metadata consumption service to cache metadata.

[0042] The above descriptions are merely examples of ways to obtain the metadata consumption service's time spent caching metadata, and are not intended to limit the implementation of these methods.

[0043] Now, the process of determining the delayed effective time is illustrated by an example: For example, if the metadata service determines that the identification information of a certain piece of metadata is associated with both metadata consumption service A and metadata consumption service B, and the time required for metadata consumption service A to cache this piece of metadata is h1, and the time required for metadata consumption service B to cache this piece of metadata is h2.

[0044] The maximum value between h1 and h2 can then be used as the maximum duration. This maximum duration can then be used as the delayed effective time for this metadata item. In this way, the metadata can be effective after all metadata consumption services have completed metadata storage, so that each metadata consumption service can promptly apply the latest version of the metadata to carry out business processing.

[0045] In another example, the delay period can be any value greater than this maximum duration. This ensures that metadata consuming services that need to use this metadata can cache the latest version of the metadata to the greatest extent possible.

[0046] S105, the metadata service caches metadata information locally and sends a metadata change notification to the metadata consumption service, so that the metadata consumption service caches the metadata information in the metadata change notification locally. When receiving an external business request, it determines the target metadata required to respond to the external business request based on the metadata information. The metadata information includes metadata, identification information, update timestamp, and delayed effective time; among which, the delayed effective time and update timestamp are used to determine whether the metadata is effective.

[0047] For example, the metadata service integrates the metadata information involved in each changed metadata. This metadata information may include, but is not limited to, the changed metadata, the metadata's identification information, the metadata's update timestamp, and the delayed effective time.

[0048] Then, the metadata information is stored locally. Simultaneously, this metadata information is written to a metadata change notification, which can then be sent asynchronously to each metadata consumption service associated with the metadata's identifier.

[0049] After receiving a metadata change notification, the metadata consumption service can parse the metadata information from the notification and then cache the parameters in the metadata information locally in a structured manner.

[0050] When the metadata consumption service receives an external business request, it will determine the target metadata required to respond to the external business request based on the metadata information cached locally by the metadata consumption service.

[0051] The delayed effective time of this metadata is the maximum time required for all associated metadata consuming services to cache it. The effective time, determined by the update timestamp and delayed effective time in the metadata information, ensures that the metadata takes effect after all metadata consuming services have completed caching. Therefore, regardless of when a metadata consuming service receives the metadata information, the effective time of the metadata is the same. This allows the same version of the metadata to be used when various metadata consuming services perform business processing (e.g., responding to external business requests or calls from other metadata consuming services), thereby ensuring the accuracy and reliability of the business system implementation.

[0052] In summary, this application proposes a metadata processing method. When a metadata service obtains changed metadata, it generates an update timestamp for the metadata. Then, it retrieves the metadata consumption service associated with the metadata's identification information from a pre-stored metadata consumption record. The metadata consumption record records the correspondence between the metadata's identification information and the metadata consumption service that needs to use the metadata for business processing. Then, based on the time required for the metadata consumption service associated with the identification information to cache the metadata, the delayed effective time of the metadata is determined. Finally, a metadata change notification is sent to the metadata consumption service, transmitting metadata information including the changed metadata, the metadata's identification information, the update timestamp, and the delayed effective time. Simultaneously, the metadata information is cached locally on the metadata service. The metadata consumption service will also cache the received metadata information locally. Then, when a response to an external business request is needed, the service determines the metadata required to respond to that request based on the locally cached metadata. The update timestamp and delay time in the metadata information determine the effective time of the metadata. Therefore, regardless of when the metadata is received by each metadata consumption service, the effective time of the metadata is the same. This ensures that the same version of metadata is applied when each service performs business processing, thereby ensuring the accuracy and reliability of the business system implementation. Furthermore, the effective time determined by the update timestamp and delay time ensures that the metadata takes effect after all metadata consumption services have completed caching. This avoids situations where a metadata consumption service has not yet received the latest version of the metadata but needs to use that version to respond to a request, resulting in frequent requests to the metadata service.

[0053] Based on the above embodiments, such as Figure 2 As shown, the metadata processing method proposed in this application embodiment may further include the following steps: S201, the metadata service receives a first metadata retrieval request sent by the metadata consumption service and retrieves the target identification information in the first metadata retrieval request; the first metadata retrieval request is triggered by the metadata consumption service when it receives an external business request and does not cache the target metadata required to respond to the external business request locally; the target identification information is obtained by parsing the external business request through the metadata consumption service.

[0054] For example, when the metadata consumption service receives an external business request, it can parse the target identifier information of the metadata from the request. The service can then search its local cache for the metadata information including the target identifier information. If the local cache does not contain the metadata information including the target identifier information, the service will send a first metadata retrieval request to the metadata service. This first metadata retrieval request includes the target identifier information provided in the external business request.

[0055] For example, in an e-commerce platform, changes to inventory metadata can affect multiple services such as product display, order placement, and shipping. When the front-end display interface is processing product recommendation business, it will trigger a product display request to the product display service. This product display request is an external business request.

[0056] The product display request includes the metadata object ID of the product to be displayed. When responding to the product display request, the product display service needs to obtain the product's metadata, such as the inventory metadata corresponding to the product. When the metadata consumption service does not cache any version of the inventory metadata locally, the metadata consumption service will send a first metadata retrieval request to the metadata service and provide the object ID of the inventory metadata in the request.

[0057] S202, The metadata service retrieves metadata information, including target identification information, from local storage, and determines the latest version of the effective metadata from the metadata information.

[0058] For example, after receiving a first metadata retrieval request sent by the metadata consumption service, the metadata service parses the target identification information from the first metadata retrieval request, then searches for the metadata corresponding to the target identification information in the metadata stored by the metadata service, and then determines whether the metadata is effective based on the update timestamp and delayed effective time of each metadata found, and then determines the latest version of the effective metadata from the effective metadata.

[0059] S203, send metadata information including the latest version of the effective metadata to the metadata consumption service so that the metadata consumption service can cache the metadata information locally and use the latest version of the effective metadata as the target metadata.

[0060] For example, after determining the latest version of the effective metadata corresponding to the target identifier information, the metadata service sends metadata information including the latest version of the effective metadata to the metadata consumption service. This metadata information may include, but is not limited to, the latest version of the effective metadata, the identifier information of the latest version of the effective metadata, and the delayed effective time of the latest version of the effective metadata.

[0061] After receiving the metadata information, the metadata consumption service performs a structured cache of the metadata information locally, and then uses the latest version of the effective metadata in the metadata information as the target metadata to respond to the aforementioned external business requests.

[0062] In this embodiment, when the metadata consumption service does not cache the target metadata required to respond to external business requests locally, it obtains the latest version of the effective metadata corresponding to the target identifier information from the metadata service. Since the latest version of the effective metadata is determined based on the update timestamp and delayed effective time determined by the metadata service after obtaining the metadata, the effective metadata is the metadata that has been effective in each of its associated metadata consumption services, and it is the latest version of the effective metadata. Therefore, different metadata consumption services can use the latest version of the effective metadata to respond to external business requests.

[0063] For example, the front end of a trading platform sends a product display request to a product display service. This request includes the inventory metadata ID of product A. If the product display service does not find any version of the metadata corresponding to this inventory metadata ID in its local cache, it confirms that the product display service does not have the metadata corresponding to this inventory metadata ID cached locally. Therefore, it sends a first metadata retrieval request to the metadata service. The metadata service stores two versions of metadata corresponding to the inventory metadata ID: metadata 1 with an update timestamp of 10:31:20 and a delay of 1 second, and metadata with an update timestamp of 15:01:20 and a delay of 1.01 seconds. If the current time is 15:01:22, the metadata service determines that the latest version of the effective metadata is metadata 2, and sends metadata 2, its identifier, update timestamp, and delay to the product display service.

[0064] In one embodiment, the metadata service may locally record a correlation between the target identifier information in the first metadata retrieval request and the metadata consumption service that sent the first metadata retrieval request. This allows the metadata consumption service to incorporate the time required to cache the metadata when determining the delayed effective time of the new metadata after receiving new modified metadata of the target identifier information. This allows for continuous updating of the metadata's delayed effective time based on actual application conditions.

[0065] In the above embodiments, the metadata information may further include a version number, which can be generated for the metadata after the metadata service obtains the changed metadata and the identification information of the metadata.

[0066] In this embodiment, the version number is a version identifier for metadata at different stages, used to distinguish different versions of metadata content of the same metadata object.

[0067] In one example, the version number can be generated based on certain rules according to the update timestamp of the metadata, or it can be generated randomly. This application does not impose any restrictions.

[0068] In one embodiment, the network state between the metadata service and the metadata consumption service is complex and variable. Therefore, even if a delay is set to ensure that each metadata consumption service can simultaneously activate a certain version of metadata, factors such as network performance and system load may prevent the metadata information transmitted by the metadata service to different metadata consumption services from being delivered on time and accurately. This can easily lead to a situation where some metadata consumption services have completed the storage of metadata information, and the metadata in that information has reached its activation time, but some metadata consumption services have not yet received the metadata information. To address this situation, such as... Figure 3 As shown, the metadata processing method proposed in this application embodiment further includes the following steps: S301, the metadata service receives a second metadata retrieval request sent by the metadata consumption service, and retrieves the target version number and target identification information contained in the second metadata retrieval request; the second metadata retrieval request is triggered by the metadata consumption service when it receives a call request sent by another metadata consumption service, and the target metadata required to respond to the call request is not cached locally; the target version number and target identification information are obtained by the metadata consumption service from the call request.

[0069] For example, a metadata consumption service can directly receive external business requests or receive calls from other metadata consumption services.

[0070] like Figure 4As shown, after metadata changes, the metadata service asynchronously sends a metadata change notification to metadata consumption services A, B, and C. For example, if metadata consumption service A is the first metadata consumption service to receive an external business request, then metadata consumption service A uses the target metadata corresponding to the target identifier information in the external business request to respond to the external business request. During the response to the external business request, if it is necessary to call metadata consumption service B, a call request is sent to metadata consumption service B, including the aforementioned target identifier information and target version number. After receiving the call request from metadata consumption service A, metadata consumption service B parses the target identifier information and target version number from the call request, and then searches its local cache for the metadata information that corresponds to both the target identifier information and target version number. If it exists, it uses the metadata information that corresponds to both the target identifier information and target version number to respond to the call request. If it is necessary to call metadata consumption service C during the response to the call request, then metadata consumption service B sends a call request to metadata consumption service C, including the aforementioned target identifier information and target version number.

[0071] Upon receiving a call request, the metadata consumption service searches its local cache for the metadata corresponding to the target identifier and target version number in the call request. If the metadata is found, it uses the metadata corresponding to the target identifier and target version number as the target metadata and responds to the call request based on this target metadata. If the metadata is not found, the metadata consumption service sends a second metadata retrieval request to the metadata service, and writes the target identifier and target version number into the second metadata retrieval request.

[0072] S302, the metadata service searches for metadata information including the target version number and target identifier information in the local storage, and sends the metadata information to the metadata consumption service so that the metadata consumption service can cache the metadata information locally and use the metadata in the metadata information as the target metadata.

[0073] After receiving the second metadata retrieval request, the metadata service parses the target identifier information and target version number from the second metadata retrieval request, then searches for the metadata that corresponds to the target version number and target identifier information in the local storage, and integrates the metadata information including the metadata, the update timestamp of the metadata, the delayed effective time of the metadata, the target version number, and the target identifier information.

[0074] The metadata service then sends this metadata information to the metadata consumption service, which caches the metadata information locally and uses the metadata in this metadata information as the target metadata. The service then responds to the call request based on this target metadata.

[0075] In this embodiment, after any metadata consumption service receives a call request from another metadata consumption service, it uses the target version number and target identifier information provided in the call request to determine the target metadata required to respond to the call request. This unifies the metadata version used by each metadata consumption service when conducting business processing, avoiding situations where the processing results of different metadata consumption services are inconsistent due to the use of different versions of metadata. Furthermore, since the metadata stored in each metadata consumption service has a corresponding delayed effective time, and this delayed effective time is the maximum time required for a metadata consumption service to store the metadata, when a metadata consumption service responds to an external business request using an already effective version of metadata, even if other metadata consumption services need to be called, most metadata consumption services have already completed the storage of that version of metadata. Only in special cases will a small number of metadata consumption services fail to complete the storage of that version of metadata. Therefore, only a small number of metadata consumption services that have not completed the storage of that version of metadata will send a second metadata retrieval request to the metadata service, thereby further reducing the network pressure between the metadata service and the metadata consumption services and improving the response efficiency of the request.

[0076] This application also proposes another metadata processing method. This method can be applied to nodes in a distributed system architecture that provide metadata consumption services. These nodes can be physical servers or virtual servers, or devices on a business system server that provide metadata consumption services. The metadata consumption service can be understood as a service that requires the use of metadata to perform business processing.

[0077] Figure 5 This is a metadata processing method illustrated in an exemplary embodiment of this application.

[0078] like Figure 5 As shown, the metadata processing method may include the following steps: S501, the metadata consumption service receives a metadata change notification sent by the metadata service, and caches the metadata information in the metadata change notification locally. The metadata information includes metadata, metadata identification information, update timestamp, and delayed effective time. The delayed effective time and update timestamp are used to determine whether the metadata has taken effect. The delayed effective time is determined based on the time required for the metadata consumption service associated with the identification information to cache the metadata. The metadata consumption service associated with the identification information refers to the metadata consumption service that needs to use the metadata corresponding to the identification information for business processing.

[0079] S502, When the metadata consumption service receives an external business request, it determines the target metadata required to respond to the external business request based on the metadata information.

[0080] It should be noted that the implementation methods of the above steps can refer to the description of the embodiments of each step of the metadata service above, and will not be repeated here.

[0081] In summary, this application proposes a metadata processing method in which the metadata consumption service locally caches the received metadata information. Then, when a response to an external business request is needed, the cached metadata information is used to determine the metadata required to respond to that request. The metadata information includes metadata, its update timestamp, its identifier, and a delayed effective time. The update timestamp and delayed effective time determine the effective time of the metadata. Therefore, regardless of when the metadata information is received by each metadata consumption service, the effective time of the metadata is the same. This allows the same version of metadata to be used when each metadata consumption service performs business processing, ensuring the accuracy and reliability of the business system implementation. Furthermore, the effective time determined based on the update timestamp and delayed effective time ensures that the metadata takes effect after each metadata consumption service has completed caching. This avoids situations where a metadata consumption service has not yet received the latest version of the metadata but needs to use that version to respond to a request, leading to frequent requests for metadata from the metadata service.

[0082] In some embodiments, such as Figure 6 As shown, step S502 above, "when the metadata consumption service receives an external business request, it determines the target metadata required to respond to the external business request based on the metadata information," may include the following steps: S601, When the metadata consumption service receives an external business request, it parses the target identification information from the external business request; S602, the metadata consumption service determines whether the metadata corresponding to the target identifier information is cached locally; S603, if so, then determine the effective time of the metadata based on the update timestamp and delayed effective time corresponding to the metadata, so as to determine the latest version of the effective metadata from the metadata corresponding to the target identifier information based on the effective time.

[0083] For example, a delay time is added to the update timestamp to obtain the effective time of the metadata. The current time is compared with the effective time. If the current time is greater than or equal to the effective time, it is determined that the metadata has taken effect. Then, the metadata with the largest update timestamp or the latest metadata is determined from the effective metadata and used as the latest version of the effective metadata.

[0084] S604 uses the latest version of the effective metadata as the target metadata.

[0085] S605, if not, determine that the target metadata required to respond to external business requests is not cached locally, send a first metadata retrieval request to the metadata service, the first metadata retrieval request includes target identification information, so that the metadata service can find the metadata information including the target identification information in the local storage, determine the latest version of the effective metadata from the metadata information, and send the metadata information including the latest version of the effective metadata to the metadata consumption service.

[0086] S606, the metadata consumption service caches metadata information locally and uses the latest version of the effective metadata as the target metadata.

[0087] It should be noted that the implementation methods of the above steps can refer to the description of the embodiments of each step of the metadata service above, and will not be repeated here.

[0088] In some embodiments, the metadata information also includes a version number. The metadata processing method of this application embodiment may further include the following steps: In the process of responding to external business requests, if it is necessary to call other metadata consumption services, the target version number and target identifier information of the target metadata should be added to the call request. The call request is passed to other metadata consumption services that need to be called, so that the other metadata consumption services use the metadata corresponding to the target version number and target identification information as the target metadata required to respond to the call request.

[0089] In some embodiments, the metadata processing method of this application may further include the following steps: When a call request is received from another metadata consumption service, it is determined whether the local cache contains metadata corresponding to the target version number and target identifier information in the call request. If so, the metadata corresponding to both the target version number and the target identifier information will be used as the target metadata required to respond to the call request; If not, it is determined that the target metadata required for the response call request is not cached locally, and a second metadata retrieval request is sent to the metadata service. The second metadata retrieval request includes the target version number and target identification information, so that the metadata service sends the metadata information including the target version number and target identification information to the metadata consumption service. The metadata consumption service caches metadata information locally and uses the metadata in the metadata information as the target metadata.

[0090] It should be noted that the implementation methods of the above steps can refer to the description of the embodiments of each step of the metadata service above, and will not be repeated here.

[0091] In some embodiments, the metadata processing method proposed in this application may further include the following steps: If the number of versions of metadata cached locally exceeds the preset number of versions, sort all update timestamps of the metadata in chronological order, delete the version corresponding to the earliest update timestamp in the sequence, until the number of versions of metadata does not exceed the preset number of versions.

[0092] For example, each metadata consumption service can periodically clear historical versions of metadata, saving only the metadata of the N latest versions. The preset number of versions is N, where N is a positive integer greater than 1.

[0093] This embodiment caches multiple versions of metadata in the metadata consumption service, improving the service's flexibility and robustness in the face of data conflicts or the need for rollback, ensuring that the service can always retrieve data when pulling from the cache. Regularly clearing historical versions of metadata frees up local storage resources, reduces storage pressure, and facilitates data retrieval.

[0094] The metadata processing method proposed in this application is applicable to distributed systems with high data consistency requirements, frequent data updates, and complex business chains, such as e-commerce, financial transactions, and big data analysis. By setting a delayed effective time, this application constructs a delayed effective mechanism for metadata, which can gradually and safely promote changed metadata to the entire distributed system without affecting business continuity.

[0095] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0096] This application also proposes a metadata processing system, such as... Figure 7 As shown, the metadata processing system 700 may include a first metadata processing device 701 that provides metadata services and a second metadata processing device 702 that provides metadata consumption services.

[0097] The first metadata processing device 701 is used to execute the steps of the metadata processing method implemented by the metadata service in the above embodiments; the second metadata processing device 702 is used to execute the steps of the metadata processing method implemented by the metadata consumption service in the above embodiments.

[0098] In one example, there may be multiple second metadata processing devices 702. The first metadata processing device 701 and the second metadata processing device 702 may be located in a distributed system.

[0099] It is understood that, based on the corresponding beneficial effects of the above embodiments, the metadata processing system of this embodiment should also have corresponding technical effects. To avoid redundancy, it will not be described in detail here.

[0100] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0101] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is used as an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0102] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A metadata processing method, characterized in that, The method includes: The metadata service obtains the changed metadata and the identification information of the metadata; The metadata service generates update timestamps for the metadata; The metadata service retrieves the metadata consumption service associated with the identification information from the pre-stored metadata consumption records; the metadata consumption records are used to record the correspondence between the identification information of the metadata and the metadata consumption services that need to use the metadata for business processing. The metadata service determines the delayed effective time of the metadata based on the time required for the metadata consumption service associated with the identification information to cache the metadata. The metadata service caches metadata information locally and sends metadata change notifications to the metadata consumption service, enabling the metadata consumption service to cache the metadata information in the metadata change notifications locally. When receiving an external business request, the service determines the target metadata required to respond to the external business request based on the metadata information. The metadata information includes the metadata, the identification information, the update timestamp, and the delayed effective time. The delayed effective time and the update timestamp are used to determine whether the metadata is effective.

2. The method according to claim 1, characterized in that, The metadata service determines the delayed effective time of the metadata based on the time required for the metadata consumption service associated with the identification information to cache the metadata, including: Obtain the time required for each metadata consumption service associated with the identification information to cache the metadata, and filter out the maximum time. The maximum duration is determined as the delayed effective time of the metadata.

3. The method according to claim 1, characterized in that, The method further includes: The metadata service receives a first metadata retrieval request sent by the metadata consumption service, and retrieves the target identification information from the first metadata retrieval request; the first metadata retrieval request is triggered by the metadata consumption service when it receives the external business request and does not cache the target metadata required to respond to the external business request locally; the target identification information is obtained by the metadata consumption service from the external business request; The metadata service retrieves metadata information including the target identifier from local storage, and determines the latest version of the effective metadata from the metadata information. The metadata information, including the latest version of the effective metadata, is sent to the metadata consumption service so that the metadata consumption service can cache the metadata information locally and use the latest version of the effective metadata as the target metadata.

4. The method according to claim 1, characterized in that, The metadata information also includes a version number; After obtaining the changed metadata and the identifier information of the metadata, the metadata service further includes: Generate the version number for the metadata; The method further includes: The metadata service receives a second metadata retrieval request sent by the metadata consumption service, and retrieves the target version number and target identifier information contained in the second metadata retrieval request; the second metadata retrieval request is triggered by the metadata consumption service when it receives a call request sent by another metadata consumption service, and the target metadata required to respond to the call request is not cached locally; the target version number and the target identifier information are obtained by the metadata consumption service from the call request; The metadata service searches for metadata information including the target version number and the target identifier information in local storage, and sends the metadata information to the metadata consumption service so that the metadata consumption service caches the metadata information locally and uses the metadata in the metadata information as the target metadata.

5. A metadata processing method, characterized in that, The method includes: The metadata consumption service receives metadata change notifications from the metadata service and caches the metadata information in the notifications locally. The metadata information includes metadata, its identifier, an update timestamp, and a delayed effective time. The delayed effective time and the update timestamp are used to determine whether the metadata has taken effect. The delayed effective time is determined based on the time required for the metadata consumption service associated with the identifier to cache the metadata. The metadata consumption service associated with the identifier refers to the metadata consumption service that needs to use the metadata corresponding to the identifier for business processing. When the metadata consumption service receives an external business request, it determines the target metadata required to respond to the external business request based on the metadata information.

6. The method according to claim 5, characterized in that, When the metadata consumption service receives an external business request, it determines the target metadata required to respond to the external business request based on the metadata information, including: When the metadata consumption service receives the external business request, it parses the target identification information from the external business request. The metadata consumption service determines whether the metadata corresponding to the target identifier information is cached locally. If so, the effective time of the metadata is determined based on the update timestamp and delayed effective time corresponding to the metadata, so as to determine the latest version of the effective metadata from the metadata corresponding to the target identifier information based on the effective time; The latest version of the effective metadata is used as the target metadata; If not, it is determined that the target metadata required to respond to the external business request is not cached locally, and a first metadata retrieval request is sent to the metadata service. The first metadata retrieval request includes the target identification information, so that the metadata service can find the metadata information including the target identification information in the local storage, determine the latest version of the effective metadata from the metadata in the metadata information, and send the metadata information including the latest version of the effective metadata to the metadata consumption service. The metadata consumption service caches the metadata information locally and uses the latest version of the effective metadata as the target metadata.

7. The method according to claim 5, characterized in that, The metadata information also includes a version number, and the method further includes: In the process of responding to the external business request, if it is necessary to call other metadata consumption services, the target version number and target identifier information of the target metadata shall be added to the call request; The call request is passed to other metadata consumption services that need to be called, so that the other metadata consumption services use the metadata corresponding to the target version number and the target identifier information as the target metadata required to respond to the call request.

8. The method according to claim 5, characterized in that, The method further includes: When receiving a call request from another metadata consumption service, determine whether the local cache is used to cache the metadata corresponding to the target version number and target identifier information in the call request; If so, the metadata corresponding to the target version number and the target identifier information shall be used as the target metadata required to respond to the call request; If not, it is determined that the target metadata required to respond to the call request is not cached locally, and a second metadata retrieval request is sent to the metadata service. The second metadata retrieval request includes the target version number and target identification information, so that the metadata service sends the metadata information including the target version number and the target identification information to the metadata consumption service. The metadata consumption service caches the metadata information locally and uses the metadata in the metadata information as the target metadata.

9. The method according to claim 5, characterized in that, The method further includes: If the number of versions of metadata cached locally exceeds the preset number of versions, all update timestamps of the metadata are sorted in chronological order, and the version corresponding to the earliest update timestamp in the sequence is deleted until the number of versions of the metadata does not exceed the preset number of versions.

10. A metadata processing system, characterized in that, The system includes a first metadata processing device that provides metadata services and a second metadata processing device that provides metadata consumption services. The first metadata processing device is used to perform the steps of the metadata processing method according to any one of claims 1-4; The second metadata processing device is used to perform the steps of the metadata processing method according to any one of claims 5-9.

Images