Data processing method and system, configuration center and gateway

Abstract

The invention discloses a data processing method and system, a configuration center and a gateway, and relates to the technical field of micro-service architecture application, and the data processing method comprises the following steps: reading metadata from a database of a first configuration center every preset duration, converting and processing the metadata, and storing the metadata in a database of a second configuration center, the first configuration center is used for managing metadata in response to a metadata management request of the gateway; when it is detected that the version of the metadata currently cached in the memory of the second configuration center changes, reading the metadata of the target version from the database of the second configuration center, and caching the metadata of the target version into the memory of the second configuration center; and in response to a metadata query request of the gateway, reading target data matched with the metadata loading condition from the memory of the second configuration center, and returning the target data. The availability problem caused by unstable management side function and physical environment of the gateway is solved, and the availability of the gateway is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of micro-service architecture application, and particularly relates to a data processing method and system, a configuration center and a gateway. BACKGROUND

[0002] Currently, a micro-service gateway performs a timing task polling to pull metadata required by each function from a configuration center, and when a user terminal requests the micro-service gateway, the micro-service gateway processes according to the pulled metadata to implement functions such as authentication, authorization, forwarding and flow limiting in a multi-tenant environment. The micro-service gateway needs to pull all metadata required by each function from the configuration center for processing, and when the gateway is unstable due to the influence of management side functions and physical environment, the availability of the gateway becomes poor. SUMMARY

[0003] The main purpose of the present application is to provide a data processing method, system, configuration center and gateway, which aims to solve the availability problem caused by the instability of the management side functions and the physical environment.

[0004] To achieve the above purpose, the present application provides a data processing method applied to a second configuration center, comprising:

[0005] Every preset time interval, read the metadata from the database of the first configuration center, and store the converted and processed metadata in the database of the second configuration center, wherein the first configuration center is used to respond to the metadata management request of the gateway to manage the metadata;

[0006] When detecting that the current cached metadata version in the memory of the second configuration center changes, read the metadata of the target version from the database of the second configuration center, and cache the metadata of the target version in the memory of the second configuration center;

[0007] In response to a metadata query request of a gateway, read target data matching a metadata loading condition from the memory of the second configuration center, and return the target data.

[0008] In an embodiment, the step of responding to a metadata query request of a gateway, reading target data matching a metadata loading condition from the memory of the second configuration center, and returning the target data comprises:

[0009] In response to metadata query requests of multiple gateways, group each gateway, wherein the metadata loading conditions of each gateway in the same group are the same, and each group corresponds to a different metadata loading condition;

[0010] Respectively read target data matching each metadata loading condition from the memory of the second configuration center;

[0011] The target data matched with the metadata loading condition is returned to all gateways in a group corresponding to the metadata loading condition.

[0012] In an embodiment, the reading of the metadata from the database of the first configuration center every preset time interval for conversion and processing and the storage of the preprocessed metadata in the database of the second configuration center include:

[0013] The metadata is read from the database of the first configuration center every preset time interval.

[0014] The metadata is format-converted, merged and combined to obtain preprocessed metadata.

[0015] Based on the content of the preprocessed metadata, a corresponding version identifier is set for the preprocessed metadata.

[0016] The preprocessed metadata and the version identifier corresponding to the preprocessed metadata are stored in the database of the second configuration center.

[0017] In an embodiment, the reading of the metadata from the database of the first configuration center every preset time interval includes:

[0018] Instance data, cluster data, template data and permission data are read from the database of the first configuration center every preset time interval.

[0019] The instance data, the cluster data, the template data and the permission data are taken as the metadata.

[0020] In addition, to achieve the above-mentioned purpose, the application further provides a data processing method applied to a first configuration center, comprising:

[0021] In response to a metadata management request of a gateway, at least one data management mode is determined.

[0022] Metadata is processed based on the data management mode.

[0023] The processed metadata is stored in the database of the first configuration center, so that the second configuration center reads the metadata from the database of the first configuration center every preset time interval for conversion and processing and stores the preprocessed metadata in the database of the second configuration center; when the second configuration center detects a change in the version of the metadata currently cached in the memory of the second configuration center, the second configuration center reads metadata of a target version from the database of the second configuration center and caches the metadata of the target version in the memory of the second configuration center; in response to a metadata query request of a gateway, the second configuration center reads target data matched with a metadata loading condition from the memory of the second configuration center and returns the target data.

[0024] In an embodiment, the metadata comprises at least one of an instance, a cluster, a template, and a permission, and the corresponding data management mode comprises at least one of instance addition, cluster addition, template addition, and permission modification.

[0025] In addition, to achieve the above object, the application further provides a data processing method applied to a gateway, comprising:

[0026] reading target data from the memory of the second configuration center every preset time length, and backing up the target data to the disk and the memory of the gateway;

[0027] reading backup data from the disk in response to a starting instruction of the gateway;

[0028] when it is detected that the data reading time is reached and the metadata version of the gateway is changed, reading the latest target data from the memory of the second configuration center, and caching the read latest target data in the disk and the memory of the gateway.

[0029] In addition, to achieve the above object, the application further provides a data processing system, comprising a first configuration center, a second configuration center, and a gateway, wherein,

[0030] the first configuration center is configured to determine at least one data management mode in response to a metadata management request of the gateway, process metadata based on the data management mode, and store the processed metadata in a database of the first configuration center;

[0031] the second configuration center is configured to read metadata from the database of the first configuration center every preset time length, store the converted and processed metadata in a database of the second configuration center, wherein the first configuration center is configured to manage metadata in response to a metadata management request of the gateway, read metadata of a target version from the database of the second configuration center when it is detected that the version of the currently cached metadata in the memory of the second configuration center is changed, cache the metadata of the target version in the memory of the second configuration center, and read target data matching a metadata loading condition from the memory of the second configuration center in response to a metadata query request of the gateway, and return the target data;

[0032] The gateway is configured to read target data from the memory of the second configuration center every preset time length, and backup the target data to the disk and the memory of the gateway; when the gateway starts, read the backup data from the disk; when it is detected that the data reading time is reached and the metadata version of the gateway is changed, read the latest target data from the memory of the second configuration center, and cache the read latest target data in the disk and the memory of the gateway.

[0033] In addition, to achieve the above-mentioned purpose, the application further provides a second configuration center, comprising: a memory, a processor and a computer program stored on the memory and executable on the processor, and the computer program is configured to implement the steps of the above-mentioned data processing method.

[0034] In addition, to achieve the above-mentioned purpose, the application further provides a first configuration center, comprising: a memory, a processor and a computer program stored on the memory and executable on the processor, and the computer program is configured to implement the steps of the above-mentioned data processing method.

[0035] In addition, to achieve the above-mentioned purpose, the application further provides a gateway, comprising: a memory, a processor and a computer program stored on the memory and executable on the processor, and the computer program is configured to implement the steps of the above-mentioned data processing method.

[0036] The application can read metadata from the database of the first configuration center every preset time length, and cache the read metadata in the database of the second configuration center; when it is detected that the version of the currently cached metadata in the memory of the second configuration center is changed, read the metadata of the target version from the database of the second configuration center, and cache the metadata of the target version to the memory of the second configuration center; in response to the metadata query request of the gateway, read the target data matching the metadata loading condition from the memory of the second configuration center, and return the target data. Since two configuration centers are set, one is used for managing metadata, and the other is used for querying metadata, the processing of metadata can be pre-processed through the cooperation of the two configuration centers, and the appropriate target data can be matched according to the metadata loading condition of the gateway subsequently. In this way, the gateway does not need to pull all the metadata required by the function for processing, solves the availability problem caused by the instability of the management side function and physical environment of the gateway, reduces the metadata processing pressure of the gateway, and improves the availability of the gateway. BRIEF DESCRIPTION OF DRAWINGS

[0037] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the application and, together with the description, serve to explain the principles of the application.

[0038] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the accompanying drawings needed to be used in the embodiments or prior art description will be briefly introduced as follows. Obviously, for those of ordinary skill in the art, the other drawings can also be obtained based on these drawings without any creative effort.

[0039] Figure 1 The structure schematic diagram of the data processing system of the present application is shown in the figure.

[0040] Figure 2 The flow schematic diagram provided by the first embodiment of the data processing method of the present application is shown in the figure.

[0041] Figure 3 The schematic diagram of the gateway grouping and data request of the present application is shown in the figure.

[0042] Figure 4 The flow schematic diagram provided by the second embodiment of the data processing method of the present application is shown in the figure.

[0043] Figure 5 The flow schematic diagram provided by the third embodiment of the data processing method of the present application is shown in the figure.

[0044] The purpose realization, functional features and advantages of the present application will be further explained in combination with the embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0045] It should be understood that the specific embodiments described herein are only used to explain the technical solutions of the present application, and are not used to limit the present application.

[0046] In order to better understand the technical solutions of the present application, the following will be described in detail in combination with the accompanying drawings and specific embodiments.

[0047] At present, the micro-service gateway performs timing task polling to pull the metadata required by each function from the configuration center, and when the user terminal requests the micro-service gateway, the micro-service gateway processes according to the pulled metadata, and realizes the functions of authentication, authorization, forwarding, flow limiting and the like in the multi-tenant environment. The micro-service gateway needs to pull all the metadata required by each function from the configuration center for processing, and when the gateway is affected by the management side function and the physical environment, it will lead to poor availability of the gateway.

[0048] To solve the above problems, the application provides a data processing method, and the main technical scheme comprises the following steps: a second configuration center reads metadata from a database of a first configuration center every preset time length, and stores the metadata in the database of the second configuration center after conversion and processing, wherein the first configuration center is used for managing metadata in response to a metadata management request of a gateway; when it is detected that the version of the currently cached metadata in the memory of the second configuration center changes, the second configuration center reads metadata of a target version from the database of the second configuration center, and caches the metadata of the target version in the memory of the second configuration center; and in response to a metadata query request of the gateway, the second configuration center reads target data matching a metadata loading condition from the memory of the second configuration center, and returns the target data. Since two configuration centers are provided, one is used for managing metadata, and the other is used for querying metadata, the processing of the metadata can be preprocessed through the cooperation of the two configuration centers, and the appropriate target data can be matched according to the metadata loading condition of the gateway, so that the gateway does not need to pull all the metadata required by the function from the configuration center for processing, the availability problem of the gateway caused by the instability of the management side function and the physical environment is solved, the metadata processing pressure of the gateway is reduced, and the availability of the gateway is improved.

[0049] In addition, in the related art, the configuration management and the query function of the metadata are coupled in one configuration center. However, in fact, the management side function frequently changes, and the query side function is relatively stable. Each change needs to upgrade the configuration center service, which increases the maintenance cost and complexity. Therefore, the application provides two configuration centers, and the configuration management function and the query function are decoupled and executed by different configuration centers. The first configuration center is upgraded each time, and the second configuration center is updated as needed when the version changes, so that the maintenance cost of the metadata is reduced.

[0050] Based on this, the application provides a data processing system, which is described with reference to Figure 1 , Figure 1 The data processing system comprises a first configuration center, a second configuration center and a gateway, wherein the second configuration center is connected with the first configuration center and the gateway. In addition, the data processing system further comprises a database and a memory corresponding to the second configuration center, and a database corresponding to the first configuration center. The gateway is provided with corresponding disks and memories, wherein the disks are used for storing backup metadata.

[0051] The difference between the above-mentioned second configuration center and the first configuration center is that the two can be physically isolated and used for executing different functions. The second configuration center is mainly used for responding to a metadata query request of the gateway and executing a query function, and the first configuration center is mainly used for responding to a metadata management request of the gateway and executing a metadata management function.

[0052] There can be multiple gateways, and each gateway has a corresponding memory. The second configuration center includes two parts, view synchronization (i.e. metadata synchronization) and view query (i.e. metadata query). The view service synchronizes metadata from the database of the first configuration center regularly. The data structure required for the ElasticSearch gateway to run is pre-converted, and the version identifier corresponding to the metadata is set, and finally the metadata and version information are saved to the database of the view service. The view service maintains a memory cache with version information. When the view version changes, the service automatically loads the latest view data from the database into memory. In addition, the view service provides corresponding query interfaces, which can be directly called by the ElasticSearch gateway service to read the required data from the memory cache without any additional conversion.

[0053] The first configuration center is configured to determine at least one data management mode in response to a metadata management request of a gateway, process metadata based on the data management mode, and store the processed metadata in a database of the first configuration center, so that the second configuration center reads the metadata from the database of the first configuration center at a preset time interval, processes and converts the metadata, and stores the processed metadata in a database of the second configuration center.

[0054] Optionally, the metadata includes at least one of an instance, a cluster, a template, and a permission, and the corresponding data management mode includes at least one of instance addition or reduction, cluster addition or reduction, template addition or reduction, and permission modification.

[0055] The second configuration center is configured to read metadata from the database of the first configuration center at a preset time interval, process and convert the metadata, and store the processed metadata in a database of the second configuration center. The first configuration center is configured to manage metadata in response to a metadata management request of a gateway. When a change in the version of the currently cached metadata in the memory of the second configuration center is detected, the second configuration center reads metadata of a target version from the database of the second configuration center and caches the metadata of the target version in the memory of the second configuration center. In response to a metadata query request of a gateway, the second configuration center reads target data matching a metadata loading condition from the memory of the second configuration center and returns the target data.

[0056] Optionally, the second configuration center is configured to group the plurality of gateways in response to the metadata query request, wherein the metadata loading conditions of the gateways in a same group are identical, and each group corresponds to a different metadata loading condition; read target data matching the metadata loading condition from the memory of the second configuration center; and return the target data matching the metadata loading condition to all gateways in the group corresponding to the metadata loading condition.

[0057] Optionally, the second configuration center is configured to read metadata from the database of the first configuration center every preset time length; perform format conversion, merging and combination on the metadata to obtain preprocessed metadata; set a corresponding version identifier for the preprocessed metadata based on the content of the preprocessed metadata; and store the preprocessed metadata and the version identifier corresponding to the preprocessed metadata in the database of the second configuration center.

[0058] Optionally, the second configuration center is configured to read instance data, cluster data, template data and permission data from the database of the first configuration center every preset time length; and take the instance data, the cluster data, the template data and the permission data as the metadata.

[0059] The gateway is configured to read target data from the memory of the second configuration center every preset time length, and backup the target data to the disk and the memory of the gateway; read backup data from the disk when the gateway starts; and when it is detected that the data reading time is reached and the metadata version of the gateway is changed, read the latest target data from the memory of the second configuration center, and cache the read latest target data in the disk and the memory of the gateway.

[0060] The data processing system provided in the present application adopts the data processing method in the above embodiments, and can solve the availability problem of the gateway caused by the instability of the management side function and the physical environment. Compared with the prior art, the data processing system provided in the present application has the same beneficial effects as the data processing method provided in the above embodiments, and other technical features in the data processing system are the same as the features disclosed in the above embodiments, which will not be repeated here.

[0061] Based on this, the embodiments of the present application provide a data processing method, which refers to Figure 2 , Figure 2 is a flowchart of an embodiment of the data processing method of the present application. In the present embodiment, the data processing method is applied to a second configuration center, and includes steps S10-S30:

[0062] In step S10, the metadata in the database of the first configuration center is read every preset time length, and after conversion and processing, the metadata is stored in the database of the second configuration center. The first configuration center is used to manage the metadata in response to a metadata management request of the gateway.

[0063] The metadata stored in the database of the first configuration center includes instance data, cluster data, template data, and permission data, and the metadata can be updated according to actual conditions. The first configuration center can manage the metadata in response to a metadata management request of the gateway. Specifically, the first configuration center determines at least one data management mode in response to a metadata management request of the gateway, processes the metadata based on the data management mode, and stores the processed metadata in the database of the first configuration center, so that the second configuration center reads the metadata from the database of the first configuration center every preset time length, converts and processes the metadata, and stores the metadata in the database of the second configuration center. The metadata includes at least one of instances, clusters, templates, and permissions, and the corresponding data management mode includes at least one of instance increase, cluster increase, template increase, and permission modification.

[0064] The preset time length can be set according to actual conditions, for example, the response speed of the gateway, the response speed of the second configuration center, and the response speed of the first configuration center. The preset time length can be set to 1 minute, for example, the second configuration center reads the metadata from the database of the first configuration center every 1 minute, thereby realizing synchronization of the metadata. The second configuration center can also read the metadata from the database of the first configuration center as needed.

[0065] The database of the first configuration center can store different versions of metadata, or can store the latest version of metadata. The second configuration center can read the latest version of metadata from the database of the first configuration center every preset time length, or read the required version of metadata.

[0066] By reading the metadata from the database of the first configuration center and storing it in the database of the second configuration center, local caching of the metadata on the second configuration center side can be realized. When the second configuration center receives a metadata query request of the gateway, the corresponding metadata can be directly queried from the second configuration center, without relying on the first configuration center. By decoupling the second configuration center from the first configuration center, the maintenance cost and data processing pressure are reduced.

[0067] In step S20, when the version of the metadata currently cached in the memory of the second configuration center is detected to change, the metadata of the target version is read from the database of the second configuration center, and the metadata of the target version is cached in the memory of the second configuration center.

[0068] The target version can be the latest version or any version required to be queried. For example, taking the latest version as the target version, the second configuration center can detect whether the currently cached metadata version in the memory thereof changes every preset time length. When detecting that the metadata version changes, the second configuration center reads the metadata of the latest version from the database thereof again and caches the metadata of the latest version in the memory of the second configuration center, so that the gateway can obtain the metadata of the latest version subsequently. By caching the metadata of the latest version in the memory of the second configuration center, the second configuration center can directly process and return based on the metadata in the memory subsequently, without querying the local database, thereby improving the processing efficiency of the metadata.

[0069] In the related art, the gateway pulls data from the configuration center through a timing task. Due to the uncertainty of the request sequence, network condition and processing sequence, the data in each gateway node can be inconsistent, which affects the accuracy of the data and the reliability of the service. By the way of the second configuration center organizing version data regularly, the probability of the inconsistency of the metadata between the gateway nodes can be reduced, and the pressure on the database is reduced.

[0070] In step S30, the target data matched with the metadata loading condition is read from the memory of the second configuration center in response to the metadata query request of the gateway, and the target data is returned.

[0071] In the second configuration center, the metadata loading condition corresponding to each gateway and the metadata corresponding to each metadata loading condition can be set in advance, that is, the metadata loading condition can be set in the second configuration center. In this case, the second configuration center responds to the metadata query request of the gateway, determines which gateway node sends the request according to the metadata query request, queries the metadata loading condition corresponding to the gateway node, obtains the target data associated with the metadata loading condition, and finally returns the obtained target data to the corresponding gateway node. The target data is processed and stored in the second configuration center in advance according to the metadata loading condition. The second configuration center provides a corresponding query interface, and the gateway can directly call the query interface to read the required target data from the memory of the second configuration center without any additional conversion. The gateway can read the target data from the memory of the second configuration center every preset time length and backup the target data to the disk of the gateway. In response to the start instruction of the gateway, the backup data is read from the disk. When it is detected that the data reading time is reached and the metadata version of the gateway changes, the latest target data is read from the database of the second configuration center and cached in the memory of the gateway.

[0072] The metadata loading condition can also be set and sent by the gateway. The gateway can set the metadata loading condition required by itself as needed and send it to the second configuration center in the form of a metadata query request.

[0073] In this embodiment, the metadata can be read from the database of the first configuration center every preset time length and cached in the database of the second configuration center. When the version of the currently cached metadata in the memory of the second configuration center is detected to change, the metadata of the target version is read from the database of the second configuration center and cached in the memory of the second configuration center. In response to the metadata query request of the gateway, the target data matching the metadata loading condition is read from the memory of the second configuration center and returned. Since two configuration centers are set, one is used for managing metadata and the other is used for querying metadata, the processing of metadata can be preprocessed through the cooperation of the two configuration centers. Subsequently, the appropriate target data can be matched according to the metadata loading condition of the gateway. In this way, the gateway does not need to pull all the metadata required by the function from the configuration center for processing, thereby solving the availability problem caused by the instability of the management side function and the physical environment of the gateway, reducing the metadata processing pressure of the gateway, and improving the availability of the gateway.

[0074] In a possible implementation, the step S30 can include:

[0075] In step S31, in response to the metadata query request of the plurality of gateways, the gateways are grouped. The metadata loading conditions of the gateways in the same group are the same, and each group corresponds to a different metadata loading condition.

[0076] In step S33, the target data matching the metadata loading condition is read from the memory of the second configuration center.

[0077] In step S33, the target data matching the metadata loading condition is returned to all gateways in the group corresponding to the metadata loading condition.

[0078] In the related art, each gateway performs a timing task polling to pull the metadata required by each function from the configuration center. In this application, two configuration centers are set, one is used for managing metadata and the other is used for querying metadata. The processing of metadata can be preprocessed through the cooperation of the two configuration centers. Subsequently, the appropriate target data can be matched according to the metadata loading condition of the gateway. In this way, the gateway does not need to pull all the metadata required by the function from the configuration center for processing, thereby reducing the metadata processing pressure of the gateway and improving the availability of the gateway.

[0079] For the application scenario of multiple gateways, each gateway needs to perform the above operation, that is, to query the corresponding metadata from the second configuration center according to the metadata loading condition of each gateway. When the number of gateways is large, this query method will also cause a large processing pressure on the second configuration center and each gateway. Therefore, on the basis of the above technical solution, the present application groups each gateway in advance, for example, each gateway can be grouped in advance according to the protection level of each gateway, so that the protection levels of each gateway in the same group are the same. In this case, the gateways in the same group usually have the same function, or each gateway in the same group is provided with a corresponding metadata loading condition, and each group corresponds to different metadata loading conditions. In this way, the gateways in the same group can synchronously obtain the required metadata based on the same metadata loading condition, which improves the metadata acquisition efficiency of the gateway compared with the related art.

[0080] The memory of the second configuration center stores the metadata preprocessed by the metadata, and the data structure of the preprocessed metadata is adapted to the data structure of the gateway, which can be directly read and used by the gateway. In this way, the gateway does not need to preprocess the metadata, which reduces the data processing pressure of the gateway.

[0081] The metadata loading condition can be set according to actual conditions, and the group to which each gateway belongs can also be set in the second configuration center. The second configuration center can set the metadata loading condition corresponding to each group and the metadata corresponding to the metadata loading condition. For example, the metadata loading condition can be to obtain instance data, cluster data, template data and permission data in the past 1 hour, or to obtain certain cluster data, which can be set according to the actual needs of the user. In actual application process, when the second configuration center receives the metadata query request of multiple gateways, the second configuration center can determine which gateway node and the group to which the gateway node belongs, match the appropriate metadata based on the metadata loading condition corresponding to the group, and synchronously distribute to all gateways in the same group.

[0082] For example, referring to Figure 3 For the multi-gateway application scenario, each gateway can be divided into different groups according to the function, and the second configuration center can be pre-set with the metadata corresponding to each group, including instance data, cluster data, template data and permission data. The metadata can be synchronously sent to all gateways in the same group, which improves the data query efficiency of each gateway.

[0083] In this embodiment, by grouping each gateway, the metadata loading conditions of each gateway in the same group are the same, so that each gateway in the same group can load corresponding metadata based on the same metadata loading condition, realizing the query efficiency of metadata in the multi-gateway application scenario.

[0084] In an implementable manner, step S10 can include:

[0085] Step S11, reading metadata from the database of the first configuration center every preset time length;

[0086] In an implementable manner, instance data, cluster data, template data and permission data are read from the database of the first configuration center every preset time length; the instance data, the cluster data, the template data and the permission data are taken as the metadata. Wherein, the instance is a specific running individual in the configuration, which is the smallest operational unit of the system, represents a single service unit actually running, such as a gateway node process, undertakes specific task execution or data storage function. The cluster is a logical collection of multiple instances, which collectively provides high availability or distributed services. The template is used to standardize the structure of the configuration, defines the deployment specification of the instance or the cluster, and ensures the consistency of the configuration. The permission can be the access permission of each gateway.

[0087] Step S12, preprocessing the metadata;

[0088] The data structure of the metadata read from the first configuration center is different from the data structure that the gateway can recognize, or there is redundant or invalid data in the metadata read from the first configuration center. For these situations, the second configuration center needs to preprocess the metadata read from the database of the first configuration center, so that the preprocessed metadata can be directly read and processed by the gateway.

[0089] The preprocessing of the metadata can include: converting the data structure of the metadata according to the target data structure that the gateway can recognize and process, repeating data filtering and deleting the metadata, and data merging the metadata. The metadata can also be preprocessed according to the metadata loading condition corresponding to each gateway group, so as to adapt to the metadata demand of each gateway group. By preprocessing the metadata through the second configuration center, the metadata processing operation is front-end, so as to reduce the data processing pressure of the gateway.

[0090] In an implementable manner, the metadata can be read from the database of the first configuration center every preset time length; the metadata is format-converted, merged and combined to obtain preprocessed metadata; a corresponding version identifier is set for the preprocessed metadata based on the content of the preprocessed metadata; and the preprocessed metadata and the corresponding version identifier of the preprocessed metadata are stored in the database of the second configuration center.

[0091] In another implementable manner, the instance data, the cluster data, the template data and the permission data can be read from the database of the first configuration center every preset time length; the instance data, the cluster data, the template data and the permission data are taken as metadata. The metadata is format-converted, merged and combined to obtain preprocessed metadata; a corresponding version identifier is set for the preprocessed metadata based on the content of the preprocessed metadata; and the preprocessed metadata and the corresponding version identifier of the preprocessed metadata are stored in the database of the second configuration center. In this way, the second configuration center can determine whether the metadata currently stored in the memory is the latest metadata based on the version identifier, and the gateway can query the latest metadata, thereby improving the accuracy of the query result of the gateway.

[0092] In step S13, the preprocessed metadata is stored in the database of the second configuration center.

[0093] The preprocessed metadata can be classified and stored in the database of the second configuration center based on a metadata loading condition, thereby facilitating subsequent reading of the metadata by the gateway.

[0094] In this embodiment, the metadata is read from the database of the first configuration center every preset time length, and the metadata is preprocessed, and the preprocessed metadata is stored in the database of the second configuration center. The metadata is preprocessed by the second configuration center, and the metadata processing operation is front-loaded, thereby reducing the data processing pressure of the gateway.

[0095] Based on the same inventive concept, the embodiments of the present application provide a data processing method, which is described with reference to Figure 4 , Figure 4 FIG. 2 is a flowchart of a data processing method according to a second embodiment of the present application. In this embodiment, the data processing method is applied to a first configuration center, and includes steps S110-S130.

[0096] In step S110, at least one data management mode is determined in response to a metadata management request of the gateway; wherein the metadata comprises at least one of instance, cluster, template, and permission. The instance is a specific running individual in the configuration, is the smallest operational unit of the system, represents an actual running single service unit, for example, a gateway node process, and undertakes specific task execution or data storage functions. The cluster is a logical collection of multiple instances, and collectively provides high availability or distributed services. The template is used to standardize the structure of the configuration, defines the deployment specification of the instance or cluster, and ensures configuration consistency. The permission can be the access permission of each gateway.

[0097] In step S120, the metadata is processed based on the data management mode.

[0098] The corresponding data management mode comprises at least one of instance increase, instance decrease, cluster increase, cluster decrease, template increase, template decrease, and permission modification. Specifically, the instance can be increased in response to an instance increase request of the gateway, and the instance can be decreased in response to an instance decrease request of the gateway. The cluster can be increased in response to a cluster increase request of the gateway. The cluster can be decreased in response to a cluster decrease request of the gateway. The template can be increased in response to a template increase request. The template can be decreased in response to a template decrease request. The permission of the gateway can be modified in response to a permission modification request of the gateway, so as to modify the access permission of the gateway to the metadata.

[0099] In step S130, the processed metadata is stored in the database of the first configuration center, so that the second configuration center reads the metadata from the database of the first configuration center at a preset time interval, and stores the converted and processed metadata in the database of the second configuration center. When the second configuration center detects that the version of the currently cached metadata in the memory of the second configuration center changes, the second configuration center reads the metadata of the target version from the database of the second configuration center, and caches the metadata of the target version in the memory of the second configuration center. In response to a metadata query request of the gateway, the second configuration center reads target data matching the metadata loading condition from the memory of the second configuration center, and returns the target data.

[0100] After processing the metadata, the processed metadata can be stored in a database of the first configuration center. Subsequently, the second configuration center can read the metadata from the database of the first configuration center at a preset time interval, perform conversion and processing, and store the metadata in a database of the second configuration center. The first configuration center is configured to manage metadata in response to a metadata management request of a gateway. When a change in a current cached metadata version in a memory of the second configuration center is detected, metadata of a target version is read from the database of the second configuration center and cached in the memory of the second configuration center. In response to a metadata query request of the gateway, target data matching a metadata loading condition is read from the memory of the second configuration center and returned.

[0101] Specifically, the second configuration center can read metadata from the database of the first configuration center at a preset time interval. The metadata is preprocessed, and the preprocessed metadata is stored in the database of the second configuration center. The preprocessing of the metadata includes format conversion, merging, and combination of the metadata to obtain preprocessed metadata. Based on the content of the preprocessed metadata, a corresponding version identifier is set for the preprocessed metadata. The preprocessed metadata and the corresponding version identifier are stored in the database of the second configuration center. The second configuration center can also group multiple gateways in response to metadata query requests of the gateways. The metadata loading conditions of the gateways in the same group are the same, and each group corresponds to a different metadata loading condition. The second configuration center reads target data matching each metadata loading condition from the memory, and returns the target data to all gateways in the group corresponding to the metadata loading condition.

[0102] In this embodiment, the first configuration center processes management requests and manages metadata. Through transaction processing, the consistency and reliability of all operations are ensured.

[0103] Based on the same inventive concept, the data processing method provided in the embodiments of the present application is described with reference to Figure 5 , Figure 5 The flowchart of the third embodiment of the data processing method of the present application is shown in the figure. In this embodiment, the data processing method is applied to a gateway, and includes steps S210-S230.

[0104] In step S210, target data is read from the memory of the second configuration center at a preset time interval, and the target data is backed up to the disk and memory of the gateway.

[0105] The preset time length can be set according to actual conditions, for example, can be determined according to the response speed of the gateway, the response speed of the second configuration center, the response speed of the first configuration center, etc. The preset time length can be set to 1 minute, for example, the gateway reads the target data from the memory of the second configuration center every 1 minute, and backs up the target data to the disk and memory of the gateway.

[0106] The target data can be the latest version of the metadata, or any required version of the metadata.

[0107] The second configuration center reads the metadata from the database of the first configuration center every preset time length for conversion and processing, and stores it in the database of the second configuration center. The first configuration center is used to respond to the metadata management request of the gateway for metadata management. When the version of the currently cached metadata in the memory of the second configuration center is detected to change, the target version of the metadata is read from the database of the second configuration center, and the target version of the metadata is cached in the memory of the second configuration center.

[0108] Step S220, reading backup data from the disk in response to the start instruction of the gateway;

[0109] In the related art, when the gateway needs to be restarted on a large scale due to underlying faults or version release, etc., the gateway pulls the metadata from the configuration center again. At this time, the configuration center may reach a performance bottleneck, causing the response time to be prolonged, affecting the recovery speed of the service. Or in the extreme case, if the network connection between the gateway and the configuration center is interrupted, and service restart occurs during this period, the gateway will not be able to obtain metadata from the configuration center, so it cannot provide normal service, causing interruption. In view of these problems, the target data is read from the memory of the second configuration center every preset time length, and the target data is backed up to the disk of the gateway. By storing the target data in the disk, the backup of the target data can be realized. When the gateway fails to restart, the old version of the target data can be obtained from the disk for processing, so that the user terminal experience is not good and the user terminal experience is not good. Complaints caused by the gateway being unable to obtain data.

[0110] Step S230, detecting that the data reading time is reached and the metadata version of the gateway is changed, reading the latest target data from the memory of the second configuration center, and caching the read latest target data in the disk and memory of the gateway.

[0111] The data reading time can be set according to actual conditions. The data reading time can indicate that the gateway is in a stable state.

[0112] The gateway may exist metadata version update during the restart time after the failure. The gateway can start timing after starting, read the latest target data from the memory of the second configuration center when reaching the data reading time, and cache the read latest target data in the memory and the disk of the gateway, so that the gateway can directly process data based on its own memory, thereby improving the data processing efficiency of the gateway.

[0113] The gateway can also start timing after starting, read the latest target data from the memory of the second configuration center when reaching the data reading time and detecting that the currently cached metadata version in the memory changes, and cache the read latest target data in the memory and the disk of the gateway.

[0114] The gateway can also start timing after starting, read the latest target data from the memory of the second configuration center when reaching the data reading time and detecting that the currently cached metadata version in the memory changes, and cache the read latest target data in the memory and the disk of the gateway.

[0115] In the embodiment, by setting the disk and the memory to store the metadata, the interruption service problem caused by network interruption and service restart of the gateway can be solved, and the configuration center bottleneck caused by batch restart of the gateway can be alleviated.

[0116] Based on the same inventive concept, the application provides a second configuration center, comprising: at least one processor; and a memory in communication connection with the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the data processing method in the above embodiment.

[0117] The second configuration center provided by the application adopts the data processing method in the above embodiment, and can solve the availability problem of the gateway caused by the instability of the management side function and the physical environment. Compared with the prior art, the data processing device provided by the application has the same beneficial effects as the data processing method provided by the above embodiment, and other technical features in the data processing device are the same as the features disclosed in the above embodiment method, which will not be repeated here.

[0118] Based on the same inventive concept, the application provides a first configuration center, comprising: at least one processor; and a memory in communication connection with the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the data processing method in the above embodiment.

[0119] The first configuration center provided in the application adopts the data processing method in the above embodiment, and can solve the availability problem caused by the instability of the management side function and the physical environment of the gateway. Compared with the prior art, the data processing device provided in the application has the same beneficial effects as the data processing method provided in the above embodiment, and other technical features in the data processing device are the same as the features disclosed in the above embodiment method, which will not be repeated here.

[0120] Based on the same inventive concept, the application provides a gateway, comprising: at least one processor; and a memory in communication connection with the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the data processing method in the above embodiment.

[0121] The gateway provided in the application adopts the data processing method in the above embodiment, and can solve the availability problem caused by the instability of the management side function and the physical environment of the gateway. Compared with the prior art, the data processing device provided in the application has the same beneficial effects as the data processing method provided in the above embodiment, and other technical features in the data processing device are the same as the features disclosed in the above embodiment method, which will not be repeated here.

[0122] It should be understood that various parts of the application can be realized by hardware, software, firmware or a combination thereof. In the description of the above embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0123] The above is merely specific implementation of the application, but the protection scope of the application is not limited thereto, and any person skilled in the art can easily think of changes or replacements within the technical range disclosed in the application, which should be covered in the protection scope of the application. Therefore, the protection scope of the application should be subject to the protection scope of the claims.

[0124] The above is merely specific implementation of the application, but the protection scope of the application is not limited thereto, and any person skilled in the art can easily think of changes or replacements within the technical range disclosed in the application, which should be covered in the protection scope of the application. Therefore, the protection scope of the application should be subject to the protection scope of the claims. The above is merely specific implementation of the application, but the protection scope of the application is not limited thereto, and any person skilled in the art can easily think of changes or replacements within the technical range disclosed in the application, which should be covered in the protection scope of the application. Therefore, the protection scope of the application should be subject to the protection scope of the claims.

Claims

1. A data processing method, characterized in that, Applied to the second configuration center, including: Metadata is read from the database of the first configuration center at preset intervals, transformed and processed, and then stored in the database of the second configuration center. The first configuration center is used to manage metadata in response to the gateway's metadata management requests. When a change in the currently cached metadata version in the memory of the second configuration center is detected, the metadata of the target version is read from the database of the second configuration center, and the metadata of the target version is cached in the memory of the second configuration center; In response to the gateway's metadata query request, target data matching the metadata loading conditions is read from the memory of the second configuration center and the target data is returned.

2. The data processing method as described in claim 1, characterized in that, The step of responding to the gateway's metadata query request by reading target data matching the metadata loading conditions from the memory of the second configuration center and returning the target data includes: In response to metadata query requests from multiple gateways, the gateways are grouped, wherein the metadata loading conditions of each gateway within the same group are the same, and each group corresponds to different metadata loading conditions. Read the target data that matches each of the metadata loading conditions from the memory of the second configuration center; The target data that matches the metadata loading conditions is returned to all gateways within the group corresponding to the metadata loading conditions.

3. The data processing method as described in claim 1, characterized in that, The step of reading metadata from the database of the first configuration center at preset intervals, transforming and processing it, and then storing it in the database of the second configuration center includes: Metadata is read from the database of the first configuration center at preset intervals; The metadata is format-converted, merged, and combined to obtain preprocessed metadata. Based on the content of the preprocessed metadata, set a corresponding version identifier for the preprocessed metadata; The preprocessed metadata and the version identifier corresponding to the preprocessed metadata are stored in the database of the second configuration center.

4. The data processing method as described in claim 3, characterized in that, The step of reading metadata from the database of the first configuration center at preset intervals includes: Instance data, cluster data, template data, and permission data are read from the database of the first configuration center at preset intervals. The instance data, the cluster data, the template data, and the permission data are used as the metadata.

5. A data processing method, characterized in that, Applied to the first configuration center, including: In response to a metadata management request from the gateway, determine at least one data management method; Metadata is processed based on the aforementioned data management method; The processed metadata is stored in the database of the first configuration center, so that the second configuration center reads the metadata from the database of the first configuration center at preset intervals, transforms and processes it, and then stores it in the database of the second configuration center; when the second configuration center detects a change in the version of the currently cached metadata in the memory of the second configuration center, it reads the metadata of the target version from the database of the second configuration center and caches the metadata of the target version in the memory of the second configuration center; in response to the metadata query request of the gateway, the second configuration center reads the target data that matches the metadata loading conditions from the memory of the second configuration center and returns the target data.

6. The data processing method as described in claim 5, characterized in that, The metadata includes at least one of instance, cluster, template, and permission, and the corresponding data management method includes at least one of instance addition / removal, cluster addition / removal, template addition / removal, and permission modification.

7. A data processing method, characterized in that, Applied to gateways, including: The target data is read from the memory of the second configuration center at preset intervals, and the target data is backed up to the disk and memory of the gateway. In response to the gateway's startup command, backup data is read from the disk; When the data reading time is detected and the metadata version of the gateway changes, the latest target data is read from the memory of the second configuration center, and the latest target data is cached in the disk and memory of the gateway.

8. A data management system, characterized in that, The data management system includes a first configuration center, a second configuration center, and a gateway, wherein, The first configuration center is used to respond to the gateway's metadata management request, determine at least one data management method; process metadata based on the data management method; and store the processed metadata in the database of the first configuration center. The second configuration center is used to read metadata from the database of the first configuration center at preset intervals, transform and process it, and then store it in the database of the second configuration center. The first configuration center is used to manage metadata in response to the gateway's metadata management requests; when a change in the currently cached metadata version in the memory of the second configuration center is detected, it reads the target version of the metadata from the database of the second configuration center and caches the target version of the metadata in the memory of the second configuration center; in response to the gateway's metadata query requests, it reads target data matching the metadata loading conditions from the memory of the second configuration center and returns the target data. The gateway is used to read target data from the memory of the second configuration center at preset intervals and back up the target data to the gateway's disk and memory; when the gateway starts, it reads the backup data from the disk; when it detects that the data reading time has been reached and the metadata version of the gateway has changed, it reads the latest target data from the memory of the second configuration center and caches the latest target data in the gateway's disk and memory.

9. A second configuration center, characterized in that, The second configuration center includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the data processing method as described in any one of claims 1 to 4.

10. A first configuration center, characterized in that, The first configuration center includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the data processing method as described in claim 5 or 6.

11. A gateway, characterized in that, The gateway includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the data processing method as described in claim 7.

Images