Data management and query method, apparatus, service mesh system, computing device

Abstract

A data management and query method, device, service grid system and computing device. The data management method comprises: adding a sidecar module for each data source, the sidecar module comprising at least a registration submodule, the registration submodule comprising identification information of the data source (S102); analyzing the registration module to obtain the identification information of the data source (S104); and registering the identification information of the data source in a service registry center (S106).

Description

Technical Field

[0001] This disclosure generally relates to the field of database technology, and more specifically, to data management and query methods, apparatus, service grid systems, and computing devices. Background Technology

[0002] Data as a Service (DaaS) facilitates the unified management and application of enterprise data assets, significantly improving efficiency. Enterprise data assets are available in various data storage formats to meet diverse needs. On the other hand, service mesh is a popular technology applied in microservice scenarios, and its sidecar model helps in service discovery and registration.

[0003] However, there is currently no technical solution that treats data as a means for services to be automatically registered and discovered. Summary of the Invention

[0004] A brief overview of the invention is given below to provide a basic understanding of certain aspects of it. It should be understood that this overview is not an exhaustive summary of the invention. It is not intended to identify key or essential parts of the invention, nor is it intended to limit the scope of the invention. Its purpose is merely to present certain concepts in a simplified form as a prelude to the more detailed description that follows.

[0005] In view of this, the present invention proposes a technical solution that applies service mesh technology to data and implements DaaS by building a unified data specification.

[0006] According to one aspect of this disclosure, a data management method is provided, comprising: adding a sidecar module for each data source, the sidecar module including at least a registration submodule, the registration submodule including identification information of the data source; parsing the registration module to obtain the identification information of the data source; and registering the identification information of the data source in a service registry.

[0007] By adding a sidecar structure to the data source, automatic registration of the data source was achieved.

[0008] Optionally, in one example of the above aspects, the sidecar module further includes: a discovery submodule, a flow control submodule, and a routing submodule, wherein the discovery submodule is used to realize mutual discovery between different data sources; the flow control submodule is used to control the flow of the data source; and the routing submodule is used to realize data routing of the data source.

[0009] In this way, automatic data source discovery, as well as traffic control and routing functions are further realized.

[0010] According to another aspect of this disclosure, a data management apparatus is provided, comprising: a sidecar addition unit configured to add a sidecar module for each data source, the sidecar module including at least a registration submodule, the registration submodule including identification information of the data source; a parsing unit configured to parse the registration submodule to determine the identification information of the data source; and a registration unit configured to register the identification information of the data source in a service registration center.

[0011] According to another aspect of this disclosure, a data query method is provided, comprising: receiving a user's query request for a data source; a driver browsing registered data sources in a service registry; if the requested data source is not found, returning a prompt message to the user; and if the requested data source is found in the service registry, the driver establishing a connection with the data source using information in a sidecar module.

[0012] Optionally, in one example of the above aspects, the method further includes: mapping the queried data to a standard format based on a pre-defined standard interface specification, and returning the data in the standard format to the user.

[0013] In this way, data sources of different formats can be converted into a unified standard format and returned to the user. This way, the user does not need to care about the underlying data storage method and can get data in a unified standard format.

[0014] According to another aspect of this disclosure, a service mesh system is provided, including a data unit and a control unit. The data unit includes multiple data sources, each data source having a sidecar module. The control unit includes a service registry and a data source parser, the data source parser being used to parse the sidecar modules and register the identifier information of the parsed data sources in the service registry.

[0015] Optionally, in one example of the above aspects, the system further includes: a driving unit, the driving unit including a driver and a service discovery module for the plurality of data sources respectively, the service discovery module being used to browse the service registry to determine whether the data source is registered in the service registry when the driving unit receives a user's query request for the data source.

[0016] According to another aspect of this disclosure, a computing device is provided, comprising: at least one processor; and a memory coupled to said at least one processor, said memory for storing instructions that, when executed by said at least one processor, cause the processor to perform the method as described above.

[0017] According to another aspect of this disclosure, a non-transitory machine-readable storage medium is provided that stores executable instructions, which, when executed, cause the machine to perform the method described above.

[0018] According to another aspect of this disclosure, a computer program is provided, including computer-executable instructions that, when executed, cause at least one processor to perform the method described above.

[0019] According to another aspect of this disclosure, a computer program product is provided, which is tangibly stored on a computer-readable medium and includes computer-executable instructions that, when executed, cause at least one processor to perform the method as described above.

[0020] According to the heterogeneous data management method, data query method, and service mesh system of the present invention, automatic registration and discovery of data sources, as well as traffic control and routing functions, are achieved by adding a sidecar structure to the data source. Furthermore, by defining standard data format specifications and APIs, raw data can be mapped to a standard format, thereby providing standard data query results. Attached Figure Description

[0021] The above and other objects, features, and advantages of the present invention will be more readily understood by referring to the following description of embodiments of the invention in conjunction with the accompanying drawings. The components in the drawings are merely for illustrating the principles of the invention. In the drawings, the same or similar technical features or components will be represented by the same or similar reference numerals. In the drawings:

[0022] Figure 1 A flowchart illustrating an exemplary process of a data management method according to a first aspect of an embodiment of the present invention;

[0023] Figure 2 A block diagram illustrating an exemplary configuration of a data management apparatus according to an embodiment of the present invention;

[0024] Figure 3 This is a flowchart illustrating an exemplary process of a data query method according to a second aspect of an embodiment of the present invention;

[0025] Figure 4 An exemplary architecture diagram of a service mesh system according to an embodiment of the present invention is shown;

[0026] Figure 5 A block diagram of a computing device for implementing data management and data querying according to embodiments of the present disclosure is shown.

[0027] The reference numerals in the attached figures are as follows:

[0028] Detailed Implementation

[0029] The subject matter described herein will now be discussed with reference to exemplary embodiments. It should be understood that these embodiments are discussed merely to enable those skilled in the art to better understand and implement the subject matter described herein, and are not intended to limit the scope, applicability, or examples set forth in the claims. The function and arrangement of the elements discussed may be changed without departing from the scope of this disclosure. Various processes or components may be omitted, substituted, or added as needed in the various examples. For example, the described methods may be performed in a different order than described, and steps may be added, omitted, or combined. Furthermore, features described in some examples may be combined in other examples.

[0030] As used herein, the term "comprising" and its variations are open terms meaning "including but not limited to". The term "based on" means "at least partially based on". The terms "one embodiment" and "an embodiment" mean "at least one embodiment". The term "another embodiment" means "at least one other embodiment". The terms "first", "second", etc., may refer to different or the same objects. Other definitions, whether explicit or implicit, may be included below. Unless explicitly indicated by the context, the definition of a term shall remain consistent throughout the specification.

[0031] The following is a brief explanation of several terms mentioned in this manual.

[0032] DaaS: Data is provided as a service. When data management is more centralized, upper-layer data consumers do not need to worry about issues such as acquiring underlying data, and can focus entirely on using the data.

[0033] Heterogeneous data sources: Different types of databases, such as JDBC and RESTful databases, have different forms of data storage and data connection configurations.

[0034] Service Mesh: Service mesh has become a popular technology in recent years for microservices and other scenarios. Its core lies in achieving rapid service discovery and registration by decoupling the data platform and control platform. The sidecar architecture provides a proxy for service discovery, load balancing, and other functions.

[0035] This invention applies service mesh technology to data and implements DaaS by building a unified data specification.

[0036] The data management method and apparatus according to embodiments of the present disclosure will now be described with reference to the accompanying drawings.

[0037] Figure 1 This is a flowchart illustrating an exemplary process of a data management method 100 according to a first aspect of an embodiment of the present invention.

[0038] First, in step S102, a sidecar module is added for each data source. The sidecar module includes at least a registration submodule, which includes the identification information of the data source.

[0039] According to embodiments of the present invention, a data management method, combined with service mesh technology, allows multiple different data sources to form a service mesh, aiming to enable data sources to be discovered and registered as individual services. These data sources can be heterogeneous, for example, stored in different types of databases such as JDBC and RESTful, or stored on websites. Therefore, the data management method according to the present invention can manage heterogeneous data. Specifically, a sidecar module is added for each data source, which can implement the service registration function of the data source. The sidecar module includes a registration submodule, which can extract identification information from the data source. Identification information refers to information that can be used to uniquely identify a specific data source, such as an IP address and / or port address. The present invention does not limit the specific type of identification information for the data source.

[0040] In one example, the sidecar module may further include a discovery submodule, a flow control submodule, and a routing submodule. The discovery submodule is used to enable mutual discovery between different data sources; the flow control submodule is used to control the flow of traffic to the data sources; and the routing submodule is used to implement data routing for the data sources.

[0041] Next, in step S104, the registration submodule is parsed to obtain the identification information of the data source.

[0042] By parsing the registration submodule of the sidecar module, the identification information of the data source can be determined.

[0043] Next, in step S106, the identification information of the data source is registered in the service registration center.

[0044] Based on the Service Mesh concept, the data source can be referred to as the data plane, while the service registry center, which handles the parsing of registration submodules and the registration of data sources, resides in the control plane. In this invention, the data plane and control plane can reside on the same server or on different servers within the same cluster.

[0045] In the data management method according to this embodiment, a sidecar structure is added for each data source, decoupling the data plane and the control plane to enable rapid service discovery and registration. The sidecar structure can provide the control plane with functions such as service registration, service discovery, routing, and traffic control. The sidecar structure decouples these services from the storage of the data source, making these services independent of the logic of the data source itself.

[0046] Corresponding to the first aspect, embodiments of the present invention also provide a data management device. Figure 2 This is a block diagram illustrating an exemplary configuration of a data management device 200 according to an embodiment of the present invention.

[0047] like Figure 2 As shown, the data management device 200 includes a sidecar addition unit 202, a parsing unit 204, and a registration unit 206.

[0048] The sidecar addition unit 202 is configured to add a sidecar module for each data source. The sidecar module includes at least a registration submodule, which includes the identification information of the data source.

[0049] The parsing unit 204 is configured to parse the registration submodule to obtain the identification information of the data source.

[0050] Registration unit 206 is configured to register the identification information of the data source in the service registry.

[0051] In embodiments of the present invention, details of the operation and function of each part of the data management device 200 can be found in reference to [reference needed]. Figure 1 The relevant parts of the embodiments of the data management method 100 of the present invention described herein are the same or similar, and will not be described in detail here.

[0052] Figure 3 This is a flowchart illustrating an exemplary process of a data query method 300 according to a second aspect of an embodiment of the present invention.

[0053] The data query method according to this embodiment is for querying data according to a reference. Figure 1 The data management method involves adding a sidecar module to the data source and querying the database registered in the service registry.

[0054] First, in step S302, a user's query request for the data source is received.

[0055] Next, in step S304, the driver browses the registered data sources in the service registry.

[0056] In step S306, if the requested data source is not found, it means that the data source is not registered in the service registry, and a prompt message is returned to the user; if the requested data source is found in the service registry, the driver establishes a connection with the data source using the information in the sidecar module.

[0057] The registration submodule in the sidecar module includes the identification information of the data source, so the driver can directly connect to the data source based on this information.

[0058] In one example, the data query method 300 may further include step S308: mapping the queried data to a standard format based on a predefined standard interface specification, and returning the data in the standard format to the user.

[0059] It is understood that different types of databases have different data storage formats. For example, in a RESTful database, data is stored in JSON format, not in a table format. In the data query method according to embodiments of the present invention, a standard data storage format can be defined, such as specifying a set of standard specifications, in which metadata information related to the data source is defined. For example, for JDBC data sources, usernames, passwords, and other information can be defined; for REST data sources, URL proxies, schema mapping information, etc., can be defined. Then, data sources of different formats are converted into a unified standard format and returned to the user. This way, the user does not need to worry about the underlying data storage method to obtain data in a unified standard format. In this invention, the user can define the standard specifications for the data as needed, without limiting the specific format of the data.

[0060] Specifically, the driver can determine the mapping rules between the data source's data format and the standard data format. This can be achieved by adding an API interface to either the sidecar module or the data source itself. The converted data is then returned to the driver, which in turn provides the standard-formatted data to the user via the gateway.

[0061] Figure 4 An exemplary architecture diagram of a service mesh system 400 according to an embodiment of the present invention is shown, including: a data unit 401 and a control unit 402.

[0062] Data unit 401 includes multiple data sources 4011, each data source 4011 having a sidecar module 4012. Data sources 4011 can be heterogeneous data stored on different types of databases or different websites.

[0063] The control unit 402 includes a service registration center 4021 and a data source parser 4022. The data source parser 4022 is used to parse the sidecar module 4012 of the data source 4011 and register the identification information of the data source obtained by parsing in the service registration center 4021.

[0064] In one example, the service mesh system 400 further includes a driver unit 403, which includes a driver 4031 for different data sources and a service discovery module 4032. The service discovery module 4032 is used to browse the service registry 4021 to determine whether the data source 4011 is registered in the service registry 4021 when the driver unit 403 receives a query request from a user for the data source 4011.

[0065] Data unit 401, control unit 402 and drive unit 403 can be located on the same server or on different servers in the same cluster.

[0066] The following reference Figure 4 This describes an exemplary process by which a user performs a data query in a service mesh system according to an embodiment of the present invention.

[0067] Figure 4 In addition to the data unit 401, control unit 402, and drive unit 403, the system also includes a gateway 404, a data user 405, and an API 406 (application programming interface) between the user and the gateway. These components are provided to help illustrate the specific process of data querying and are not essential components of the service mesh system according to embodiments of the present invention.

[0068] In the service mesh system 400, multiple data sources 4011 have already followed the above references. Figure 1 The data management method described herein has been registered in the service registry 4021.

[0069] First, when user 405 initiates a query request for data source 4011 to driver 403 through gateway 404, service discovery module 4032 of driver 403 browses the identification information of registered data sources in service registry 4021; if the requested data source is not found, a prompt message (e.g., "error") is returned to the user; if the requested data source is found in service registry 4021, driver 403 establishes a connection with data source 4011 using the information in sidecar module 4012.

[0070] After establishing a connection with data source 4011, the driver determines the mapping rules between the data source's data format and the standard format based on predefined standard specifications. It then sends these mapping rules to the data source's API interface. The API interface maps the queried data to the standard format and returns this standard-formatted data to the driver 403. The driver 403 then returns the unified-formatted data query results to the user 405 via gateway 404. As mentioned above, the data source format conversion can be achieved by adding an API interface to the sidecar module 4012 or the data source 40211; details will not be elaborated here.

[0071] The following describes a specific application scenario of the data management method and data query method according to embodiments of this disclosure. In a complex production system, such as in an industrial production process, data sources and storage methods are diverse. Data may originate from different instruments and equipment, and the data formats collected from these devices may also differ. Equipment suppliers can select different databases to store data based on the characteristics of the equipment. When dealing with complex situations, such as when modeling and simulating a group of devices, upper-layer applications need to connect to multiple data sources. By employing the method according to the present invention, deployed data sources can be automatically registered. Different data sources can also discover each other. This reduces the development work for data sources from different vendors and improves scalability. The added sidecar structure integrates some functions unrelated to the data source logic itself, such as registration, discovery, flow control, and routing functions. The sidecar can decouple data storage and control functions, thereby reducing DevOps (Development and Operations) costs. For example, when multiple data sources are stored in the system, the flow control function can solve the problem of network congestion.

[0072] According to the data management method, data query method, and service mesh system of this invention, a novel DaaS solution based on service mesh is proposed. By adding a sidecar structure to the data source, automatic registration and discovery of the data source, as well as traffic control and routing functions, are realized. On the other hand, by defining standard data format specifications and APIs, raw data can be mapped to a standard format, thereby providing standard data query results. Furthermore, the connection information of the data source can also be standardized according to the data source type template. When the data source is deployed in a cluster, using the sidecar, data sources that can provide standard connection information will automatically register with the registry center.

[0073] According to the data management method and data query method of the present invention, the sidecar structure is used without configuration files, thereby reducing the data source deployment time and having better scalability.

[0074] When data is retrieved from the underlying database, it will be provided in a standardized format. From the data consumer's perspective, this eliminates the need to concern themselves with the underlying data storage method.

[0075] As per the above reference Figures 1 to 4 The present disclosure describes a data management method, a data query method, and a service mesh system according to embodiments thereof. The various units of the data management apparatus and service mesh system described above can be implemented in hardware, software, or a combination of hardware and software.

[0076] Figure 5 A block diagram of a computing device 500 implementing data management and data querying according to an embodiment of the present disclosure is shown. According to one embodiment, the computing device 500 may include at least one processor 502 that executes at least one computer-readable instruction (i.e., the elements implemented in software above) stored or encoded in a computer-readable storage medium (i.e., memory 504).

[0077] It should be understood that the computer-executable instructions stored in memory 504, when executed, cause at least one processor 502 to perform the above-described combinations in the various embodiments of this disclosure. Figure 1-4 The description includes various operations and functions.

[0078] According to one embodiment, a non-transitory machine-readable medium is provided. This non-transitory machine-readable medium may have machine-executable instructions (i.e., the elements implemented in software as described above), which, when executed by a machine, cause the machine to perform the above-described combinations in the various embodiments of this disclosure. Figure 1-5 The description includes various operations and functions.

[0079] According to one embodiment, a computer program is provided, including computer-executable instructions that, when executed, cause at least one processor to perform the above-described embodiments of the present disclosure. Figure 1-5 The description includes various operations and functions.

[0080] According to one embodiment, a computer program product is provided, including computer-executable instructions that, when executed, cause at least one processor to perform the above-described embodiments of the present disclosure. Figure 1-5 The description includes various operations and functions.

[0081] It should be understood that the various embodiments in this specification are described in a progressive manner, and the same or similar parts between the various embodiments can be referred to each other. Each embodiment focuses on describing the differences from other embodiments. For example, the embodiments concerning the apparatus, the computing device, and the machine-readable storage medium described above are basically similar to the method embodiments, so the descriptions are relatively simple, and the relevant parts can be referred to the descriptions of the method embodiments.

[0082] The foregoing description has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0083] Not all steps and units in the above process and system structure diagrams are necessary; some steps or units can be omitted according to actual needs. The device structure described in the above embodiments can be a physical structure or a logical structure. That is, some units may be implemented by the same physical entity, or some units may be implemented by multiple physical entities, or they may be jointly implemented by certain components in multiple independent devices.

[0084] The specific embodiments described above with reference to the accompanying drawings are exemplary embodiments, but do not represent all embodiments that can be implemented or fall within the scope of the claims. The term "exemplary" as used throughout this specification means "serving as an example, instance, or illustration" and does not imply that it is "preferred" or "advantageous" compared to other embodiments. Specific details are included to provide an understanding of the described techniques. However, these techniques can be practiced without these specific details. In some instances, well-known structures and apparatuses are shown in block diagram form to avoid obscuring the concepts of the described embodiments.

[0085] The foregoing description of this disclosure is provided to enable any person skilled in the art to implement or use this disclosure. Various modifications to this disclosure will be apparent to those skilled in the art, and the general principles defined herein can be applied to other variations without departing from the scope of this disclosure. Therefore, this disclosure is not limited to the examples and designs described herein, but is consistent with the widest scope of the principles and novel features disclosed herein.

[0086] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. Data management methods, including: A sidecar module is added for each data source so that the data source can be used as a separate service. The sidecar module includes at least a registration submodule and a discovery submodule. The registration submodule includes the identification information of the data source. The discovery submodule is used to enable mutual discovery between different data sources; The data management device or data source parser parses the registration submodule to obtain the identification information of the data source; and registers the identification information of the data source in the service registration center.

2. The method as described in claim 1, wherein, The sidecar module further includes a flow control submodule and a routing submodule. The flow control submodule is used to control the flow of the data source, and the routing submodule is used to implement data routing of the data source.

3. A data management device, comprising: The sidecar addition unit is configured to add a sidecar module for each data source so that the data source can be used as a separate service. The sidecar module includes at least a registration submodule and a discovery submodule. The registration submodule includes the identification information of the data source. The discovery submodule is used to enable mutual discovery between different data sources; The parsing unit is configured to parse the registration submodule to determine the identification information of the data source; The registration unit is configured to register the identification information of the data source in the service registry.

4. Data query methods, including: Receive user query requests for the data source; The driver browses registered data sources in the service registry; If the requested data source is not found, a prompt message is returned to the user. If the requested data source is found in the service registry, the driver establishes a connection with the data source using the information in the sidecar module.

5. The method of claim 4, further comprising: Based on pre-defined standard interface specifications, the queried data is mapped to a standard format, and the data in the standard format is returned to the user.

6. Service mesh system, including data units and control units. The data unit includes multiple data sources, each data source having a sidecar module to enable the data source to function as a separate service; each sidecar module includes at least a registration submodule and a discovery submodule, the registration submodule including the identification information of the data source; the discovery submodule is used to enable mutual discovery between different data sources; and The control unit includes a service registry and a data source parser. The data source parser is used to parse the sidecar module and register the identifier information of the data source obtained by parsing in the service registry.

7. The system of claim 6, further comprising: The driving unit includes a driver and a service discovery module for the plurality of data sources respectively. The service discovery module is used to browse the service registry to determine whether the data source is registered in the service registry when the driving unit receives a user's query request for the data source.

8. A computing device (500), comprising: At least one processor (502); as well as A memory (504) coupled to the at least one processor (502) for storing instructions that, when executed by the at least one processor (502), cause the processor (502) to perform the method as described in any one of claims 1 to 2 and 4 to 5.

9. A non-transitory machine-readable storage medium storing executable instructions that, when executed, cause the machine to perform the method as claimed in any one of claims 1 to 2 and 4 to 5.

10. A computer program product tangibly stored on a computer-readable medium and comprising computer-executable instructions that, when executed, cause at least one processor to perform the method according to any one of claims 1 to 2 and 4 to 5.

Images