Data processing method, apparatus, device, and medium

Abstract

The application provides a data processing method, device, equipment and medium. When a data query request for querying data in a service database is received, data query information carried by the data query request is acquired, an associated service field having an association relationship with a target service field indicated by the data query information is acquired from a meta information database, the target service field and the associated service field are taken as to-be-queried service fields, service data corresponding to the to-be-queried service fields is acquired from the service database as candidate service data, the candidate service data is filtered based on a query parameter included in the data query information, and target service data is obtained. According to the above scheme, the acquisition of the associated service field can be realized based on the meta information database each time data is queried, and the data query can be realized based on the target service field and the associated service field, so that the data query logic can be standardized, and the data query efficiency is improved.

Description

Technical Field

[0001] The embodiments of the present invention relate to the field of data processing technology, and more specifically, the embodiments of the present invention relate to a data processing method, apparatus, device and medium. Background Technology

[0002] This section is intended to provide background or context for embodiments of the invention as set forth in the claims. The description herein is not an admission that it is prior art simply because it is included in this section.

[0003] With the advent of the digital age, the amount of internet data has grown exponentially. Traditional file systems can no longer meet the huge data storage needs. In this context, databases, as a data storage structure that can effectively store and manage large-scale structured data, are being used more and more widely.

[0004] Databases typically store large amounts of data, and database queries, as a means for users to extract and retrieve specific data from massive datasets, help locate and retrieve the data they actually need, effectively improving data acquisition efficiency. Therefore, there is an urgent need for a data processing method to conveniently and quickly implement data queries within databases, thereby improving query efficiency. Summary of the Invention

[0005] In this context, embodiments of the present invention aim to provide a data processing method, apparatus, device, and medium to provide a way to conveniently and quickly perform data queries in a database, thereby improving data query efficiency.

[0006] In a first aspect of the present invention, a data processing method is provided, the method comprising:

[0007] In response to receiving a data query request for querying data in the business database, the system obtains the data query information carried in the data query request. The data query information includes the target business field to be queried and the query parameters used to filter the business data in the business database.

[0008] From the metadata database used to record the relationships between business fields included in the business database, retrieve the associated business fields that are related to the target business field, and use the target business field and the associated business fields as the business fields to be queried;

[0009] Retrieve the business data corresponding to the business field to be queried from the business database as candidate business data;

[0010] Candidate business data is filtered based on query parameters to obtain target business data.

[0011] In one embodiment of the present invention, the business database includes multiple business data tables, and the meta information database includes multiple information sets. Different information sets are composed of business fields from different business data tables. For any information set, the information set also records reference business fields that have a reference relationship with the business fields that make up the information set. Each information set corresponds to a main business data table.

[0012] From the metadata database used to record the relationships between business fields included in the business database, retrieve the related business fields that are associated with the target business field, including:

[0013] Determine the target information set to which the target business field belongs from the metadata database, and obtain the main business data table corresponding to the target information set;

[0014] Retrieve candidate business fields from the main business data table corresponding to the target information set from the metadata database, and obtain candidate business fields that have a reference relationship with the candidate business fields. Use the obtained candidate business fields as associated business fields that have a relationship with the target business fields.

[0015] In one embodiment of the present invention, the method further includes:

[0016] Multiple information sets in the metadata database are cached in advance;

[0017] Before determining the target information set to which the target business field belongs from the metadata database and obtaining the main business data table corresponding to the target information set, this method also includes:

[0018] Check if the target information set is stored in the cache;

[0019] If the target information set is stored in the cache, then retrieve the target information set from the cache and retrieve the main business data table corresponding to the target information set;

[0020] If the target information set is not stored in the cache, the steps of determining the target information set to which the target business field belongs from the metadata database and obtaining the main business data table corresponding to the target information set are executed.

[0021] In one embodiment of the present invention, the metadata database is pre-built, and the construction process of the metadata database includes:

[0022] Based on business query requirements, business data tables in the business database that correspond to the same business query requirements are integrated into an information set;

[0023] For any set of information, based on the reference relationships between various business fields in the business database, obtain the referenced business field corresponding to each business field in the set of information.

[0024] In one embodiment of the present invention, the method further includes:

[0025] In response to an update operation in the business database, based on the target business data table indicated by the update operation, the set of information to be updated to which the target business data table belongs is determined from the metadata database, and the main business data table corresponding to the set of information to be updated is obtained.

[0026] From the main business data table, retrieve the business fields to be updated that have undergone update operations, as well as the business data of the business fields to be updated.

[0027] Follow the instructions of the update operation to update the business data of the business fields to be updated.

[0028] In one embodiment of the present invention, after updating the business data of the business field to be updated according to the instructions of the update operation, the method further includes:

[0029] Update the cached information set according to the updated business data in the metadata database.

[0030] In one embodiment of the present invention, the method further includes:

[0031] All business data in the business database is cached in advance on each database node of the business database;

[0032] Before retrieving the business data corresponding to the business field to be queried from the business database, this method also includes:

[0033] The query is performed in the cache of the database node that receives the data query request;

[0034] If the cache stores the business data corresponding to the business field to be queried, then retrieve the business data corresponding to the business field to be queried from the cache.

[0035] If the cache does not contain the business data corresponding to the business field to be queried, then the step of retrieving the business data corresponding to the business field to be queried from the business database will be executed.

[0036] In a second aspect of the present invention, a data processing apparatus is provided, the apparatus comprising:

[0037] The information acquisition module is used to respond to a data query request received for querying data in the business database, and to acquire the data query information carried in the data query request. The data query information includes the target business field to be queried and the query parameters used to filter the business data in the business database.

[0038] The field acquisition module is used to retrieve associated business fields that are related to the target business field from the metadata database that records the relationships between business fields included in the business database, and to use the target business field and associated business fields as the business fields to be queried.

[0039] The data acquisition module is used to retrieve the business data corresponding to the business field to be queried from the business database as candidate business data;

[0040] The filtering module is used to filter candidate business data based on query parameters in order to obtain target business data.

[0041] In a third aspect of the present invention, a computing device is provided, the computing device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to perform operations as described in the first aspect and any one of the first aspects.

[0042] In a fourth aspect of the present invention, a computer-readable storage medium is provided, on which a program is stored, the program being executed by a processor as performed by the data processing method described in the first aspect and any one of the first aspects.

[0043] The solution provided by this invention allows for the retrieval of associated business fields based on the metadata database each time data is queried. This enables data queries based on the target business field and associated business fields, standardizing the data query / update logic and improving query efficiency. Furthermore, the standardization of the data query / update logic avoids strong binding to a particular type of database, thus allowing for dynamic adjustments to various data structure definitions. This not only adapts to data persistence implementations from different data sources but also facilitates rapid iteration and upgrades of data processing capabilities in the future. Attached Figure Description

[0044] The above and other objects, features, and advantages of exemplary embodiments of the present invention will become readily apparent from the following detailed description taken in conjunction with the accompanying drawings. Several embodiments of the invention are illustrated in the drawings by way of example and not limitation, wherein:

[0045] Figure 1 This is a flowchart illustrating a data processing method according to an exemplary embodiment of the present invention.

[0046] Figure 2 This is a flowchart illustrating a process for obtaining candidate service data according to an exemplary embodiment of the present invention.

[0047] Figure 3This is a flowchart illustrating an update process for a metadata database according to an exemplary embodiment of the present invention.

[0048] Figure 4 This is a flowchart illustrating an update process for a metadata database according to an exemplary embodiment of the present invention.

[0049] Figure 5 This is a block diagram illustrating a data processing apparatus according to an exemplary embodiment of the present invention.

[0050] Figure 6 This is a schematic diagram illustrating a computer-readable storage medium according to an exemplary embodiment of the present invention.

[0051] Figure 7 This is a schematic diagram of the structure of a computing device according to an exemplary embodiment of the present invention.

[0052] In the accompanying drawings, identical or corresponding reference numerals denote identical or corresponding parts. Furthermore, any number of elements in the drawings is for illustrative purposes only and not for limitation, and any naming is for distinction only and has no limiting meaning. Detailed Implementation

[0053] The principles and spirit of the invention will now be described with reference to several exemplary embodiments. It should be understood that these embodiments are provided merely to enable those skilled in the art to better understand and implement the invention, and are not intended to limit the scope of the invention in any way. Rather, these embodiments are provided to make the invention more thorough and complete, and to fully convey the scope of the invention to those skilled in the art.

[0054] Those skilled in the art will recognize that embodiments of the present invention can be implemented as a system, apparatus, device, method, or computer program product. Therefore, the present invention can be specifically implemented in the following forms: entirely hardware, entirely software (including firmware, resident software, microcode, etc.), or a combination of hardware and software.

[0055] According to embodiments of the present invention, a data processing method, apparatus, device, and medium are proposed. Invention Overview

[0057] The inventors have discovered that in related technologies, to complete a data query operation in a database, it is often necessary to first perform Entity-Relationship (ER) modeling on the database to be queried. This clarifies the relationships between different entities in the database, allowing code to be generated based on these relationships to query different data. The data query can then be achieved by running the code. An entity is a logical combination of data information and can consist of multiple entity fields. In relational databases, a data table can be considered an entity, and entity fields are essentially equivalent to table fields in a relational database.

[0058] However, the code used to implement data query in the above scheme is strongly bound to the database design. It needs to be customized based on the ER modeling results of the database, which means that the above query process can only process data statically. Once the data structure of the database is adjusted, the code customized based on the original data structure will not run smoothly, resulting in data query failure.

[0059] In view of this, the present invention aims to provide a data processing method for implementing data query operations in a database, and to ensure the smooth implementation of the data query process even when the data structure of the database is adjusted.

[0060] Application Scenarios Overview

[0061] The data processing method provided by this invention can be used to perform data queries in various types of databases. For example, the data processing method provided by this invention can be used to perform data queries in various types of databases such as hierarchical databases, network databases, object-oriented databases, document databases, and columnar storage databases. This invention does not limit the specific database to which it is applied.

[0062] The above data processing method can be executed by a computing device, which can be a server, such as a single server, multiple servers, a server cluster, a cloud computing platform, etc. This invention does not limit the type of computing device.

[0063] It should be noted that the above application scenarios are shown only to facilitate understanding of the spirit and principles of the present invention. The implementation of the present invention is not limited in any way in this respect. On the contrary, the implementation of the present invention can be applied to any applicable scenario.

[0064] Exemplary methods

[0065] The data processing method provided by this invention will be described in detail below, in conjunction with the above-mentioned application scenarios.

[0066] See Figure 1 , Figure 1 This is a flowchart illustrating a data processing method according to an exemplary embodiment of the present invention, such as... Figure 1 As shown, the method includes:

[0067] S101. In response to receiving a data query request for querying data in the business database, obtain the data query information carried in the data query request. The data query information includes the target business field to be queried and the query parameters used to filter the business data in the business database.

[0068] It should be noted that data query requests can be triggered by users through terminal devices. For example, when a user has a data query need, they can trigger a data query request through a terminal device, which will then send the data query request to a computing device so that the computing device can receive the data query request and perform a query in the business database based on the received data query request.

[0069] In this context, when storing business data, the business database can determine the corresponding business fields (or entity fields) based on the meaning of the data. This allows for the storage of data with different meanings through different business fields. For example, for data describing product output, a "output" field can be maintained in the business database, thus enabling the storage of data describing product output through this field.

[0070] Based on this, when a data query request is triggered, the target business field to be queried can be set according to the meaning of the data to be queried. A data query request to obtain the business data corresponding to the target business field can be generated based on the specified target business field, so that the computing device can query the data corresponding to the target business field of the business data based on the received data query request.

[0071] However, business databases store a large amount of data. Even if a target business field is specified, not all business data corresponding to that field is needed for the query. Therefore, when triggering a data query request, query parameters for filtering the business data in the database can be determined based on the attribute characteristics of the data to be queried. For example, for numeric data, the query parameters can be determined based on the range of values ​​in the business data to be queried; for character data, the query parameters can be determined based on the specific characters included in the business data to be queried, and so on.

[0072] The query parameters can correspond to at least one business field. Due to the correlation or hierarchical relationship between data, some business fields may be related. Therefore, when determining the query parameters, the business field corresponding to the determined query parameters can be the target business field, or the business field corresponding to the determined query parameters can be a related business field of the target business field.

[0073] By setting query parameters, data query requests can be generated based on the set parameters, enabling the computing device to filter out the required data from the business data in the business database based on the received data query request.

[0074] S102. Obtain the associated business fields that are related to the target business field from the metadata database used to record the relationships between the business fields included in the business database, and use the target business field and the associated business fields as the business fields to be queried.

[0075] It should be noted that a metadata database can be pre-built to record the relationships between business fields included in the business data. This allows the acquisition of associated business fields corresponding to the target business field based on the target business field included in the data query information after obtaining the data query information carried in the data query request. This enables the target business field and its associated business fields to be used together as the business fields to be queried.

[0076] S103. Obtain the business data corresponding to the business field to be queried from the business database as candidate business data.

[0077] S104. Filter candidate business data based on query parameters to obtain target business data.

[0078] The above solution allows for the retrieval of associated business fields from the metadata database each time data is queried. This enables data queries based on the target business field and associated business fields, standardizing the data query / update logic and improving data processing capabilities. Furthermore, this standardization avoids strong binding to a particular type of database, allowing for dynamic adjustments to various data structure definitions. This not only adapts to data persistence implementations from different data sources but also facilitates rapid iteration and upgrades of the business system's data processing capabilities in the future.

[0079] After introducing the basic implementation process of the present invention, the various optional embodiments of the present invention will be described below.

[0080] The metadata database can be pre-built to ensure the smooth implementation of the data query process. The construction process of the metadata database will be introduced below.

[0081] In some embodiments, data metadata can be defined based on business fields corresponding to business data stored in the business database. Metadata is data used to describe data characteristics and attributes, providing contextual and attribute information about the data. For example, metadata can be used to indicate the data type of a business field, including but not limited to value types and reference types.

[0082] If the data type of a business field is a value type, it indicates that there is no reference relationship between the business field and other business fields. If the data type of a business field is a reference type, it indicates that the business field will reference other business fields.

[0083] The value type can be directly represented by the database content, serving as the data representation of the database content. After data is stored in the database, the data value type can be obtained based on the data representation of the stored data. Optionally, the value type can include text, numbers, dates, etc., and this invention does not limit the specific content of the value type.

[0084] Reference types are used to indicate the reference relationships between different business fields. For example, when defining the reference type for any business field, the reference type can be defined by storing the identification information of the business field referenced by that business field (or the referenced business field). Optionally, the identification information can be the identifier of the referenced business field (such as a primary key ID), or it can be the encoding of the referenced business field, and so on.

[0085] After defining the metadata for each business field in the business database, the metadata database can be constructed using the following steps:

[0086] Step 1: Based on business query requirements, integrate the business data tables in the business database that correspond to the same business query requirements into a single information set.

[0087] It should be noted that a business database can include multiple business data tables. These tables can be used to record business data of the same entity in different aspects, and / or, they can be used to record business data of different entities. Generally, a business query request can be used to retrieve relevant data of a specific entity within a business scenario, and different business query requests can be used to retrieve relevant data of different entities. Therefore, when integrating business data in a business database corresponding to the same business query request into an information set, the business data tables used to store relevant data of the entity corresponding to that business query request can be integrated into an information set.

[0088] Taking a business database used to store company personnel data as an example, generally speaking, this type of business database will store employee basic data, employee education experience data, employee contact data, departmental division data, etc. Employee basic data may include employee identifier (i.e., employee ID), employee name, employee number, and employee department identifier (i.e., employee department ID). Employee education experience data may include employee identifier, graduating school (or educational institution), enrollment date (or start date), and graduation date (or end date).

[0089] Contact data can include contact identifier, contact name, contact information, and relationship with employees. Some departmental data may include department identifier (or department ID), department name, and relevant personnel. For a business database storing company personnel data, the data primarily corresponds to two entities: employees and departments. Therefore, the business data tables storing basic employee data, employee education / experience data, and employee contact data can be integrated into a single information set, as can the business data tables storing departmental data.

[0090] Optionally, each business data table can be treated as an entity in the metadata database to be created, and a corresponding identifier can be generated for each business data table. In addition, a corresponding identifier can be generated for each business field in the business data table, so that the association between the business field and its corresponding business data table can be realized based on the generated identifier.

[0091] For example, the identifier generated for a business data table can include an entity identifier (i.e., entity ID) and an entity code, while the identifier generated for a business field can include a field identifier (i.e., field ID) and a field code. When recording the business data table to which each business field belongs, the entity identifiers of the business fields and their respective business data tables can be stored together, so that the business data table to which each business field belongs can be determined by the entity identifier associated with it.

[0092] In one possible implementation, by integrating business data tables in the business database that correspond to the same business query requirement into an information set, the subject corresponding to the business data recorded in each business data table can be determined, thereby integrating business data tables corresponding to the same subject into an information set. Alternatively, based on the actual situation of the business query requirement, partial content from business data tables corresponding to the same subject can be integrated into an information set.

[0093] An information set is a data unit that can be processed by a user, consisting of some or all of the data from one or more entities (i.e., business data tables). If there are multiple entities in an information set, only one entity will be the master entity, and the others will be referencing entities. In other words, each information set can be composed of data from multiple business data tables, and among the multiple business data tables that make up the information set, there will be a master business data table.

[0094] Optionally, an information set identifier (i.e., information set ID) can be generated for each information set, and the entity ID of the main business data table corresponding to each information set can be recorded.

[0095] Step 2: For any information set, based on the references between various business fields in the business database...

[0096] Use relationships to retrieve the referenced business fields corresponding to each business field in the information set.

[0097] It should be noted that since the reference types of each business field in the business database have been predefined, that is, the reference relationships between each business field in the business database have been predefined, the referenced business fields corresponding to each business field in the information set can be obtained based on the predefined reference relationships.

[0098] In one possible implementation, a table can be used to integrate the fields included in multiple information sets. For example, a table can record the business fields in the business data tables included in the information sets, the information sets to which each business field belongs, and the business data tables to which each business field belongs. In addition, the table can also record the reference relationships between the business fields in the information sets.

[0099] Optionally, field expressions can be used to record the reference relationships between business fields in an information set. That is, for any business field in any information set, a field expression can be generated for that business field to record the referenced business fields corresponding to that business field.

[0100] The field expression can have the following types:

[0101] (1) Direct reference: For example, directly specify the field identifier of the business field (i.e. the referenced business field) in a certain business field;

[0102] (2) Hierarchical reference: Hierarchical references are expressed by separators (such as decimal points). For example, if the field expression corresponding to a certain business field is ABC, it means that to get the data corresponding to the reference business field, you must first get the data corresponding to business field A, then get the data corresponding to business field B, and finally get the data corresponding to business field C. Among them, business field A and business field B must be reference types.

[0103] (3) Object reference: Ends with .*, indicating that the entire object is referenced here. In addition, a sub-reference needs to be defined. The sub-reference can be a direct reference, a hierarchical reference, or an object reference. For business fields in the information set that serve as the parent node of other business fields, the business field needs to be set as an object reference.

[0104] The metadata database enables the integration of multiple business data tables in the business database. The metadata database can include multiple information sets, and different information sets can be composed of business fields from different business data tables. For any information set, the information set can also record reference business fields that have a reference relationship with the business fields that make up the information set.

[0105] To facilitate understanding, the following example will be used to introduce the process of building a metadata database.

[0106] Taking a business database used to store company personnel data as an example, this business database includes the business data tables shown in Tables 1 to 4 below:

[0107] Table 1

[0108]

[0109] Table 2

[0110]

[0111] Table 3

[0112]

[0113] Table 4

[0114]

[0115] The business data tables 1 to 4 above define four employees, one piece of relevant information for these four employees, and two departments to which these four employees belong. Table 1 displays the basic information of these four employees, Table 2 displays the two departments to which these four employees belong and their department heads, Table 3 displays the educational background of employee "Wang Wu," and Table 4 displays the contact information of employee "Wang Wu." Based on the business data tables shown above, the metadata database shown in Tables 5 to 4 below can be constructed:

[0116] Table 5

[0117]

[0118] Table 6

[0119]

[0120] Table 7

[0121]

[0122]

[0123] Table 8

[0124]

[0125] As shown in Tables 5 to 8 above, Table 5 shows the business data tables included in the business database and the identifiers of each business data table; Table 6 shows the business fields included in each business data table and the relevant information of each business field; Table 7 shows the information sets included in the metadata database and the identifiers of each information set; and Table 8 shows the business fields included in each information set and the relevant information of each business field.

[0126] By pre-building a metadata database, data queries can be performed based on the pre-built metadata database.

[0127] In some embodiments, when retrieving the associated business field that is associated with the target business field from the metadata database used to record the relationship between business fields included in the business database in step S102, it can be achieved through the following steps:

[0128] S1021. Determine the target information set to which the target business field belongs from the metadata database, and obtain the main business data table corresponding to the target information set.

[0129] In one possible implementation, the information set identifier of the target information set to which the target business field belongs can be obtained from the metadata database. The target information set can be determined based on the information set identifier. In the metadata database, each information set is associated with the entity identifier of its corresponding main business data table. Therefore, after the target information set is determined, the entity identifier of the main business data table corresponding to the target information set can be obtained to obtain the main business data table corresponding to the target information set.

[0130] S1022. Obtain candidate business fields from the main business data table corresponding to the target information set from the metadata database, and obtain candidate business fields that have a reference relationship with the candidate business fields, and use the obtained candidate business fields as associated business fields that have an association relationship with the target business fields.

[0131] After determining the main business data table corresponding to the target information set, the business fields in the main business data table can be obtained as candidate business fields. Furthermore, the candidate business fields referenced by each candidate business field can be obtained, and these two types of candidate business fields can be used as related business fields.

[0132] After obtaining the associated business field of the target business field through the above embodiments, the obtained associated business field and the target business field can be used together as the business field to be queried, so as to obtain the business field to be queried.

[0133] To facilitate understanding, the following example uses a metadata database as shown above to illustrate the process of obtaining the business fields to be queried.

[0134] Taking the data query request as an example of a request for obtaining the details of employees whose department leaders' names contain the character "Six", according to the information set definition shown in Table 7 above, it is determined that the information set ID of the information set named "Employee Details" is 3000. Then, the information set with the information set ID of 3000 can be determined as the target information set. According to the information set definition shown in Table 7 above, it can be known that the entity ID of the main business data table corresponding to the target information set is 1000. Then, according to the entity field definition shown in Table 6 above, the business fields included in the data table with the entity ID of 1000 can be obtained, which are the employee ID (emp_no), employee name (emp_name), department identifier (dept_id), and emergency contact (contact_ids). Among them, the data types of emp_no and emp_name are both value types, and emp_no and emp_name can be directly used as the business fields to be queried. However, the data type of dept_id is a reference type, and the entity ID of the business data table it references is 1001. Therefore, the business fields included in the business data table with the entity ID of 1001 need to be obtained, which are the department name (dept_name) and the department leader's name (dept_leader_name). In addition, the data type of contact_ids is also a reference type, and the entity ID of the business data table it references is 1003. Therefore, the business fields included in the business data table with the entity ID of 1003 need to be obtained, which are the contact name (name) and the contact method (phone).至此,即可获取到所有的待查询业务字段,所获取到的待查询业务字段包括员工工号(emp_no)、员工姓名(emp_name)、部门名称(dept_name)、部门负责人姓名(dept_leader_name)、联系人姓名(name)和联系方式(phone)。

[0135] The above embodiments are described by taking the query of the target information set and candidate business fields directly from the meta-information database as an example. In more possible implementation manners, multiple information sets in the meta-information database can also be cached in advance.

[0136] In some embodiments, a method of caching multiple information sets in the meta-information database in memory can be adopted to achieve the caching of multiple information sets in the meta-information database.

[0137] Therefore, before determining the target information set to which the target business field belongs in the meta-information database and obtaining the main business data table corresponding to the target information set, it is also possible to first detect whether the target information set is stored in the cache. If the target information set is stored in the cache, directly obtain the target information set from the cache and obtain the main business data table corresponding to the target information set; and in the case where the target information set is not stored in the cache, then determine the target information set to which the target business field belongs in the meta-information database and obtain the main business data table corresponding to the target information set.

[0138] In addition, it should be noted that in the case where the target information set is not stored in the cache, it is also possible to first obtain multiple information sets from the meta-information database and then cache the multiple information sets, so that queries can be directly performed from the cache. When subsequent query operations need to be performed, queries can also be directly performed from the cache without accessing the meta-information database.

[0139] By this way of pre-caching the information sets in the meta-information database, it is not necessary to use the meta-information database for queries every time, and only access the meta-information database when the cache cannot meet the query requirements, thereby reducing the access operations to the meta-information database and reducing the processing pressure on the meta-information database.

[0140] Optionally, for a distributed database, the multi-node synchronous replication (Infinispan Replicated) method can be used to implement the caching of multiple information sets in the meta-information database. That is, multiple information sets can be cached on each database node of the distributed database, so that each database node in the distributed database caches the full amount of information sets, enabling each database node to obtain the business fields to be queried without complex remote calls, improving the data query speed and data query efficiency.

[0141] After obtaining the business fields to be queried through the above embodiments, the business data corresponding to the business fields to be queried can be obtained from the business database through step S103 as the candidate business data.

[0142] Still taking the data query request as an example of a request for obtaining the details of employees whose department leaders' names contain the character 'Six', after obtaining the business fields to be queried such as employee number (emp_no), employee name (emp_name), department name (dept_name), department head name (dept_leader_name), contact name (name), and contact information (phone), the candidate business data can be obtained based on the obtained business fields to be queried.

[0143] The `emp_no` and `emp_name` fields are value types, meaning the candidate business data corresponding to these fields can be directly retrieved from the main business data table of the target information set. After obtaining the candidate business data for the `emp_no` and `emp_name` fields, the candidate business data for multiple query fields can be assembled as shown in Table 9 below:

[0144] Table 9

[0145] emp_no emp_name dept_name dept_leader_name contacts GH001 Zhang San GH002 Li Si GH003 Wang Wu GH004 Zhao Liu

[0146] The `dept_name` field is obtained from the business data table with entity ID 1002 through the `dept_id` field. In the business data table with entity ID 1001, the `dept_name` field is a value type, so the candidate business data corresponding to the `dept_name` field can be directly obtained from the business data table with entity ID 1001. After obtaining the candidate business data corresponding to the `dept_name` field, the candidate business data for multiple query fields can be assembled as shown in Table 10 below:

[0147] Table 10

[0148]

[0149]

[0150] The field expression for the `dept_leader_name` field is F006.F002, indicating that this field is a hierarchical reference. When retrieving candidate business data corresponding to the `dept_leader_name` field, it's necessary to first retrieve the field encoded as F006. The field encoded as F006 originates from the business data table with entity ID 1001, which references entity ID 1000. Therefore, the data corresponding to the business field encoded as F002 is retrieved from the business data table with entity ID 1000 and used as the candidate business data corresponding to the `dept_leader_name` field. After retrieving the candidate business data corresponding to the `dept_leader_name` field, the candidate business data for multiple query fields can be assembled as shown in Table 11 below.

[0151] Table 11

[0152] emp_no emp_name dept_name dept_leader_name contacts GH001 Zhang San Development Department Zhang San GH002 Li Si Development Department Zhang San GH003 Wang Wu Sales Department Zhao Liu GH004 Zhao Liu Sales Department Zhao Liu

[0153] The field expression for the `contacts` field is `F004.*`, indicating that this field is a global object reference. The ID of the `contacts` field is 4004, and it has two child nodes: a `name` field encoded as `F010` and a `phone` field encoded as `F011`. Therefore, by retrieving the business data corresponding to the `name` field (encoded as `F010`) and the `phone` field (encoded as `F011`), the candidate business data corresponding to the `contacts` field can be obtained. After retrieving the candidate business data corresponding to the `contacts` field, the candidate business data for multiple query fields can be assembled as shown in Table 12 below.

[0154] Table 12

[0155]

[0156]

[0157] At this point, the candidate business data corresponding to each business field to be queried can be obtained.

[0158] The process of obtaining candidate service data provided in the above embodiments can be found in [reference needed]. Figure 2 , Figure 2 This is a flowchart illustrating a process for acquiring candidate service data according to an exemplary embodiment of the present invention, as shown below. Figure 2 As shown, users can specify the information set to be queried and the query conditions (i.e., query parameters) so that the definition of the corresponding information set can be queried based on the metadata database. The main entity data (i.e., the main business data table) of the information set to be queried can be obtained from the business database (i.e., the predefined data source). Based on the entity definition in the metadata database, it can be determined whether the main entity is associated with other entities. If not, the candidate business data can be assembled directly. Otherwise, it is necessary to collect the associated entities and obtain the entity data of the associated entities from the business database to assemble the candidate business data.

[0159] The above embodiment is illustrated by taking the acquisition of candidate business data directly from the business database as an example. In more possible implementations, the business data in the business database can also be cached in advance.

[0160] In some embodiments, the entire set of business data in the business database can be cached in memory to achieve caching of business data within the business data. For example, the entire set of business data in the business database can be cached on each database node of the business database.

[0161] Therefore, before obtaining the business data corresponding to the business fields to be queried from the business database, it is also possible to query in the cache of the database node that receives the data query request. If the business data corresponding to the business fields to be queried is stored in the cache, then the business data corresponding to the said business fields to be queried can be obtained from the cache; if the business data corresponding to the business fields to be queried is not stored in the cache, then obtain the business data corresponding to the business fields to be queried from the business database.

[0162] In addition, it should be noted that in the case where the required business data is not stored in the cache, it is also possible to first obtain the full amount of business data from the business database, and then cache the obtained full amount of business data, so that it is possible to directly query from the cache. When subsequent query operations are required, it is also possible to directly query from the cache without accessing the business database.

[0163] By this way of pre-caching the business data in the business database, it is not necessary to use the business database for each query operation, and only access the business database when the cache cannot meet the query requirements, thereby reducing the access operations to the business database and reducing the processing pressure on the business database.

[0164] In addition, by caching the full amount of business data in the business database to each database node in the business database, the effect of multi-node synchronous replication (Infinispan Replicated) can be achieved, so that each database node in the database caches the full amount of business data, enabling each database node to obtain the candidate business data without complex remote calls, improving the data query speed and data query efficiency.

[0165] After obtaining the candidate business data, through step S104, the candidate business data can be filtered based on the query parameters to obtain the target business data.

[0166] Still taking the data query request as an example for obtaining the details of employees whose department leaders' names contain the character 'Six', after obtaining the candidate business data as shown in Table 12 above, according to the query parameters corresponding to the query conditions indicated by the data query request, that is, the dept_leader_name field contains 'Six', the candidate business data can be filtered to obtain the target business data. The filtered target business data can be seen in Table 13 below:

[0167] Table 13

[0168] <000 /

[0169] Thus, the acquisition of the target business data can be achieved.

[0170] The implementation scheme provided by this invention enables runtime metadata modeling of business data, resulting in a metadata database capable of retrieving relationships between different business fields. This allows subsequent query operations to be performed simply by retrieving the relevant business fields from the metadata database based on the query requirements, thus enabling data retrieval without the need to pre-define query code according to the database structure, thereby improving the flexibility of the data query process. Furthermore, through this operational abstraction of modeling the metadata of business data, the resulting information set does not require a fixed structure, can adapt to any type of database, exhibits strong portability, and can effectively support data source switching and persistence implementations across different databases.

[0171] In other possible implementations, the existing metadata database can be updated based on actual technical requirements.

[0172] See Figure 3 , Figure 3 This is a flowchart illustrating an update process for a metadata database according to an exemplary embodiment of the present invention, such as... Figure 3 As shown, the process may include the following steps:

[0173] S301. In response to an update operation in the business database, based on the target business data table indicated by the update operation, determine the information set to be updated belonging to the target business data table from the metadata database, and obtain the main business data table corresponding to the information set to be updated.

[0174] S302. From the main business data table, obtain the business field to be updated and the business data of the business field to be updated that has been updated.

[0175] S303. Update the business data of the business field to be updated according to the update operation instructions.

[0176] To make it easier to understand, the following example of updating an employee's name and contact person will be used to illustrate the process of updating the metadata database.

[0177] For example, update the employee's name from employee ID GH003 to Wang Xiaowu, and add a contact person named Wang Xiaoxiao (133000000).

[0178] The updated information set contains the following:

[0179]

[0180]

[0181] By querying, we can determine that the information set ID to be updated is 3000. The main business data table of the information set ID 3000 is the entity with ID 1000. Therefore, we can modify the data in the emp_name field corresponding to the employee with emp_no = GH003 in the business data table of the corresponding entity with ID 1000. The modification results are shown in Table 14 below:

[0182] Table 14

[0183]

[0184] By querying the metadata database, it can be found that the `contacts` field is a reference type, which is associated with data for entity contacts 400 and 401, as shown in Table 14 below:

[0185] Table 14

[0186]

[0187] By comparing Table 14 above with the updated information set, it can be found that if a contact's information needs to be added, the data can be added based on the updated information set to obtain the updated information set shown in Table 15 below:

[0188] Table 15

[0189]

[0190]

[0191] At this point, the update of the information set in the metadata database can be completed.

[0192] The process of updating the metadata database provided in the above embodiments can be found in [reference needed]. Figure 4 , Figure 4 This is a flowchart illustrating an update process for a metadata database according to an exemplary embodiment of the present invention, as shown below. Figure 4 As shown, based on the information set and content to be updated specified by the user, the system queries the definition of the corresponding information set from the metadata database and retrieves the main entity data (i.e., the main business data table) of the information set to be updated from the business database (i.e., the predefined data source) to update the main entity data. Furthermore, based on the entity definition in the metadata database, the system determines whether the main entity is associated with other entities. If not, the update process can be exited directly. Otherwise, it is necessary to collect the associated entities and retrieve the entity data of the associated entities from the business database to update the business data of the associated entities.

[0193] Optionally, after updating the information set in the metadata database, the cached information set can also be updated according to the updated business data in the metadata database.

[0194] The solution provided by this invention allows for adjustments to the relevant logic of business data processing without modifying the code, enabling real-time dynamic adjustments, accelerating the iterative updates of the underlying data capabilities of the information system, and further improving code reusability.

[0195] Exemplary device

[0196] In addition, the present invention also provides a data processing apparatus, see [link to relevant documentation]. Figure 5 , Figure 5 This is a block diagram illustrating a data processing apparatus according to an exemplary embodiment of the present invention, such as... Figure 5 As shown, the device includes:

[0197] The information acquisition module 501 is used to respond to receiving a data query request for querying data in the business database, and to acquire the data query information carried by the data query request. The data query information includes the target business field to be queried and the query parameters used to filter the business data in the business database.

[0198] The field acquisition module 502 is used to acquire related business fields that are related to the target business field from the metadata database used to record the relationship between business fields included in the business database, and to use the target business field and related business fields as business fields to be queried.

[0199] Data acquisition module 503 is used to obtain the business data corresponding to the business field to be queried from the business database as candidate business data;

[0200] The filtering module 504 is used to filter candidate business data based on query parameters to obtain target business data.

[0201] In some embodiments, the business database includes multiple business data tables, and the meta information database includes multiple information sets. Different information sets are composed of business fields from different business data tables. For any information set, the information set also records reference business fields that have a reference relationship with the business fields that make up the information set. Each information set corresponds to a main business data table.

[0202] Field acquisition module 502, when acquiring associated business fields that are related to the target business field from the metadata database used to record the relationships between business fields included in the business database, is used to:

[0203] Determine the target information set to which the target business field belongs from the metadata database, and obtain the main business data table corresponding to the target information set;

[0204] Retrieve candidate business fields from the main business data table corresponding to the target information set from the metadata database, and obtain candidate business fields that have a reference relationship with the candidate business fields. Use the obtained candidate business fields as associated business fields that have a relationship with the target business fields.

[0205] In some embodiments, the device further includes:

[0206] The first caching module is used to cache multiple information sets in the metadata database in advance;

[0207] The detection module is used to detect whether the target information set is stored in the cache before determining the target information set to which the target business field belongs from the metadata database and obtaining the main business data table corresponding to the target information set.

[0208] The field acquisition module 502 is also used to retrieve the target information set from the cache if the target information set is stored in the cache, and to retrieve the main business data table corresponding to the target information set.

[0209] The field acquisition module 502 is also used to perform the steps of determining the target information set to which the target business field belongs from the metadata database and obtaining the main business data table corresponding to the target information set if the target information set is not stored in the cache.

[0210] In some embodiments, the device further includes:

[0211] Modules are used to pre-build the metadata database;

[0212] The building module, when used to pre-build a metadata database, is used for:

[0213] Based on business query requirements, business data tables in the business database that correspond to the same business query requirements are integrated into an information set;

[0214] For any set of information, based on the reference relationships between various business fields in the business database, obtain the referenced business field corresponding to each business field in the set of information.

[0215] In some embodiments, the device further includes an update module, which is configured to:

[0216] In response to an update operation in the business database, based on the target business data table indicated by the update operation, the set of information to be updated to which the target business data table belongs is determined from the metadata database, and the main business data table corresponding to the set of information to be updated is obtained.

[0217] From the main business data table, retrieve the business fields to be updated that have undergone update operations, as well as the business data of the business fields to be updated.

[0218] Follow the instructions of the update operation to update the business data of the business fields to be updated.

[0219] In some embodiments, the update module is further configured to:

[0220] Update the cached information set according to the updated business data in the metadata database.

[0221] In some embodiments, the device further includes:

[0222] The second caching module is used to pre-cacherate all business data in the business database to each database node in the business database;

[0223] The query module is used to perform a query in the cache of the database node that receives the data query request before retrieving the business data corresponding to the business field to be queried from the business database.

[0224] The data acquisition module 503 is also used to retrieve the business data corresponding to the business field to be queried from the cache if the cache stores the business data corresponding to the business field to be queried.

[0225] The data acquisition module 503 is also used to perform the step of retrieving the business data corresponding to the business field to be queried from the business database if the business data corresponding to the business field to be queried is not stored in the cache.

[0226] It should be noted that although several modules of the email reply device have been mentioned in the detailed description above, this division is merely exemplary and not mandatory. In fact, according to embodiments of the present invention, the features and functions of two or more modules described above can be embodied in a single module. Conversely, the features and functions of a single module described above can be further divided and embodied by multiple modules.

[0227] Exemplary media

[0228] This invention also provides a computer-readable storage medium. Figure 6 This is a schematic diagram illustrating a computer-readable storage medium according to an exemplary embodiment of the present invention, such as... Figure 6 As shown, the storage medium stores a computer program 601, which, when executed by a processor, can perform the data processing method provided in any embodiment of the present invention.

[0229] Exemplary device

[0230] This invention also provides a computing device, which may include a memory and a processor. The memory stores computer instructions that can run on the processor, and the processor, when executing the computer instructions, implements the data processing method provided in any embodiment of this invention. See also... Figure 7 , Figure 7 This is a schematic diagram of the structure of a computing device according to an exemplary embodiment of the present invention. The computing device 700 may include, but is not limited to, a processor 710, a memory 720, and a bus 730 connecting different system components (including the memory 720 and the processor 710).

[0231] The memory 720 stores computer instructions that can be executed by the processor 710, enabling the processor 710 to perform the data processing method provided in any embodiment of the present invention. The memory 720 may include a random access memory (RAM) 721, a cache memory 722, and / or a read-only memory (ROM) 723. The memory 720 may also include a program tool 725 having a set of program modules 724, including but not limited to: an operating system, one or more application programs, other program modules, and program data. One or more combinations of these program modules may include an implementation of a network environment.

[0232] Bus 730 may include, for example, a data bus, an address bus, and a control bus. The computing device 700 can also communicate with external devices 750 via I / O interface 740, such as a keyboard or a Bluetooth device. The computing device 700 can also communicate with one or more networks via network adapter 770, such as a local area network (LAN), a wide area network (WAN), or a public network. Figure 7 As shown, the network adapter 760 can also communicate with other modules of the computing device 700 via the bus 730.

[0233] Exemplary Products

[0234] This invention also provides a computer program product, which includes a computer program. When the program is executed by the processor 710 of the computing device 700, it can implement the data processing method provided in any embodiment of this invention.

[0235] Furthermore, although the operations of the method of the present invention are described in a specific order in the accompanying drawings, this does not require or imply that these operations must be performed in that specific order, or that all the operations shown must be performed to achieve the desired result. Additionally or alternatively, certain steps may be omitted, multiple steps may be combined into one step, and / or one step may be broken down into multiple steps.

[0236] While the spirit and principles of the invention have been described with reference to several specific embodiments, it should be understood that the invention is not limited to the disclosed specific embodiments, and the division of aspects does not imply that features in these aspects cannot be combined for benefit; such division is merely for ease of description. The invention is intended to cover various modifications and equivalent arrangements included within the spirit and scope of the appended claims.

Claims

1. A data processing method, characterized in that, The method includes: In response to receiving a data query request for querying data in a business database, the system obtains the data query information carried by the data query request. The data query information includes the target business field to be queried and query parameters for filtering business data in the business database. The business database includes multiple business data tables, and business fields from different business data tables form different information sets. Multiple information sets constitute a metadata database. For any information set, the information set also records reference business fields that have a reference relationship with the business fields that make up the information set. Each information set corresponds to a main business data table. From the metadata database, determine the target information set to which the target business field belongs, and obtain the main business data table corresponding to the target information set; Candidate business fields from the main business data table corresponding to the target information set are obtained from the metadata database, and candidate business fields that have a reference relationship with the candidate business fields are obtained. The obtained candidate business fields are used as associated business fields that have an association relationship with the target business fields, so that the target business fields and the associated business fields are used as business fields to be queried. Retrieve the business data corresponding to the business field to be queried from the business database, as candidate business data; The candidate business data is filtered based on the query parameters to obtain the target business data; The metadata database is pre-built, and the construction process of the metadata database includes: Based on business query requirements, the business data tables in the business database that correspond to the same business query requirement are integrated into an information set; For any information set, based on the reference relationships between various business fields in the business database, obtain the referenced business field corresponding to each business field in the information set.

2. The method according to claim 1, characterized in that, The method further includes: Multiple information sets in the metadata database are cached in advance; Before determining the target information set to which the target business field belongs from the metadata database and obtaining the main business data table corresponding to the target information set, the method further includes: Check whether the target information set is stored in the cache; If the target information set is stored in the cache, then the target information set is retrieved from the cache, and the main business data table corresponding to the target information set is retrieved. If the target information set is not stored in the cache, then the steps of determining the target information set to which the target business field belongs from the metadata database and obtaining the main business data table corresponding to the target information set are executed.

3. The method according to claim 1, characterized in that, The method further includes: In response to an update operation in the business database, based on the target business data table indicated by the update operation, the set of information to be updated to which the target business data table belongs is determined from the metadata database, and the main business data table corresponding to the set of information to be updated is obtained. From the main business data table, obtain the business field to be updated that has undergone an update operation and the business data of the business field to be updated; The business data of the business field to be updated is updated according to the instructions of the update operation.

4. The method according to claim 3, characterized in that, After updating the business data of the business field to be updated according to the instructions of the update operation, the method further includes: Update the cached information set according to the updated business data in the metadata database.

5. The method according to claim 1, characterized in that, The method further includes: All business data in the business database is pre-cached on each database node of the business database; Before retrieving the business data corresponding to the business field to be queried from the business database, the method further includes: The query is performed in the cache of the database node that received the data query request; If the cache stores the business data corresponding to the business field to be queried, then retrieve the business data corresponding to the business field to be queried from the cache. If the cache does not contain the business data corresponding to the business field to be queried, then the step of retrieving the business data corresponding to the business field to be queried from the business database is executed.

6. A data processing apparatus, characterized in that, The device includes: The information acquisition module is used to respond to a received data query request for querying data in a business database, and to acquire the data query information carried by the data query request. The data query information includes the target business field to be queried and query parameters for filtering business data in the business database. The business database includes multiple business data tables, and business fields from different business data tables form different information sets. Multiple information sets constitute a meta-information database. For any information set, the information set also records reference business fields that have a reference relationship with the business fields that make up the information set. Each information set corresponds to a main business data table. The field acquisition module is used to determine the target information set to which the target business field belongs from the metadata database, and to obtain the main business data table corresponding to the target information set; The field acquisition module is further configured to acquire candidate business fields from the main business data table corresponding to the target information set from the metadata database, acquire candidate business fields that have a reference relationship with the candidate business fields, and use the acquired candidate business fields as associated business fields that have an association relationship with the target business fields, so as to use the target business fields and the associated business fields as business fields to be queried. The data acquisition module is used to acquire the business data corresponding to the business field to be queried from the business database as candidate business data; The filtering module is used to filter the candidate business data based on the query parameters to obtain the target business data; A construction module is used to pre-build the metadata database; The construction module, when used to pre-build the metadata database, is used for: Based on business query requirements, the business data tables in the business database that correspond to the same business query requirement are integrated into an information set; For any information set, based on the reference relationships between various business fields in the business database, obtain the referenced business field corresponding to each business field in the information set.

7. The apparatus according to claim 6, characterized in that, The device further includes: The first caching module is used to cache multiple information sets in the metadata database in advance; The detection module is used to detect whether the target information set is stored in the cache before determining the target information set to which the target business field belongs from the metadata database and obtaining the main business data table corresponding to the target information set. The field acquisition module is further configured to, if the target information set is stored in the cache, retrieve the target information set from the cache and retrieve the main business data table corresponding to the target information set; The field acquisition module is further configured to, if the target information set is not stored in the cache, perform the steps of determining the target information set to which the target business field belongs from the metadata database and obtaining the main business data table corresponding to the target information set.

8. The apparatus according to claim 6, characterized in that, The device further includes an update module, the update module being configured to: In response to an update operation in the business database, based on the target business data table indicated by the update operation, the set of information to be updated to which the target business data table belongs is determined from the metadata database, and the main business data table corresponding to the set of information to be updated is obtained. From the main business data table, obtain the business field to be updated that has undergone an update operation and the business data of the business field to be updated; The business data of the business field to be updated is updated according to the instructions of the update operation.

9. The apparatus according to claim 8, characterized in that, The update module is also used for: Update the cached information set according to the updated business data in the metadata database.

10. The apparatus according to claim 6, characterized in that, The device further includes: The second caching module is used to pre-cacherate all the business data in the business database to each database node in the business database; The query module is used to perform a query in the cache of the database node that receives the data query request before retrieving the business data corresponding to the business field to be queried from the business database. The data acquisition module is further configured to retrieve the business data corresponding to the business field to be queried from the cache if the cache stores the business data corresponding to the business field to be queried. The data acquisition module is further configured to, if the cache does not store the business data corresponding to the business field to be queried, then execute the step of retrieving the business data corresponding to the business field to be queried from the business database.

11. A computing device, characterized in that, The computing device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to perform the operations performed by the data processing method as described in any one of claims 1 to 5.

12. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a program that is executed by a processor as the data processing method described in any one of claims 1 to 5.

Images