Data access method, electronic device, and computer-readable storage medium

Abstract

The application discloses a data access method, an electronic device and a computer readable storage medium, comprising: obtaining a plurality of first meta information corresponding to a plurality of source table fields, a plurality of second meta information corresponding to a plurality of target table fields and a plurality of configuration mapping information through a configuration file provided by a configuration interface; the configuration mapping information comprises a preset mapping formula between a preset source table field and a preset target table field; detecting the plurality of configuration mapping information based on the plurality of first meta information and the plurality of second meta information, and generating corresponding actual mapping information for each target table field based on the detection result; the actual mapping information is the preset mapping formula in the corresponding configuration mapping information or is empty; and data of the source table field is accessed into the target table field according to the actual mapping information corresponding to each target table field. The application can improve the compatibility of the configuration file to a plurality of physical models, thereby maximizing the data access demand of the actual business scenario.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to data access methods, electronic devices, and computer-readable storage media. Background Technology

[0002] With the rapid development of big data technology, big data analysis of various data by accessing data from different products is an important aspect of big data technology.

[0003] In existing technologies, the upstream data source information, data access object information, and data access indicator information required in the data access phase are usually obtained based on configuration files, and then the configuration information is converted to generate a standard data access task.

[0004] However, the configuration files in the above methods are heavily dependent on upstream data access products. A single configuration file can only be used for the same upstream access source (physical model) and is not compatible with other versions of the physical model. Once the upstream physical model is updated, users need to update and maintain the configuration files, which increases the maintenance cost of the configuration files and reduces the efficiency of data access, thus failing to meet the data access needs of actual business scenarios. Summary of the Invention

[0005] The main technical problem addressed by this application is to provide a data access method, electronic device, and computer-readable storage medium that can solve the problem of the inability to efficiently access data using configuration files in the prior art.

[0006] To address the aforementioned technical problems, the first technical solution adopted in this application is to provide a data access method, comprising: obtaining multiple first-level information corresponding to multiple source table fields, multiple second-level information corresponding to multiple target table fields, and multiple configuration mapping information through a configuration file provided by a configuration interface; wherein, the configuration mapping information includes a preset mapping formula between preset source table fields and preset target table fields; detecting the multiple configuration mapping information based on the multiple first-level information and multiple second-level information, and generating corresponding actual mapping information for each target table field based on the detection results; wherein, the actual mapping information is the preset mapping formula in the corresponding configuration mapping information or the mapping is empty; and accessing the data of the source table fields into the target table fields according to the actual mapping information corresponding to each target table field.

[0007] The step of detecting multiple configuration mapping information based on multiple first metadata and multiple second metadata, and generating corresponding actual mapping information for each target table field based on the detection results, includes: matching the second metadata corresponding to each target table field with the metadata of the corresponding preset target table field in the multiple configuration mapping information to determine whether there is configuration mapping information corresponding to each target table field in the multiple configuration mapping information; in response to the existence of corresponding configuration mapping information for the target table field, matching the metadata of the preset source table field in the corresponding configuration mapping information with multiple first metadata, and setting the actual mapping information corresponding to the target table field to the preset mapping formula in the configuration mapping information or mapping it to empty based on the corresponding matching results; in response to the absence of corresponding configuration mapping information for the target table field, setting the actual mapping information corresponding to the target table field to empty.

[0008] The step of matching the second metadata corresponding to each target table field with the metadata of the corresponding preset target table fields in multiple configuration mapping information to determine whether there is configuration mapping information corresponding to each target table field in the multiple configuration mapping information includes: in response to the existence of metadata of a preset target table field that matches the second metadata in a certain configuration mapping information, determining that the target table field has corresponding configuration mapping information; in response to the existence of metadata of a preset target table field that matches the second metadata in any configuration mapping information, determining that the target table field does not have configuration mapping information.

[0009] The steps of matching the metadata of the preset source table field in the corresponding configuration mapping information with multiple first metadata, and setting the actual mapping information corresponding to the target table field to the preset mapping formula in the configuration mapping information or mapping it to empty based on the matching results, include: in response to the existence of first metadata that matches the metadata of the preset source table field in the corresponding configuration mapping information, setting the actual mapping information corresponding to the target table field to the preset mapping formula in the corresponding configuration mapping information; and in response to the absence of first metadata that matches the metadata of the preset source table field in the corresponding configuration mapping information, setting the actual mapping information corresponding to the target table field to empty.

[0010] The step of detecting multiple configuration mapping information based on multiple first meta-information and multiple second meta-information, and generating corresponding actual mapping information for each target table field based on the detection results, further includes: deleting the configuration mapping information in response to the absence of second meta-information matching the meta-information of the preset source table field in the configuration mapping information.

[0011] The step of connecting the data of the source table field to the target table field according to the actual mapping information corresponding to each target table field includes: in response to the actual mapping information corresponding to the target table field being a preset mapping formula in the corresponding configuration mapping information, calculating the matched first meta information according to the preset mapping formula, and connecting the calculation result to the target table field; in response to the actual mapping information corresponding to the target table field being empty, setting the target table field to an empty value.

[0012] The steps of obtaining multiple first-level information corresponding to multiple source table fields, multiple second-level information corresponding to multiple target table fields, and multiple configuration mapping information through the configuration file provided by the configuration interface include: obtaining the configuration file through the configuration interface; parsing the configuration file to obtain configuration parameters; wherein the configuration parameters include the source database, the target database, and multiple configuration mapping information; and obtaining multiple first-level information corresponding to the source table fields and multiple second-level information corresponding to the target table fields from the source database and the target database.

[0013] The source database and target database include various types of databases. The steps for obtaining multiple first-level information corresponding to the source table fields and multiple second-level information corresponding to the target table fields from the source database and target database include: obtaining the database types corresponding to the source database and target database respectively; calling the corresponding operation instructions from the field retrieval function based on different database types, and obtaining the corresponding database interface according to the operation instructions; wherein the field retrieval function includes multiple operation instructions, each operation instruction providing a database interface; and using the corresponding database interface to obtain multiple first-level information corresponding to the source table fields and multiple second-level information corresponding to the target table fields from the source database and target database.

[0014] To solve the above-mentioned technical problems, the second technical solution adopted in this application is to provide an electronic device, including: a memory for storing program data, wherein when the program data is executed, the steps in the data access method described above are implemented; and a processor for executing the program data stored in the memory to implement the steps in the data access method described above.

[0015] To solve the above-mentioned technical problems, the third technical solution adopted in this application is to provide a computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a processor, it implements the steps in the data access method described above.

[0016] The beneficial effects of this application are as follows: Unlike existing technologies, this application provides a data access method, electronic device, and computer-readable storage medium. It obtains first metadata corresponding to source table fields, second metadata corresponding to target table fields, and configuration mapping information through a configuration file. Furthermore, it performs detection on the configuration mapping information based on the first and second metadata, and generates corresponding actual mapping information for each target table field based on the detection results. This allows for the verification and supplementation of the configuration mapping information provided in the configuration file using metadata obtained from the upstream data access end and the downstream target end. This enables the same configuration file to adapt to different versions of physical models, thereby reducing the maintenance cost of the configuration file and improving the efficiency of data access. Through the above method, this application improves the compatibility of the configuration file with multiple physical models, maximizing the satisfaction of data access needs in actual business scenarios. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is a flowchart illustrating the first embodiment of the data access method of this application;

[0019] Figure 2 This is a flowchart illustrating the second embodiment of the data access method of this application;

[0020] Figure 3 This is a flowchart illustrating the workflow of this application for detecting the metadata of the configuration mapping information and the actual acquired physical model;

[0021] Figure 4 This is a flowchart illustrating the third embodiment of the data access method of this application;

[0022] Figure 5 This is a schematic diagram of one embodiment of the data access device of this application;

[0023] Figure 6 This is a schematic diagram of the structure of one embodiment of the electronic device of this application;

[0024] Figure 7 This is a schematic diagram of a computer-readable storage medium according to an embodiment of the present invention. Detailed Implementation

[0025] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0026] The terminology used in the embodiments of this application is for the purpose of describing particular embodiments only and is not intended to limit the application. The singular forms “a,” “said,” and “the” used in the embodiments of this application and the appended claims are also intended to include the plural forms, unless otherwise clearly indicated above. “Multiple” generally includes at least two, but does not exclude the inclusion of at least one.

[0027] It should be understood that the term "and / or" used in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Additionally, the character " / " in this article generally indicates that the preceding and following related objects have an "or" relationship.

[0028] It should be understood that the terms "comprising," "including," or any other variations used herein are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0029] Please see Figure 1 , Figure 1 This is a flowchart illustrating the first embodiment of the data access method of this application. In this embodiment, the data access method includes:

[0030] S11: Obtain multiple first-level information corresponding to multiple source table fields, multiple second-level information corresponding to multiple target table fields, and multiple configuration mapping information through the configuration file provided by the configuration interface; among them, the configuration mapping information includes the preset mapping formula between preset source table fields and preset target table fields.

[0031] In this embodiment, the configuration interface is the software's visual interface.

[0032] In this embodiment, the configuration file includes relevant configuration parameters for data access, such as data source information (names / naming of the source and target databases), data object information (source and target tables), and configuration mapping information.

[0033] The configuration mapping information includes preset mapping formulas between preset source table fields and preset target table fields.

[0034] The first metadata refers to the metadata describing the fields of the source table, while the second metadata refers to the metadata describing the fields of the target table. Specifically, metadata mainly refers to information describing data properties, used to support functions such as indicating storage location, historical data, resource lookup, and file records; it is data used to describe data.

[0035] In a specific implementation scenario, the configuration file data structure is as follows:

[0036]

[0037] In this context, "source" refers to the data source; "target" refers to the target; "table" refers to the data table; "dataSource" refers to the database source; "source_hive" refers to the source database; "target_hive" refers to the target database; "stg_xxx" refers to the source table; "ods_xxx" refers to the target table; "mapping" refers to the mapping configuration; "column_A" refers to the field named A (the corresponding column in the data table); "column_B" refers to the field named B; "column_C" refers to the field named C; "col" refers to the column abbreviation; "exp" refers to the mapping formula; "column_a" refers to the metadata of column_A; and "column_b" refers to the metadata of column_B.

[0038] Understandably, column_A, column_B, and column_C refer to fields that exist in both the preset source table and the preset target table. The `col` and `exp` values ​​corresponding to column_A, column_B, and column_C respectively refer to the preset mapping formulas. During data access, the data corresponding to the source table fields needs to be processed according to the preset mapping formulas before being accessed into the target table fields.

[0039] In this embodiment, by using the data source information and data object information provided in the configuration file, the corresponding database interface can be used to obtain multiple first-level information corresponding to multiple source table fields and multiple second-level information corresponding to multiple target table fields from the corresponding database.

[0040] In a specific implementation scenario, multiple first-level information items corresponding to the source table (stg_xxx) and multiple second-level information items corresponding to the target table (ods_xxx) are obtained through the configuration parameters provided in the configuration file mentioned above:

[0041] stg_xxx

[0042] Column_A Column a field a Column_B column b field b Column_B Column c field c Column_D column d field d

[0043] ods_xxx

[0044] Column_A Column a field a Column_B column b field b Column_C Column c field c Column_D Column d field d

[0045] Here, column_A, column_B, column_C, and column_D refer to the names of the fields; column a, column b, column c, and column d refer to the data types in the fields; and field a, field b, field c, and field d refer to the descriptive information of the fields.

[0046] As can be seen from the metadata obtained from the source and target tables above, there is no mapping relationship between Column_D in the source and target tables in the mapping of the configuration file, that is, the configuration file does not maintain the newly added metadata.

[0047] Understandably, if the fields at the upstream data access end and the downstream target end remain unchanged, the preset source table fields and preset target table fields in the configuration mapping information are equivalent to the fields in the source table and the target table, respectively. If the fields at the upstream data access end and the downstream target end change, but the configuration file has not been updated, the preset source table fields and preset target table fields in the configuration mapping information will not be completely consistent with the fields in the source table and the target table. This will result in a situation where the actual first metadata and second metadata obtained are not completely consistent with the metadata of the preset source table fields and preset target table fields in the configuration mapping information. In this case, if access is performed solely according to the configuration mapping information, data access will fail.

[0048] Based on this, this implementation method will verify and supplement the configuration mapping information provided in the configuration file when performing data access.

[0049] S12: Detect multiple configuration mapping information based on multiple first-level information and multiple second-level information, and generate corresponding actual mapping information for each target table field based on the detection results; wherein, the actual mapping information is the preset mapping formula in the corresponding configuration mapping information or the mapping is empty.

[0050] In this embodiment, the configuration mapping information is checked based on the second metadata of each target table field. If the second metadata does not match the metadata of the preset target table field in the configuration mapping relationship, it indicates that the target table field corresponding to the second metadata does not have configuration mapping information in the configuration file. In this case, the actual mapping relationship corresponding to the target table field is set to empty to ensure the data access task succeeds. If the second metadata matches the metadata of the preset target table field in the configuration mapping relationship, it indicates that the target table field corresponding to the second metadata has configuration mapping information in the configuration file. In this case, further detection is required.

[0051] Furthermore, it is determined whether the metadata of the preset source table fields in the configuration mapping information exists among the multiple first metadata obtained, that is, whether the metadata in the preset mapping formula actually exists in the queried source table. If it exists, the configuration mapping relationship can be implemented, and this configuration mapping relationship is the actual mapping information of the target table field. Data can be accessed using the preset mapping formula in this configuration mapping relationship. If it does not exist, it indicates that the source table on the upstream data end has not been upgraded, and the metadata of the preset source table fields does not exist in the source table. In this case, the actual mapping relationship corresponding to the target table field is set to null to ensure that the data access task is successful.

[0052] To more clearly illustrate how this implementation generates corresponding actual mapping information for each target table field, the configuration mapping relationship provided in the above configuration file will be used as an example.

[0053] In a specific implementation scenario, if multiple target table fields corresponding to multiple second-level metadata all have configuration mapping information in the configuration file, and the metadata of the preset source table fields in each configuration mapping information exists in the multiple first-level metadata, then the data structure of the actual mapping relationship between the multiple target table fields is as follows: [

[0055] {"column_A":"column_a"}

[0056] ,{"column_B":"column_b"}

[0057] ,{"column_C":"column_a+column_b"} ]

[0059] Understandably, the actual mapping information corresponding to the three target table fields column_A, column_B, and column_C is completely consistent with the configuration mapping information provided in the configuration file.

[0060] In another specific implementation scenario, if multiple target table fields corresponding to multiple second-level metadata all have configuration mapping information in the configuration file, but the metadata of a certain preset source table field in the configuration mapping information does not exist in the multiple first-level metadata, then the data structure of the actual mapping relationship corresponding to the multiple target table fields is as follows: [

[0062] {"column_A": "null"}

[0063] ,{"column_B":"column_b"}

[0064] ,{"column_C":"null"} ]

[0066] Here, null represents an empty value.

[0067] Understandably, when the metadata column_a corresponding to the target table field column_A does not exist in the multiple first metadata entries of the source table, all configuration mapping information containing column_a is replaced with null values.

[0068] S13: Connect the data from the source table fields to the target table fields based on the actual mapping information corresponding to each target table field.

[0069] In this embodiment, after generating corresponding actual mapping information for each target table field based on the detection results, each piece of actual mapping information is combined with the data source information and data object information provided in the configuration file to generate a new data access text. Based on the data access text, a unified access function is called to access the data of the source table field into the target table field using the actual mapping information corresponding to each target table field, thereby realizing data access.

[0070] In a specific implementation scenario, in response to the actual mapping information corresponding to the target table field, which is the preset mapping formula in the corresponding configuration mapping information, the first element information of the matching is calculated according to the preset mapping formula, and the calculation result is entered into the target table field.

[0071] In another specific implementation scenario, in response to the actual mapping information corresponding to the target table field being empty, the target table field is set to an empty value.

[0072] Unlike existing technologies, this implementation obtains the first metadata corresponding to the source table fields, the second metadata corresponding to the target table fields, and the configuration mapping information through a configuration file. It then performs detection on the configuration mapping information based on the first and second metadata and generates corresponding actual mapping information for each target table field based on the detection results. This allows for the verification and supplementation of the configuration mapping information provided in the configuration file using metadata obtained from the upstream data access end and the downstream target end. This enables the same configuration file to adapt to different versions of physical models, thereby reducing the maintenance cost of the configuration file and improving data access efficiency. Furthermore, this implementation improves the compatibility of the configuration file with multiple physical models, maximizing the satisfaction of data access needs in actual business scenarios.

[0073] Please see Figure 2 , Figure 2 This is a flowchart illustrating the second embodiment of the data access method of this application. In this embodiment, the data access method includes:

[0074] S21: Obtain multiple first-level information corresponding to multiple source table fields, multiple second-level information corresponding to multiple target table fields, and multiple configuration mapping information through the configuration file provided by the configuration interface; wherein, the configuration mapping information includes the preset mapping formula between the preset source table fields and the preset target table fields.

[0075] Please refer to the description in S11 for the specific process, which will not be repeated here.

[0076] S22: Match the second metadata corresponding to each target table field with the metadata of the corresponding preset target table fields in the multiple configuration mapping information to determine whether there is configuration mapping information corresponding to each target table field in the multiple configuration mapping information.

[0077] In a specific implementation scenario, in response to the existence of metadata in a certain configuration mapping information that matches the second metadata, it is determined that the target table field has corresponding configuration mapping information.

[0078] In another specific implementation scenario, in response to the absence of metadata in any configuration mapping information that matches the second metadata, it is determined that the target table field does not contain configuration mapping information.

[0079] S23: In response to the existence of corresponding configuration mapping information for the target table field, the metadata of the preset source table field in the corresponding configuration mapping information is matched with multiple first metadata, and the actual mapping information corresponding to the target table field is set to the preset mapping formula in the configuration mapping information or mapped to empty based on the corresponding matching result.

[0080] In a specific implementation scenario, in response to the existence of a first meta-information among multiple first meta-information that matches the meta-information of a preset source table field in the corresponding configuration mapping information, the actual mapping information corresponding to the target table field is set to the preset mapping formula in the corresponding configuration mapping information.

[0081] In another specific implementation scenario, in response to the absence of a first meta-information among multiple first meta-information that matches the meta-information of the preset source table field in the corresponding configuration mapping information, the actual mapping information corresponding to the target table field is set to empty.

[0082] Understandably, if none of the first metadata obtained from the source table matches the metadata of the preset source table field in the corresponding configuration mapping information, it indicates that the source table on the upstream data side has not been upgraded. The source table is equivalent to a low-version physical model, and the configuration mapping information in the configuration file is equivalent to high-version data. Replacing the configuration mapping information with empty data can enable the high-version configuration file to adapt to the low-version physical model and business content, thus ensuring that the data access task will not fail.

[0083] S24: In response to the absence of corresponding configuration mapping information for the target table field, set the actual mapping information corresponding to the target table field to null.

[0084] Understandably, if the metadata of the second metadata does not match the metadata of the preset target table field in the configuration mapping relationship, it indicates that the target table field corresponding to the second metadata does not have configuration mapping information in the configuration file. That is, the configuration file does not maintain the metadata of the newly added target table field. The target table is equivalent to a higher version of the physical model, and the configuration mapping information in the configuration file is equivalent to lower version data. Replacing the actual mapping information of the newly added field with empty data can enable the lower version configuration file to adapt to the higher version physical model and business content, so as to ensure that the data access task will not fail.

[0085] The above method is based on the detection of multiple second-order information of the target table fields to generate corresponding actual mapping information for each actual target table field.

[0086] In some implementations, when querying multiple sets of second metadata based on the metadata of a preset target table field in the configuration mapping information, there may be cases where the metadata of a certain preset target table field does not exist in the multiple sets of second metadata. That is, there is no second metadata in the multiple sets of second metadata corresponding to the target table that matches the metadata of the preset source table field in the configuration mapping information. This indicates that the metadata of the preset source table field does not exist in the actual target table, and the configuration mapping information corresponding to the metadata of the preset target table field is directly deleted.

[0087] The configuration mapping relationship provided by the configuration file in the first embodiment will be used for explanation.

[0088] In a specific implementation scenario, if multiple target table fields corresponding to multiple second-level metadata all have configuration mapping information in the configuration file, but the metadata of a certain preset target table field in the configuration mapping information does not exist in the multiple second-level metadata, then the data structure of the actual mapping relationship corresponding to the multiple target table fields is as follows: [

[0090] {"column_B":"column_b"}

[0091] ,{"column_C":"null"} ]

[0093] Understandably, since the metadata column_a of the target table field column_A does not exist in the multiple second metadata of the target table, only the configuration mapping information of column_a is deleted, and the configuration mapping information of column_a and column_a is replaced with empty, in order to avoid the data access task from failing.

[0094] S25: Connect the data from the source table fields to the target table fields based on the actual mapping information corresponding to each target table field.

[0095] Please refer to the description in S13 for the specific process, which will not be repeated here.

[0096] In this embodiment, the acquired first element information, second element information, and configuration mapping information are compared in a loop using an encapsulation function to determine the final actual mapping information.

[0097] Specifically, please refer to Figure 3 , Figure 3This is a flowchart illustrating the workflow of this application for detecting the configuration mapping information against the metadata of the actually acquired physical model. In this embodiment, firstly, multiple first metadata from the source table and multiple second metadata from the target table are obtained using the data source information and data object information provided in the configuration file. If metadata cannot be successfully obtained, an exception is displayed, indicating that the data source information or data object information in the configuration file may be incorrect. After successfully obtaining the metadata, to improve the query efficiency of the first metadata, it can be converted into a dictionary. Next, the number of fields in the target table is obtained, and the index is initialized to 0. It is determined whether the index is less than the number of fields. If it is, the configuration mapping information is detected based on the second metadata of the target table fields using a wrapper function to determine whether there is matching configuration mapping information for the target table field. If the second metadata does not match the metadata of the preset target table field in the configuration mapping relationship, it indicates that the target table field corresponding to the second metadata does not have configuration mapping information in the configuration file, and the actual mapping relationship corresponding to the target table field is set to null. If the second metadata matches the metadata of a preset target table field in the configuration mapping relationship, it indicates that the target table field corresponding to the second metadata has configuration mapping information in the configuration file, and the next step of detection is performed. It is determined whether any of the multiple first metadata obtained from the source table matches the metadata of the preset source table field in the configuration mapping information. If it does, the actual mapping information corresponding to the target table field is set to the preset mapping formula in the configuration mapping information. If it does not, the actual mapping information corresponding to the target table field is set to empty. After completing one round of the above mapping judgment using the encapsulation function, the index is incremented by one, and the next loop is entered to assemble the actual mapping information of the remaining target table fields. Once the number of tables equals the number of fields, the actual mapping information corresponding to each target table field is obtained, and the loop ends.

[0098] Understandably, by combining the configuration mapping information (the preset mapping formula between the preset source table fields and the preset target table fields) with the physical fields (source table fields and target table fields) in the actual acquired physical model (source table and target table), the initial configuration mapping provided in the configuration file can be verified and supplemented, thereby ensuring that the overall task will not fail and improving the efficiency of data access.

[0099] Please see Figure 4 , Figure 4 This is a flowchart illustrating the third embodiment of the data access method of this application. In this embodiment, the data access method includes:

[0100] S41: Obtain the configuration file through the configuration interface.

[0101] In this embodiment, the configuration file includes the mapping relationship between the source table and the target table specified by the user.

[0102] The configuration file includes relevant configuration parameters for data access, such as data source information (names / naming of the source and target databases), data object information (source and target tables), and configuration mapping information.

[0103] In some implementations, sensitive information such as data source type, account, and password can be configured in other files, while the data object information only exposes the data source command for secure isolation. This application does not limit this.

[0104] S42: Parse the configuration file to obtain configuration parameters; the configuration parameters include the source database, the target database, and multiple configuration mapping information.

[0105] S43: Obtain multiple first-level information corresponding to the fields in the source table and multiple second-level information corresponding to the fields in the target table from the source database and the target database.

[0106] In this implementation, the source and target databases include various types of databases, such as Oracle, Hive, and MySQL, requiring different JDBC (Java Database Connectivity) methods depending on the database type.

[0107] Furthermore, based on different database types, the corresponding operation instructions are invoked from the field retrieval function, and the corresponding database interface is obtained according to the operation instructions. The field retrieval function includes multiple operation instructions, each providing a database interface.

[0108] The function for retrieving fields is `get_table_columns`, and the command for operation is `desctable`. `get_table_columns` is a function encapsulated by the software based on a common function, while `desc` is a command-line interface within the database, which can be understood as an interface provided by the database.

[0109] Furthermore, the corresponding database interfaces are used to obtain multiple first-level information items corresponding to the source table fields and multiple second-level information items corresponding to the target table fields from the source and target databases. Specifically, `get_table_columns` can obtain the corresponding JDBC packages from different types of databases based on the `desctable` command, and obtain the column name lists corresponding to the source and target tables from the JDBC packages. See Section S11 for the multiple first-level and multiple second-level information items obtained based on the physical models `stg_xxx` and `ods_xxx`.

[0110] In some implementations, multiple first metadata of the source table and multiple second metadata of the target table can also be obtained by reading the metadata of the data. The corresponding reading method can be called for different data types. This application does not limit this.

[0111] S44: Detect multiple configuration mapping information based on multiple first-level information and multiple second-level information, and generate corresponding actual mapping information for each target table field based on the detection results; wherein, the actual mapping information is the preset mapping formula in the corresponding configuration mapping information or the mapping is empty.

[0112] For details, please refer to the descriptions in S12 and S22 to S24, which will not be repeated here.

[0113] S45: Connect the data from the source table fields to the target table fields based on the actual mapping information corresponding to each target table field.

[0114] Please refer to the description in S13 for the specific process, which will not be repeated here.

[0115] Understandably, this implementation method uses desc naming to detect metadata in different types of databases, which can be compatible with different database types, thereby further improving the efficiency of data access.

[0116] Correspondingly, this application provides a data access device.

[0117] Please see Figure 5 , Figure 5 This is a schematic diagram of one embodiment of the data access device of this application. For example... Figure 5 As shown, the data access device 50 includes an acquisition module 51, a mapping generation module 52, and a data access module 53.

[0118] The acquisition module 51 is used to obtain multiple first-level information corresponding to multiple source table fields, multiple second-level information corresponding to multiple target table fields, and multiple configuration mapping information through the configuration file provided by the configuration interface; wherein, the configuration mapping information includes a preset mapping formula between preset source table fields and preset target table fields.

[0119] The mapping generation module 52 is used to detect multiple configuration mapping information based on multiple first-level information and multiple second-level information, and generate corresponding actual mapping information for each target table field based on the detection results; wherein, the actual mapping information is the preset mapping formula in the corresponding configuration mapping information or the mapping is empty.

[0120] The data access module 53 is used to access the data of the source table fields into the target table fields according to the actual mapping information corresponding to each target table field.

[0121] For details of the process, please refer to the relevant textual descriptions in S11-S13, S21-S25, and S41-S45, which will not be repeated here.

[0122] Unlike existing technologies, this implementation method obtains the first metadata corresponding to the source table fields, the second metadata corresponding to the target table fields, and the configuration mapping information based on the configuration file through the acquisition module 51. The mapping generation module 52 then detects the configuration mapping information based on the first and second metadata and generates corresponding actual mapping information for each target table field based on the detection results. This allows for the verification and supplementation of the configuration mapping information provided in the configuration file using metadata obtained from the upstream data access end and the downstream target end. This enables the same configuration file to adapt to different versions of the physical model, thereby reducing the maintenance cost of the configuration file and improving the efficiency of data access. Furthermore, this implementation method improves the compatibility of the configuration file with multiple physical models, maximizing the satisfaction of data access needs in actual business scenarios.

[0123] Correspondingly, this application provides an electronic device.

[0124] Please see Figure 6 , Figure 6 This is a schematic diagram of one embodiment of the electronic device of this application. For example... Figure 6 As shown, the electronic device 60 includes a memory 61 and a processor 62.

[0125] In this embodiment, the memory 61 is used to store program data, and when the program data is executed, it implements the steps in the data access method described above; the processor 62 is used to execute the program instructions stored in the memory 61 to implement the steps in the data access method described above.

[0126] Specifically, processor 62 controls itself and memory 61 to implement the steps in the data access method described above. Processor 62 can also be referred to as a CPU (Central Processing Unit). Processor 62 may be an integrated circuit chip with signal processing capabilities. Processor 62 can also be a general-purpose processor, digital signal processor (DSP), application-specific integrated circuit (ASIC), field-programmable gate array (FPGA), or other programmable logic device, discrete gate or transistor logic device, or discrete hardware component. A general-purpose processor can be a microprocessor or any conventional processor. Furthermore, processor 62 can be implemented using multiple integrated circuit chips.

[0127] Unlike existing technologies, this implementation uses processor 62 to obtain first metadata corresponding to source table fields, second metadata corresponding to target table fields, and configuration mapping information based on a configuration file. It then performs detection on the configuration mapping information based on the first and second metadata and generates corresponding actual mapping information for each target table field based on the detection results. This allows for the verification and supplementation of the configuration mapping information provided in the configuration file using metadata obtained from upstream data access and downstream target ends. This enables the same configuration file to adapt to different versions of physical models, thereby reducing the maintenance cost of the configuration file and improving data access efficiency. Furthermore, this implementation improves the compatibility of the configuration file with multiple physical models, maximizing the satisfaction of data access needs in actual business scenarios.

[0128] Correspondingly, this application provides a computer-readable storage medium.

[0129] Please see Figure 7 , Figure 7 This is a schematic diagram of a computer-readable storage medium according to an embodiment of the present invention.

[0130] The computer-readable storage medium 70 includes a computer program 701 stored on it. When executed by the processor, the computer program 701 implements the steps of the data access method described above. Specifically, if the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium 70. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a computer-readable storage medium 70 and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the methods of various embodiments of this application. The aforementioned computer-readable storage medium 70 includes various media capable of storing program code, such as a USB flash drive, a portable hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0131] In the several embodiments provided in this application, it should be understood that the disclosed methods and apparatus can be implemented in other ways. For example, the apparatus implementations described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0132] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment, depending on actual needs.

[0133] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0134] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the methods of various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0135] The above description is merely an embodiment of this application and does not limit the patent scope of this application. Any equivalent structural or procedural transformations made using the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.

Claims

1. A data access method, characterized in that, include: The configuration interface provides a configuration file that allows you to obtain multiple first-level information corresponding to multiple source table fields, multiple second-level information corresponding to multiple target table fields, and multiple configuration mapping information. The configuration mapping information includes a preset mapping formula between preset source table fields and preset target table fields. Multiple configuration mapping information is detected based on multiple first metadata and multiple second metadata, and corresponding actual mapping information is generated for each target table field based on the detection results; wherein, the actual mapping information is the preset mapping formula in the corresponding configuration mapping information or the mapping is empty; wherein, in response to the existence of corresponding configuration mapping information for the target table field, and the existence of first metadata that matches the metadata of the preset source table field in the corresponding configuration mapping information, the actual mapping information corresponding to the target table field is set to the preset mapping formula in the corresponding configuration mapping information; in response to the absence of first metadata that matches the metadata of the preset source table field in the corresponding configuration mapping information, or in response to the absence of corresponding configuration mapping information for the target table field, the actual mapping information corresponding to the target table field is set to empty, so that the same configuration file can adapt to different versions of the physical model; The data from the source table fields is accessed into the target table fields based on the actual mapping information corresponding to each target table field.

2. The data access method according to claim 1, characterized in that, The step of detecting multiple configuration mapping information based on multiple first metadata and multiple second metadata, and generating corresponding actual mapping information for each target table field based on the detection results, includes: The second metadata corresponding to each target table field is matched with the metadata of the preset target table field in the plurality of configuration mapping information to determine whether there is configuration mapping information corresponding to each target table field in the plurality of configuration mapping information; In response to the existence of corresponding configuration mapping information for the target table field, the meta-information of the preset source table field in the corresponding configuration mapping information is matched with multiple pieces of the first meta-information, and based on the corresponding matching result, the actual mapping information corresponding to the target table field is set to the preset mapping formula in the configuration mapping information or the mapping is empty; In response to the absence of corresponding configuration mapping information for the target table field, the actual mapping information corresponding to the target table field is set to empty.

3. The data access method according to claim 2, characterized in that, The step of matching the second metadata corresponding to each of the target table fields with the metadata of the corresponding preset target table fields in the plurality of configuration mapping information to determine whether there is configuration mapping information in the plurality of configuration mapping information that corresponds to each of the target table fields includes: In response to the existence of metadata in a preset target table field that matches the second metadata in a certain configuration mapping information, it is determined that the target table field has corresponding configuration mapping information; If no metadata matching the second metadata exists in any of the configuration mapping information, it is determined that the target table field does not exist in the configuration mapping information.

4. The data access method according to claim 3, characterized in that, The step of detecting multiple configuration mapping information based on multiple first metadata and multiple second metadata, and generating corresponding actual mapping information for each target table field based on the detection results, further includes: If none of the second metadata matches the metadata of the preset source table field in the configuration mapping information, the configuration mapping information is deleted.

5. The data access method according to claim 4, characterized in that, The step of accessing the data of the source table field into the target table field according to the actual mapping information corresponding to each target table field includes: In response to the actual mapping information corresponding to the target table field being the preset mapping formula in the corresponding configuration mapping information, the first meta information that is matched is calculated according to the preset mapping formula, and the calculation result is entered into the target table field; In response to the actual mapping information corresponding to the target table field being empty, the target table field is set to an empty value.

6. The data access method according to claim 1, characterized in that, The steps of obtaining multiple first-level information corresponding to multiple source table fields, multiple second-level information corresponding to multiple target table fields, and multiple configuration mapping information through the configuration file provided by the configuration interface include: The configuration file can be obtained through the configuration interface; The configuration file is parsed to obtain configuration parameters; wherein, the configuration parameters include a source database, a target database, and multiple configuration mapping information; Multiple first metadata corresponding to the source table fields and multiple second metadata corresponding to the target table fields are obtained from the source database and the target database.

7. The data access method according to claim 6, characterized in that, The source database and the target database include various types of databases; The step of obtaining multiple first metadata corresponding to the source table fields and multiple second metadata corresponding to the target table fields from the source database and the target database includes: The database types corresponding to the source database and the target database are obtained respectively; Based on different database types, the corresponding operation instructions are called from the field retrieval function, and the corresponding database interface is obtained according to the operation instructions; wherein, the field retrieval function includes multiple operation instructions, and each operation instruction provides a database interface; Using the corresponding database interface, multiple first metadata corresponding to the source table fields and multiple second metadata corresponding to the target table fields are obtained from the source database and the target database.

8. An electronic device, characterized in that, include: A memory for storing program data, which, when executed, implements the steps of the data access method as described in any one of claims 1 to 7; A processor is configured to execute the program data stored in the memory to implement the steps in the data access method as described in any one of claims 1 to 7.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the data access method as described in any one of claims 1 to 7.

Images