Automatic interface data mapping method and device based on data-in-data table

Abstract

The invention provides an interface data automatic mapping method and device based on a data-in-data platform, and the method comprises the steps: collecting the business data of a plurality of business systems through a data source layer; accessing the service data to a data middle table through an ETL tool of a data access layer to obtain an access data field; the data medium table comprises a first field information table, and the first field information table is used for storing existing fields of the data medium table and a plurality of column information values of the existing fields; and based on the existing field in the first field information table, the plurality of column information values of the existing field and a data mapping rule, performing data mapping processing on the access data field through a data processing layer to obtain target system data. The technical problem that data cannot be automatically mapped in the prior art is solved.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a method, device, electronic device and storage medium for automatic mapping of interface data based on a data middle station. Background Art

[0002] The coal mining industry has multiple sources of data, scattered and repeated data, and data quality needs to be improved. It cannot meet the business needs of business process collaboration and synchronization of various business data. With the rapid development of informatization construction, the number of business systems will gradually increase. It is urgent to establish a platform system that unifies data standards, breaks down information barriers, connects data sharing, improves data quality, and enhances the value of data applications.

[0003] Data mapping is the process of mapping data in the real world to a computer system. This process can be achieved through specific software or tools, integrating data from different sources, formats and structures onto the same platform for analysis and processing. The existing data mapping method generally involves negotiation between the data source and the platform to determine a unified standard data interface to achieve data mapping. Due to the increasing number and types of data sources, the workload of data mapping is relatively large and the efficiency is low. Therefore, it is necessary to seek an automated data mapping method to solve the above problems. Summary of the invention

[0004] The present application aims to solve one of the technical problems in the related art at least to some extent.

[0005] To this end, the first purpose of this application is to propose an automatic mapping method for interface data based on a data middle platform to achieve automatic mapping of interface data and solve the problem that automated data mapping cannot be achieved in related technologies.

[0006] The second objective of this application is to propose an automatic interface data mapping device based on a data middle station.

[0007] The third objective of the present application is to provide an electronic device.

[0008] A fourth objective of the present application is to provide a computer-readable storage medium.

[0009] A fifth object of the present application is to provide a computer program product.

[0010] To achieve the above-mentioned purpose, the first embodiment of the present application proposes an automatic mapping method of interface data based on a data middle station, including:

[0011] Collect business data from multiple business systems through the data source layer;

[0012] The business data is connected to the data middle platform through the ETL tool of the data access layer to obtain the access data field; the data middle platform includes a first field information table, the first field information table is used to store the existing fields of the data middle platform and multiple column information values ​​of the existing fields, and the multiple column information values ​​are used to evaluate the mapping hit value of the existing fields;

[0013] Based on the existing fields in the first field information table, multiple column information values ​​of the existing fields and data mapping rules, the access data fields are subjected to data mapping processing by the data processing layer to obtain target system data.

[0014] To achieve the above-mentioned purpose, the second embodiment of the present application proposes an automatic mapping device for interface data based on a data middle station, including:

[0015] The data acquisition module is used to collect business data from multiple business systems through the data source layer;

[0016] A data access module, used to access the business data to the data middle platform through the ETL tool of the data access layer to obtain the access data field; the data middle platform includes a first field information table, the first field information table is used to store the existing fields of the data middle platform and multiple column information values ​​of the existing fields, and the multiple column information values ​​are used to evaluate the mapping hit value of the existing fields;

[0017] The data mapping module is used to perform data mapping processing on the access data field through the data processing layer based on the existing fields in the first field information table, multiple column information values ​​of the existing fields and data mapping rules to obtain target system data.

[0018] To achieve the above-mentioned purpose, the third aspect embodiment of the present application proposes an electronic device, comprising: a processor, and a memory communicatively connected to the processor; the memory stores computer-executable instructions; the processor executes the computer-executable instructions stored in the memory to implement the method described in the first aspect.

[0019] To achieve the above-mentioned purpose, the fourth aspect of the present application proposes a computer-readable storage medium, in which computer-readable storage medium is stored computer-executable instructions, and when the computer-executable instructions are executed by a processor, they are used to implement the method described in the first aspect.

[0020] To achieve the above-mentioned purpose, the fifth aspect of the present application proposes a computer program product, including a computer program, which implements the method described in the first aspect when executed by a processor.

[0021] The present application provides a method, device, electronic device and storage medium for automatic interface data mapping based on a data middle station: after obtaining the access data field, based on the existing fields in the first field information table, multiple column information values ​​of the existing fields and data mapping rules, the access data field is mapped to obtain the target system data, thereby automatically mapping the source system data to the target system data to improve data access efficiency.

[0022] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through the practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0024] Figure 1 A flowchart of an automatic mapping method for interface data based on a data middle station provided in an embodiment of the present application;

[0025] Figure 2 A flowchart of another method for automatically mapping interface data based on a data middle station provided in an embodiment of the present application;

[0026] Figure 3 A block diagram of an automatic interface data mapping device based on a data middle station provided in an embodiment of the present application;

[0027] Figure 4 A block diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0028] Embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present application, and should not be construed as limiting the present application.

[0029] Terminology explanation:

[0030] ETL is the abbreviation of Extract-Transform-Load, which is used to describe the process of extracting (extract), transforming (transform), and loading (load) data from the source end to the destination end.

[0031] The following describes the method, device and equipment for automatic mapping of interface data based on the data middle platform in an embodiment of the present application with reference to the accompanying drawings.

[0032] Figure 1A flowchart of an automatic interface data mapping method based on a data middle station provided in an embodiment of the present application.

[0033] It should be noted that the executor of the automatic interface data mapping method based on the data middle station in the embodiment of the present application is the automatic interface data mapping device based on the data middle station in the embodiment of the present application. The automatic interface data mapping device based on the data middle station can be configured in an electronic device so that the electronic device can perform the automatic interface data mapping function based on the data middle station.

[0034] like Figure 1 As shown, the automatic mapping method of interface data based on the data middle station includes the following steps:

[0035] Step 101: collect business data from multiple business systems through the data source layer.

[0036] It should be noted that the cross-system interface based on the data center can effectively integrate the data of multiple business systems and realize the automatic flow and sharing of data. The systems through which the data interface flows are divided into:

[0037] 1. Source system, the initiator of the data interface service.

[0038] 2. Data middle-office system, the processor of data interface services.

[0039] 3. Target system, the recipient of the data interface service.

[0040] As an implementation method, through interface standardization, the data interface connected to the data middle platform is unified according to the unified API specification, using RESTful API to describe the interface specification, and generate API documents and interface code templates.

[0041] In this embodiment, from the perspective of the data middle platform architecture, the platform processing data interface service of the data middle platform system includes a data source layer, a data access layer, a data processing layer, a data service layer, and an application layer, among which:

[0042] The data source layer is used to collect data sources from multiple business systems, including structured data and unstructured data, such as databases, file systems, etc. This module completes the docking of the interface docking system data source.

[0043] Step 102, connect the business data to the data middle platform through the ETL tool of the data access layer to obtain the access data field; the data middle platform includes a first field information table, the first field information table is used to store the existing fields of the data middle platform and multiple column information values ​​of the existing fields, and the multiple column information values ​​are used to evaluate the mapping hit value of the existing fields.

[0044] In some embodiments, the data access layer is used to connect data from different systems to the data center through ETL tools, solving the problem of gradually converting and processing data warehouses from raw data to report data. It provides a drag-and-drop graphical operation interface, which provides a new hierarchical and visual expression method for traditional script structures, and can more clearly display the context of data processing, truly achieving "zero" coding, eliminating manual coding, reducing the probability of errors, and reducing labor costs. This module specifically completes the extraction, conversion, and loading of interface docking system data.

[0045] The data processing layer is used to clean, convert, map and perform other processing on the accessed data. The data mapping processing includes defining data mapping rules, converting the data formats of different business systems into the standard format of the data middle platform, and translating the source system data into the target system data according to the data mapping rules.

[0046] The data service layer is used to provide a unified data access interface.

[0047] The application layer is used by various business systems to achieve data sharing and interaction through the interface provided by the data center.

[0048] Through the data middle platform, data interaction and sharing across systems and technology systems can be realized, duplication of work can be reduced, and data islands can be connected. Through the reuse of the interactive interface part of the data middle platform, the difficulty of data interaction can be reduced, data standards can be standardized, data quality can be improved, and unified management and control of sensitive data can be strengthened, the problem of inconsistent data formats between different systems can be solved, and automatic data conversion and docking can be realized.

[0049] In some embodiments, data from each business system is accessed to obtain access data fields; including: accessing data from each business system to a data middle platform through an ETL tool to obtain access data fields.

[0050] It should be noted that after the fields are processed by ETL in the data center, some column information values ​​corresponding to the fields will be stored in the data center. These column information values ​​are used in subsequent steps to calculate the correlation between the access data fields and the existing fields in the data center, thereby realizing automatic field mapping. The specific content of the column information values ​​will be described in the following steps.

[0051] Step 103: Based on the existing fields in the first field information table, multiple column information values ​​of the existing fields, and data mapping rules, the data processing layer performs data mapping processing on the access data fields to obtain target system data.

[0052] In some embodiments, based on the existing fields in the first field information table, multiple column information values ​​of the existing fields, and data mapping rules, the access data fields are subjected to data mapping processing by the data processing layer to obtain the target system data, including the following steps:

[0053] Step 201: Acquire, from existing fields in the first field information table, a plurality of preliminarily selected fields whose similarity with the access data field meets a preset similarity threshold.

[0054] In some embodiments, a method for obtaining multiple preliminary selection fields whose similarity with the access data field meets a preset similarity threshold; includes: calculating the similarity between the access data field and each existing field in the first field information table through a distance similarity algorithm, and obtaining multiple preliminary selection fields whose similarity with the access data field meets the preset similarity threshold; obtaining the distance value L between each preliminary selection field in the multiple preliminary selection fields and the access data field.

[0055] Exemplarily, a Levenshtein distance similarity algorithm is used to calculate a plurality of preliminary selected fields similar to the access data field in the existing fields in the first field information table.

[0056] Exemplarily, a preset similarity threshold threshold is set. If the distance obtained by the distance similarity algorithm is less than or equal to the predefined threshold, it is considered that the two current field names are similar, and the distance value is recorded as L.

[0057] The purpose of this step is to quickly match multiple preliminary fields that are similar to the access data fields through similarity judgment, so as to facilitate further screening and obtain the target mapping fields.

[0058] Step 202: For each of the multiple preliminary selection fields, multiple column information parameter values ​​are calculated based on the multiple column information values ​​of the preliminary selection field.

[0059] Multiple column information values ​​are used to evaluate the mapping hit value of existing fields

[0060] In some embodiments, the plurality of column information values ​​include field name, storage time, read amount, read time, modification amount, modification time, manual error correction amount, manual error correction time, and whether it is high value, wherein:

[0061] Field name (name_id), the field name is used to uniquely identify the field, record the field id, which is the unique id of the field in the data center;

[0062] Storage time (time): The storage time is used to record the time when the field is first extracted to the data center;

[0063] Reading amount (read_number): The reading amount is used to record the number of times the data platform reads the field. The default value is 0. Each time it is read, the reading value is +1. This value can only increase but not decrease.

[0064] Read time (read_time), which is used to record the time when the field was last read;

[0065] Modification amount (modify_number): The modification amount is used to record the number of times the data platform modifies the field. The default value is 0. Each time it is read, the reading value is +1. This value can only increase but not decrease.

[0066] Modification time (modify_time), which is used to record the modification time of the field when it was last changed;

[0067] Manual correction amount (correct_number): The manual correction amount is used to record the number of manual interventions on fields with automatic interface mapping errors during the actual interface debugging. The default value is 0. Each time a modification is made, the read value is +1. This value only increases but never decreases.

[0068] Manual correction time (correct_time), which is used to record the indicator modification time when the manual correction amount was changed most recently;

[0069] Is high value (is_high): Is high value is used to identify whether the field is high value data. 0 indicates a low value field, 1 indicates a high value field, and the default value is 0.

[0070] In this embodiment, the multiple column information parameter values ​​include a reading amount parameter y1, a modification amount parameter y2, a manual error correction amount parameter y3, a reading time parameter t1, a modification time parameter t2, a manual error correction time parameter t3, a high value parameter m and a distance parameter s.

[0071] For example, if a similarity field is x, the calculation methods of multiple column information parameter values ​​are as follows:

[0072] Reading parameter y1: 2arccot(read_number(x)) / π;

[0073] Modification parameter y2: 2arctan(modify_number(x)) / π;

[0074] Manual error correction parameter y3: 2arctan(correct_number(x)) / π;

[0075] Reading time parameter t1: 2arccot(read_time(x)-time(x)) / π;

[0076] Modify time parameter t2: 2arctan(modify_time(x)-time(x)) / π;

[0077] Manual error correction time parameter t3: 2arctan(correct_time(x)-time(x)) / π;

[0078] High value parameter m: is_high;

[0079] Distance parameter s: (threshold-L) / threshold.

[0080] Through the above formula, multiple column information parameter values ​​of each preliminary selection field can be obtained, so as to calculate the mapping matching value of each preliminary selection field in the subsequent steps.

[0081] Step 203: Obtain a mapping matching value of the primary selection field based on a plurality of column information parameter values ​​and preset thresholds corresponding to the column information parameter values.

[0082] In some embodiments, the preset thresholds of the above-mentioned multiple column information parameter values ​​are p1, p2, p3, p4, p5, p6, p7, and p8 respectively.

[0083] It should be noted that the preset thresholds of the multiple column information parameter values ​​are set with default values ​​and can be adjusted at any time.

[0084] In some embodiments, based on multiple column information parameter values ​​and preset thresholds corresponding to each column information parameter value, a mapping matching value of the preliminary selection field is obtained; including: obtaining a weighted average of multiple column information parameter values ​​and preset thresholds corresponding to each column information parameter value; and using the weighted average as the mapping matching value of the preliminary selection field.

[0085] That is to say, the mapping matching value of each primary selection field is calculated by the following formula, which is expressed as follows:

[0086] V=(p1*y1+p2*y2+p3*y3+p4*t1+p5*t2+p6*t3+p7*m+p8*L) / (p1+p2+p3+p4+p5+p6+p7+p8)

[0087] Where V is a value between (0,1].

[0088] Therefore, by calculating the V value of each preliminary selection field, the mapping matching value of the preliminary selection field is obtained.

[0089] Step 204: Determine a target mapping field that matches the access data field from the multiple preliminary selection fields based on the mapping matching value corresponding to each of the multiple preliminary selection fields.

[0090] In some implementations, based on the mapping matching value corresponding to each of the multiple preliminary fields, a target mapping field that matches the access data field is determined from the multiple preliminary fields; including: determining the primary field with the largest mapping matching value among the multiple preliminary fields as the target mapping field that matches the access data field.

[0091] It can be understood that the larger the V value of several preliminary selected fields, that is, similar fields, the higher the mapping hit value. Therefore, the field with the largest V value can be used as the target mapping field.

[0092] It should be noted that after obtaining the target mapping field, there is also a manual error correction link. If a mapping error is found, manual error correction can be performed to modify the manual error correction amount to improve the accuracy of data mapping.

[0093] The automatic interface data mapping method based on the data middle station of the embodiment of the present application, after obtaining the access data field, performs data mapping on the access data field based on the existing fields in the first field information table, multiple column information values ​​of the existing fields, and data mapping rules to obtain the target system data, thereby automatically mapping the source system data to the target system data to improve data access efficiency. Furthermore, multiple preliminary selection fields are obtained through the similarity between the fields, and then the mapping hit value of the preliminary selection field is calculated through the multiple column information values ​​of the preliminary selection field, so as to further determine the target mapping field from the multiple preliminary selection fields to improve the accuracy and efficiency of data mapping.

[0094] In order to implement the above embodiments, the present application also proposes an automatic interface data mapping device based on a data middle station. Figure 3 A block diagram of an automatic interface data mapping device based on a data middle station provided in an embodiment of the present application. Figure 3 As shown, the interface data automatic mapping device based on the data middle platform may include: a data acquisition module 301, a data access module 302 and a data mapping module 303.

[0095] The data acquisition module 301 is used to collect business data of multiple business systems through the data source layer;

[0096] The data access module 302 is used to access the business data to the data platform through the ETL tool of the data access layer to obtain the access data field; the data platform includes a first field information table, the first field information table is used to store the existing fields of the data platform and multiple column information values ​​of the existing fields, and the multiple column information values ​​are used to evaluate the mapping hit value of the existing fields;

[0097] The data mapping module 303 is used to perform data mapping processing on the access data field through the data processing layer based on the existing fields in the first field information table, multiple column information values ​​of the existing fields and data mapping rules to obtain target system data.

[0098] In some implementations, the data mapping module 303 is specifically configured to:

[0099] Acquire, from existing fields in the first field information table, a plurality of preliminarily selected fields whose similarity with the access data field meets a preset similarity threshold;

[0100] For each of the plurality of initially selected fields, based on the plurality of column information values ​​of the initially selected field, a plurality of column information parameter values ​​are calculated;

[0101] Based on the multiple column information parameter values ​​and the preset thresholds corresponding to the column information parameter values, a mapping matching value of the primary selection field is obtained;

[0102] Determine, from the plurality of preliminary selection fields, a target mapping field that matches the access data field based on a mapping matching value corresponding to each of the plurality of preliminary selection fields;

[0103] The target mapping field is used as the target system data of the access data field.

[0104] In some implementations, when the data mapping module 303 obtains multiple preliminary selected fields whose similarity with the access data field meets a preset similarity threshold from the existing fields in the first field information table, it is configured to:

[0105] Calculate the similarity between the access data field and each existing field in the first field information table by using a distance similarity algorithm, and obtain a plurality of preliminarily selected fields whose similarity with the access data field meets a preset similarity threshold;

[0106] Obtain a distance value L between each of the multiple preliminary selection fields and the access data field.

[0107] In some implementations, the plurality of column information values ​​include:

[0108] Field name, which is used to uniquely identify the access data field;

[0109] Storage time: The storage time is used to record the time when the access data field is first extracted to the data center;

[0110] Read volume: The read volume is used to record the number of times the data center reads the access data field;

[0111] Reading time: the reading time is used to record the time when the access data field was last read;

[0112] Modification amount, which is used to record the number of times the data center has modified the access data field;

[0113] Modification time: The modification time is used to record the modification time when the access data field was last changed;

[0114] Manual error correction quantity: The manual error correction quantity is used to record the number of manual interventions on the fields with automatic interface mapping errors during the actual interface debugging of the access data field;

[0115] Manual error correction time: Manual error correction time is used to record the indicator modification time when the most recent manual error correction amount was changed;

[0116] Whether it is high value: Whether it is high value is used to identify whether the access data field is high value data.

[0117] In some implementations, when the data mapping module 303 calculates multiple column information parameter values ​​based on multiple column information values ​​of the primary selected field, it is used to:

[0118] Based on the reading amount, the reading amount parameter y1 is obtained by the first formula; the first formula is expressed as follows:

[0119] y1=2arccot(read_number(x)) / π

[0120] Among them, read_number represents the read amount, and x represents the primary selection field;

[0121] Based on the modification amount, the modification amount parameter y2 is obtained by the second formula; the second formula is expressed as follows:

[0122] y2=2arctan(modify_number(x)) / π

[0123] Where modify_number indicates the modification amount, and x indicates the primary selection field;

[0124] Based on the manual error correction amount, the manual error correction amount parameter y3 is obtained by the third formula; the third formula is expressed as follows:

[0125] y3=2arctan(correct_number(x)) / π

[0126] Among them, correct_number represents the amount of manual error correction, and x represents the primary selection field;

[0127] Based on the storage time and the reading time, the reading time parameter t1 is obtained by the fourth formula; the fourth formula is expressed as follows:

[0128] t1=2arccot(read_time(x)-time(x)) / π

[0129] Among them, read_time represents the reading time, time represents the storage time, and x represents the primary selection field;

[0130] Based on the storage time and modification time, the modification time parameter t2 is obtained by the fifth formula; the fifth formula is expressed as follows:

[0131] t2=2arctan(modify_time(x)-time(x)) / π

[0132] Where modify_time indicates the modification time, time indicates the storage time, and x indicates the primary selection field;

[0133] Based on the storage time and the manual error correction time, the manual error correction time parameter t3 is obtained by the sixth formula; the sixth formula is expressed as follows:

[0134] t3=2arctan(correct_time(x)-time(x)) / π

[0135] Where modify_time indicates the manual correction time, time indicates the storage time, and x indicates the primary selection field;

[0136] Based on whether it is high value, the high value parameter m is obtained by the seventh formula; the eighth formula is expressed as follows:

[0137] m=is_high

[0138] Among them, is_high indicates whether it is high value;

[0139] Based on the distance value L, the distance parameter s is obtained by the eighth formula; the eighth formula is expressed as follows:

[0140] s = (threshold - L) / threshold

[0141] Here, threshold represents a preset similarity threshold.

[0142] In some implementations, when the data mapping module 303 obtains the mapping matching value of the primary selected field based on the multiple column information parameter values ​​and the preset thresholds corresponding to the column information parameter values, it is specifically configured to:

[0143] Obtaining a weighted average value of the plurality of column information parameter values ​​and a preset threshold value corresponding to each column information parameter value;

[0144] The weighted average value is used as the mapping matching value of the primary selection field.

[0145] In some implementations, the data mapping module 303, when determining a target mapping field matching the access data field from the multiple preliminary selection fields based on a mapping matching value corresponding to each of the multiple preliminary selection fields, is configured to:

[0146] A primary selection field with the largest mapping matching value among multiple primary selection fields is determined as a target mapping field that matches the access data field.

[0147] It should be noted that the above explanation of the embodiment of the method for automatic mapping of interface data based on the data middle station is also applicable to the automatic mapping device for interface data based on the data middle station of this embodiment, and will not be repeated here.

[0148] In order to implement the above embodiment, the present application also proposes an electronic device. Figure 4 , Figure 4 is a block diagram of an electronic device provided in an embodiment of the present application. Figure 4 As shown, the electronic device 400 includes: a processor 401, and a memory 402 communicatively connected to the processor 401; the memory 402 stores computer-executable instructions; the processor 401 executes the computer-executable instructions stored in the memory to implement the method provided by the aforementioned embodiment.

[0149] In order to implement the above embodiments, the present application also proposes a computer-readable storage medium, in which computer-executable instructions are stored. When the computer-executable instructions are executed by a processor, they are used to implement the methods provided by the above embodiments.

[0150] In order to implement the above embodiments, the present application also proposes a computer program product, including a computer program, which implements the methods provided by the above embodiments when executed by a processor.

[0151] The collection, storage, use, processing, transmission, provision and disclosure of user personal information involved in this application are in compliance with relevant laws and regulations and do not violate public order and good morals.

[0152] It should be noted that personal information from users should be collected for legitimate and reasonable purposes and should not be shared or sold outside of these legitimate uses. In addition, such collection / sharing should be carried out after receiving the user's informed consent, including but not limited to notifying the user to read the user agreement / user notice and sign the agreement / authorization including authorization of relevant user information before the user uses the function. In addition, any necessary steps should be taken to protect and safeguard access to such personal information data and ensure that others who have access to personal information data comply with its privacy policy and procedures.

[0153] The present application is expected to provide an implementation scheme for users to selectively block the use or access of personal information data. That is, the present disclosure is expected to provide hardware and / or software to prevent or block access to such personal information data. Once the personal information data is no longer needed, the risk can be minimized by limiting data collection and deleting the data. In addition, when applicable, such personal information is de-identified to protect the privacy of the user.

[0154] In the description of the aforementioned embodiments, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0155] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In the description of this application, the meaning of "plurality" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0156] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, fragment or portion of code comprising one or more executable instructions for implementing the steps of a custom logical function or process, and the scope of the preferred embodiments of the present application includes alternative implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in the reverse order depending on the functions involved, which should be understood by technicians in the technical field to which the embodiments of the present application belong.

[0157] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute the instructions), or in combination with these instruction execution systems, devices or apparatuses. For the purpose of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in combination with these instruction execution systems, devices or apparatuses. More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or more wires (electronic device), a portable computer disk box (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), a fiber optic device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium and then editing, interpreting or processing in other suitable ways if necessary, and then stored in a computer memory.

[0158] It should be understood that the various parts of the present application can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0159] A person skilled in the art may understand that all or part of the steps in the method for implementing the above-mentioned embodiment may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiment.

[0160] In addition, each functional unit in each embodiment of the present application may be integrated into a processing module, or each unit may exist physically separately, or two or more units may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium.

[0161] The storage medium mentioned above may be a read-only memory, a magnetic disk or an optical disk, etc. Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as limiting the present application. A person of ordinary skill in the art may change, modify, replace and modify the above embodiments within the scope of the present application.

Claims

1. A method for automatic mapping of interface data based on a data middle station, characterized in that: The following steps are involved: Collect business data from multiple business systems through the data source layer; The business data is connected to the data middle platform through the ETL tool of the data access layer to obtain the access data field; the data middle platform includes a first field information table, the first field information table is used to store the existing fields of the data middle platform and multiple column information values ​​of the existing fields, and the multiple column information values ​​are used to evaluate the mapping hit value of the existing fields; Based on the existing fields in the first field information table, multiple column information values ​​of the existing fields and data mapping rules, the access data fields are subjected to data mapping processing by the data processing layer to obtain target system data.

2. The method according to claim 1, characterized in that Based on the existing fields in the first field information table, the multiple column information values ​​of the existing fields, and the data mapping rules, the access data fields are subjected to data mapping processing by the data processing layer to obtain the target system data: Acquire, from existing fields in the first field information table, a plurality of preliminarily selected fields whose similarity with the access data field meets a preset similarity threshold; For each of the plurality of initially selected fields, based on the plurality of column information values ​​of the initially selected field, a plurality of column information parameter values ​​are calculated; Based on the multiple column information parameter values ​​and the preset thresholds corresponding to the column information parameter values, obtaining a mapping matching value of the primary selection field; Determine, from the plurality of preliminary selection fields, a target mapping field that matches the access data field based on a mapping matching value corresponding to each of the plurality of preliminary selection fields; The target mapping field is used as the target system data of the access data field.

3. The method according to claim 2, characterized in that The step of acquiring, from the existing fields in the first field information table, a plurality of preliminarily selected fields whose similarity with the access data field meets a preset similarity threshold, comprises: Calculate the similarity between the access data field and each existing field in the first field information table by using a distance similarity algorithm, and obtain a plurality of preliminarily selected fields whose similarity with the access data field meets a preset similarity threshold; Obtain a distance value L between each of the multiple preliminary selection fields and the access data field.

4. The method according to claim 3, characterized in that: The plurality of column information values ​​include: Field name, which is used to uniquely identify the field; Storage time, which is used to record the time when the field is first extracted to the data center; The reading amount is used to record the number of times the data platform reads the field; Reading time, which is used to record the time when the field was last read; A modification amount, which is used to record the number of times the data center platform modifies the field; Modification time, which is used to record the modification time when the field was last changed; The manual error correction amount is used to record the number of manual interventions on the fields with automatic interface mapping errors during the actual interface debugging; Manual error correction time, which is used to record the indicator modification time when the manual error correction amount was last changed; Whether it is high value, the high value is used to identify whether the field is high value data.

5. The method according to claim 4, characterized in that The calculating of multiple column information parameter values ​​based on the multiple column information values ​​of the initially selected fields includes: Based on the reading amount, the reading amount parameter y1 is obtained by a first formula; the first formula is expressed as follows: y1=2arccot(read_number(x)) / π Among them, read_number represents the read amount, and x represents the primary selection field; Based on the modification amount, the modification amount parameter y2 is obtained by a second formula; the second formula is expressed as follows: y2=2arctan(modify_number(x)) / π Where modify_number indicates the modification amount, and x indicates the primary selection field; Based on the manual error correction amount, the manual error correction amount parameter y3 is obtained by a third formula; the third formula is expressed as follows: y3=2arctan(correct_number(x)) / π Among them, correct_number represents the amount of manual error correction, and x represents the primary selection field; Based on the storage time and the reading time, the reading time parameter t1 is obtained by a fourth formula; the fourth formula is expressed as follows: t1=2arccot(read_time(x)-time(x)) / π Among them, read_time represents the reading time, time represents the storage time, and x represents the primary selection field; Based on the storage time and the modification time, the modification time parameter t2 is obtained by the fifth formula; the fifth formula is expressed as follows: t2=2arctan(modify_time(x)-time(x)) / π Where modify_time indicates the modification time, time indicates the storage time, and x indicates the primary selection field; Based on the storage time and the manual error correction time, the manual error correction time parameter t3 is obtained by the sixth formula; the sixth formula is expressed as follows: t3=2arctan(correct_time(x)-time(x)) / π Where modify_time indicates the manual correction time, time indicates the storage time, and x indicates the primary selection field; Based on whether the value is high, the high value parameter m is obtained by the seventh formula; the seventh formula is expressed as follows: m=is_high Among them, is_high indicates whether it is high value; Based on the distance value L, the distance parameter s is obtained by the eighth formula; the eighth formula is expressed as follows: s = (threshold - L) / threshold Here, threshold represents a preset similarity threshold.

6. The method according to claim 2 or 5, characterized in that: The step of obtaining a mapping matching value of the primary selection field based on the plurality of column information parameter values ​​and a preset threshold value corresponding to each column information parameter value comprises: Obtaining a weighted average value of the plurality of column information parameter values ​​and a preset threshold value corresponding to each column information parameter value; The weighted average value is used as the mapping matching value of the primary selection field.

7. The method according to claim 1, characterized in that The step of determining a target mapping field matching the access data field from the multiple preliminary selection fields based on a mapping matching value corresponding to each of the multiple preliminary selection fields comprises: A preliminary selection field having the largest mapping matching value among the multiple preliminary selection fields is determined as a target mapping field that matches the access data field.

8. An automatic interface data mapping device based on a data middle station, characterized in that: include: The data acquisition module is used to collect business data from multiple business systems through the data source layer; A data access module, used to access the business data to the data middle platform through the ETL tool of the data access layer to obtain the access data field; the data middle platform includes a first field information table, the first field information table is used to store the existing fields of the data middle platform and multiple column information values ​​of the existing fields, and the multiple column information values ​​are used to evaluate the mapping hit value of the existing fields; The data mapping module is used to perform data mapping processing on the access data field through the data processing layer based on the existing fields in the first field information table, multiple column information values ​​of the existing fields and data mapping rules to obtain target system data.

9. An electronic device, characterized in that: include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory to implement the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, which are used to implement the method according to any one of claims 1 to 7 when executed by a processor.

Images