Data mapping method and related products

Abstract

The application discloses a data mapping method and related products, which are applied to the technical field of data mapping. The data mapping method comprises the following steps: acquiring field values of a plurality of first source fields of one or more target data sources, and writing the field values of the plurality of first source fields into a plurality of first general fields of a general source table according to a first mapping rule, so that efficient data mapping between the target data source and the general source table can be realized. Further, a plurality of target fields of a target system are acquired, and a plurality of second general fields respectively matched with the plurality of target fields are determined from the plurality of first general fields according to the similarity between the semantic of each target field and the preset semantic of the plurality of first general fields. Further, the field values of the first source fields stored in each second general field are written into the matched target field, so that efficient data mapping between the target data source and the target system can be realized. The above method is beneficial to improving the data mapping efficiency.

Description

Technical Field

[0001] This application relates to the technical field of data mapping, and in particular to a data mapping method and related products. Background Technology

[0002] In enterprise IT infrastructure development, the need for data migration between source and target systems is becoming increasingly common. Traditional data migration techniques utilize Extract-Transform-Load (ETL) tools to predefine mapping rules for fields between the source and target systems, writing data from the source system's data source into the target system. For example, if the source system's data source includes 50 source fields and the target system includes 30 target fields, 1500 analyses are needed to predefine 30 mapping rules to write the data from the source system into the target system. If there are many fields between the source and target systems, the workload of configuring mapping rules increases significantly. Data mapping based on predefined mapping rules is inefficient. Summary of the Invention

[0003] Based on this, this application addresses the aforementioned technical problems by providing a data mapping method and related products to improve data mapping efficiency. The related products include data mapping apparatus, computer equipment, and computer-readable storage media. Alternatively, the related products may also include computer program products containing computer program instructions.

[0004] Firstly, this application provides a data mapping method, which can be implemented by a server running a computer program. The data mapping method includes: the server obtaining field values ​​of multiple first source fields from one or more target data sources; and writing the field values ​​of the multiple first source fields into multiple first general fields according to a first mapping rule. The multiple first general fields are at least a portion of multiple general fields in a general source table, and the first mapping rule is a mapping relationship between the multiple first source fields, their semantic labels, and the multiple first general fields in the general source table. It also includes: the server obtaining multiple target fields from a target system; determining multiple second general fields from the multiple first general fields that match the multiple target fields based on the similarity between the semantics of each target field and the preset semantics of the multiple first general fields; and writing the field value of each second general field into the matching target field. The preset semantics of the first general fields are determined according to the semantic labels mapped to the first general fields in the first mapping rule. The target data source includes any data source of the source system to which data mapping is required, such as an Excel spreadsheet or message queue of the source system. The first source field is the source field of the target data source, such as any header field in an Excel spreadsheet, like customer ID or order date. The general source table is a standardized data carrier, including multiple general fields (e.g., field1 to field200) and metadata storage fields. The general fields store the field values ​​of the source fields from the data source, while the metadata storage fields establish the mapping rules between the data source and the general source table, and / or between the general source table and the target system. During data mapping, after obtaining the field values ​​of multiple first source fields from one or more target data sources, the server writes these values ​​into multiple first general fields according to pre-established first mapping rules. These multiple first general fields are at least a portion of the multiple general fields in the general source table. In the first mapping rules, there is a one-to-one mapping between the first source fields, their semantic tags, and the first general fields. This allows for the automatic and efficient mapping of field values ​​from the target data source to the general source table. Furthermore, after obtaining multiple target fields from the target system, the server automatically determines multiple second general fields from multiple first general fields with preset semantics in the general source table that match the multiple target fields, based on the similarity between the semantics of the target fields and the semantics of the first general fields. That is, it determines the second general field that matches each target field in the target system. It should be understood that a target field matching a second general field based on semantic similarity means that the target field also matches the first source field mapped to the second general field.Based on this, the field values ​​of the first source field in each second general field are written to the matching target field, i.e., written to the target system. This enables the automatic and efficient mapping of field values ​​from the target data source stored in the general source table to the target system, achieving efficient data mapping between the target data source and the target system. Executing this data mapping method eliminates the need for manual configuration of mapping rules between each first source field of the target data source and each target field of the target system, effectively avoiding one-to-one hard-coded mapping between the first source fields of the target data source and the target fields of the target system, thus improving data mapping efficiency.

[0005] In one feasible implementation of the first aspect, the server determines multiple second general fields that match the multiple target fields from the multiple first general fields based on the similarity between the semantics of each target field and the preset semantics of the multiple first general fields. This includes: the server calculating the similarity between the semantics of each target field and the preset semantics of the multiple first general fields to obtain multiple semantic similarities for each target field; if there is a first similarity among the multiple semantic similarities of the target field that is greater than or equal to a first threshold, then based on the first similarity, the server determines the second general field that matches the target field from the multiple first general fields; if all the multiple semantic similarities of the target field are less than the first threshold, then the server receives a field writing instruction to determine the second general field that matches the target field. The field writing instruction includes the following information: the correspondence between the target field and the matching second general field. For each target field among the multiple target fields, multiple semantic similarities of the target field can be obtained by calculating the similarity between the semantics of the target field and the preset semantics of the multiple first general fields. Furthermore, based on the multiple semantic similarities, the server can determine the second general field that matches the target field from the multiple first general fields. Specifically, if among multiple semantic similarities, there exists a first similarity greater than or equal to a first threshold, the server determines a second general field matching the target field based on the first similarity. A first similarity greater than or equal to the first threshold indicates that the similarity in business meaning between the first general field corresponding to the first similarity and the target field meets a preset requirement, allowing direct data mapping between these two fields. For example, if multiple first similarities are greater than or equal to the first threshold, the first general field matching the target field is determined from the multiple first general fields corresponding to the multiple first similarities. If multiple semantic similarities are all less than the first threshold, it indicates that the similarity in business meaning between the multiple first general fields of the general source table and the target field does not meet the preset requirement. In this case, the server receives a field write instruction and determines the second general field matching the target field based on the instruction. For example, the field write instruction is input by the user through a graphical user interface. Using this data mapping method, the server first automatically determines the second general field matching the target field by ensuring semantic similarity meets the standard. If none of the semantic similarities meet the standard, the server receives a field write instruction and determines the second general field matching the target field based on the instruction. By combining semantic matching and instruction indication, it is possible to determine the second general field that matches each target field of the target system from multiple first general fields, which is beneficial for efficiently mapping the field value of the second general field to the target system.

[0006] In one feasible implementation of the first aspect, the server writes the field value of each second general field to a matching target field, including: the server validating each piece of data to be written according to target rules to obtain corresponding target data, wherein the data to be written is the field value of the second general field that matches the target field; and writing each piece of target data to the matching target field. The target rules include at least one of the following: data format conditions, business rules of the industries associated with multiple target fields, a first mapping rule, and a matching relationship between multiple target fields and the second general field. The data format conditions refer to the format conditions that the data written to the target system must meet. During the process of writing the matching target field, the server first uses the field value of the second general field that matches each target field as the data to be written, performs preset processing on each piece of data to be written to obtain the corresponding target data, and then writes each piece of target data to the matching target field. The preset processing involves validating the data to be written according to the target rules, and the validation includes verification and correction. The target rules include at least one of the following: data format requirements, business rules of the industries associated with multiple target fields, a first mapping rule, and a matching relationship between multiple target fields and the second general field. For example, data format requirements include: date format must be "yyyy-MM-dd", target data length must be within a preset range, target data structure must be a preset structure, vehicle unit must be "unit", and business serial number must be completed, etc. Data validation based on the target system's data format requirements ensures that the target data written to the target system conforms to its data format requirements, improving the accuracy and standardization of data mapping. Industry business rules associated with the target fields, such as order quantity must be greater than or equal to 1 and vehicle type must be within an enumerated value, also contribute to data validation. Validating the target data based on these industry rules reduces errors and improves accuracy. The matching relationship between the first mapping rule, multiple target fields, and the second general field can be used to verify the consistency of data mapping between the target data source and the target system, ensuring that the target data written to the target system matches the field values ​​of the target data source.

[0007] In one feasible implementation of the first aspect, the data mapping method further includes: the server writing mapping information between one or more target data sources and the target system into the target system. The mapping information includes at least one of the following: a first mapping rule, and a matching relationship between multiple target fields and multiple second general fields. In addition to writing the field values ​​of the second general fields into the target system to achieve efficient data mapping between the target data source and the target system, this data mapping method also writes the mapping information between the target data source and the target system into the target system. This enables the target system to perform data reverse lookups, trace back the original data and mapping process in the general source table or the target data source, and achieve traceable writing of field values, which is beneficial for meeting auditing and rule rollback requirements.

[0008] In one feasible implementation of the first aspect, before obtaining the field values ​​of multiple first source fields from one or more target data sources, the data mapping method further includes: the server obtaining multiple first source fields from one or more target data sources; determining semantic labels for the multiple first source fields based on the multiple first source fields, the semantic labels indicating the meaning of the corresponding first source fields in the business context; and determining a first mapping rule based on the multiple first source fields, the semantic labels of the multiple first source fields, and multiple first general fields of a general source table. In this data mapping method, by obtaining multiple first source fields from one or more target data sources and determining the semantic labels of the multiple first source fields, a first mapping rule can be determined. The first mapping rule is the mapping relationship between the multiple first source fields, the semantic labels of the multiple first source fields, and the multiple first general fields of the general source table. On the one hand, establishing a first mapping rule facilitates the efficient writing of the field values ​​of the multiple first source fields into the multiple first general fields, and facilitates the efficient mapping of the field values ​​of the target data sources to the general source table. On the other hand, the semantic label of the first source field is not only used to indicate the meaning of the corresponding first source field in the business, but also provides a preset semantic for the corresponding first general field. Establishing the first mapping rule is also beneficial to determine the second general field that matches the target field of the target system from multiple first general fields based on the semantic similarity between fields.

[0009] In one feasible implementation of the first aspect, after determining the first mapping rule based on multiple first source fields, the semantic tags of the multiple first source fields, and multiple first general fields of the general source table, the data mapping method further includes: the server recording the first mapping rule in the metadata dictionary of the general source table. The metadata dictionary is a centralized rule base and specification established for the general source table, which serves as an intermediate carrier. It systematically defines the business meaning (preset semantics), mapping rules, etc., of the general fields in the general source table and can be used to interpret the data stored in the general source table. Recording the first mapping rule in the metadata dictionary can be used to explain the source of the field values ​​stored in the multiple first general fields of the general source table, as well as the source of the preset semantics of the multiple first general fields, facilitating subsequent backtracking of the data mapping process.

[0010] In one feasible implementation of the first aspect, multiple adapters run on the server. These adapters are used for data interaction between various types of data sources and a common source table, with each adapter corresponding to one type of data source. The server obtains multiple first source fields from one or more target data sources, including: obtaining multiple first source fields from one or more target data sources through a target adapter that matches the type of the one or more target data sources. The target adapter is one of the multiple adapters. The server determines the semantic labels of the multiple first source fields based on the multiple first source fields, including: determining the semantic labels of the multiple first source fields based on the multiple first source fields through the target adapter. Here, the adapter is used for data interaction (data access) between the data source and the common source table. Each adapter corresponds to a type of data source; for example, an API access adapter matches the API data source type; a message queue access adapter matches the message queue (Kafka) data source type; and a file access adapter matches the file data source type. For different types of data sources, different adapters can be used to access the field values ​​from each data source into the common source table. This data mapping method can achieve unified data access from multiple types of data sources. For example, for one or more target data sources, multiple first source fields from one or more target data sources are obtained and their semantic labels are determined through a target adapter that matches the type of one or more target data sources.

[0011] In one feasible implementation of the first aspect, obtaining multiple first source fields from one or more target data sources includes: obtaining the field names and field descriptions of the multiple first source fields from one or more target data sources. The field description is information used to describe the meaning of the corresponding first source field in the business context. Determining semantic labels for the multiple first source fields based on the multiple first source fields includes: determining semantic labels for the multiple first source fields based on the field names and field descriptions. Here, the field description refers to information describing the meaning of the first source field in the business context; it can be understood as a supplementary explanation of the meaning of the first source field based on the field name. The server or a target adapter running on the server can determine semantic labels used to indicate the meaning of the first source fields in the business context based on the field names and field descriptions of the first source fields. For example, the field name is "Time," and the field description is "order completion timestamp in the business context." The semantic label for the first source field "Time" is determined to be "Order|Time" based on the field name and field description, thus the determined semantic label can more accurately indicate the business meaning of the first source field.

[0012] In one feasible implementation of the first aspect, the server writes the field values ​​of multiple first source fields into multiple first general fields according to a first mapping rule, including: the server generating a data mapping script according to the first mapping rule; and executing the data mapping script. The data mapping script is used to write the field values ​​of multiple first source fields into multiple first general fields. The server generates the data mapping script for writing the field values ​​of multiple first source fields into multiple first general fields according to the first mapping rule, which includes the mapping relationship between multiple first source fields and multiple first general fields. Then, by executing the data mapping script, it is possible to write the field values ​​of each first source field into the corresponding first general field, thereby realizing the writing of the field values ​​of the target data source into the general source table.

[0013] In one feasible implementation of the first aspect, after writing the field values ​​of multiple first source fields into multiple first general fields according to the first mapping rule, the data mapping method further includes: the server correcting the field values ​​in the multiple first general fields according to a preset industry data template. The semantics of the template fields in the preset industry data template are associated with the preset semantics of the first general fields, and the preset industry data template is used to provide a data representation format. The preset industry data template can be understood as a predefined representation template for data in common business scenarios of various industries. For example, a representation template for automotive industry-related data, such as car brands and models. During the generation of the target data source, the representation format of the field values ​​in the target data source may be diverse. For example, the representation format of certain data within the industry is the full name, but the target data source only records the abbreviation of this data. The semantics of the template fields in the preset industry data template are associated with the preset semantics of the first general fields in the general source table. The semantics of the template fields are related to the business scenario, and the preset semantics of the first general fields are determined based on semantic tags with mapping relationships, which also refer to the business-level meaning. The semantic association between the two indicates that the business scenario involved in the preset industry data template is related to the business to which the field values ​​of the multiple first general fields belong. Based on the preset industry data template, the field values ​​of multiple first general fields in the general source table are corrected, which can realize the unification of data expression format and business standardization, and help reduce misunderstandings.

[0014] In one feasible implementation of the first aspect, the data mapping method further includes: if a second source field is added to one or more target data sources, then a second mapping rule is determined. The second mapping rule is obtained by adding the mapping relationship between the second source field, the semantic label of the second source field, and a third general field, based on the first mapping rule. The third general field is a general field other than the first general field. If a second source field is added to one or more target data sources, the mapping rule between the one or more target data sources and the general source table needs to be updated. Specifically, the mapping relationship and other information of the second source field are automatically added to the original first mapping rule to obtain the latest version of the second mapping rule. This data mapping method does not require reconfiguring the mapping rules of the second source field and each of the first source fields in one or more target data sources; it only needs to update the changed parts, thereby improving the efficiency of updating the mapping rules. Furthermore, this data mapping method only needs to dynamically adjust based on the determined first mapping rule to obtain the latest second mapping rule, resulting in fast maintenance of the mapping rules, low maintenance cost, and reduced risk of errors when updating the mapping rules.

[0015] In one feasible implementation of the first aspect, the data mapping method further includes: if a third source field is deleted from one or more target data sources, then a second mapping rule is determined. The third source field is one of multiple first source fields, and the second mapping rule is obtained by marking the mapping relationship between the third source field, its semantic label, and the corresponding first general field as invalid, based on the first mapping rule. If a third source field is deleted from multiple first source fields in one or more target data sources, the mapping rule between the one or more target data sources and the general source table needs to be updated. Specifically, the existing mapping relationship and other information of the third source field are automatically marked as invalid based on the original first mapping rule to obtain the latest version of the second mapping rule. This data mapping method does not require reconfiguring the mapping rules of each first source field in one or more target data sources; it only needs to update the changed parts, thereby improving the efficiency of updating the mapping rules. Furthermore, this data mapping method only needs to dynamically adjust based on the determined first mapping rule to obtain the latest second mapping rule, resulting in fast maintenance of the mapping rules, low maintenance cost, and reduced risk of errors when updating the mapping rules.

[0016] In one feasible implementation of the first aspect, after determining the second mapping rule, the data mapping method further includes: the server recording the second mapping rule in the metadata dictionary of the general source table. The metadata dictionary also records the first mapping rule, and the second mapping rule is marked as being in use. When the second mapping rule is determined, the metadata dictionary records at least both the first and second mapping rules. Upon determining and recording the second mapping rule, it automatically takes effect and is marked as being in use, indicating that the latest second mapping rule is the currently used version, and the field values ​​of the target data source in the general source table are stored based on the second mapping rule. The server's recording of different versions of the mapping rules in the metadata dictionary and marking the currently used version facilitates users or the system in more conveniently viewing different versions of the mapping rules, tracing the data mapping process, and detecting mapping conflicts.

[0017] Secondly, this application also provides a data mapping apparatus for implementing the method described in the first aspect or any feasible embodiment of the first aspect. For example, the data mapping apparatus includes: a data source acquisition module for acquiring field values ​​of multiple first source fields from one or more target data sources; a field value writing module for writing the field values ​​of the multiple first source fields into multiple first general fields according to a first mapping rule; the multiple first general fields are at least a portion of multiple general fields in a general source table; the first mapping rule is a mapping relationship between the multiple first source fields, the semantic tags of the multiple first source fields, and the multiple first general fields of the general source table; an information acquisition module for acquiring multiple target fields of a target system; a field matching module for determining multiple second general fields that match the multiple target fields from the multiple first general fields based on the similarity between the semantics of each target field and the preset semantics of the multiple first general fields; the preset semantics of the first general fields are determined according to the semantic tags mapped to the first general fields in the first mapping rule; and a data writing module for writing the field value of each second general field into the matching target field.

[0018] Thirdly, this application also provides a computer device, including a processor and a memory, wherein the memory is used to store computer program instructions, and the processor is used to invoke the computer program instructions stored in the memory, so that the computer device implements the method described in the first aspect or any feasible implementation of the first aspect.

[0019] Fourthly, this application also provides a computer-readable storage medium for storing computer program instructions that, when executed by a processor, cause an apparatus including a processor to implement the method described in the first aspect or any feasible embodiment of the first aspect.

[0020] Fifthly, this application also provides a computer program product including computer program instructions, which, when executed by a processor, cause a device including a processor to implement the method described in the first aspect or any feasible implementation thereof. Attached Figure Description

[0021] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0022] Figure 1 Hardware architecture diagram of the data mapping method provided in the embodiments of this application; Figure 2 This is a flowchart illustrating the data mapping method provided in an embodiment of this application; Figure 3 This is a schematic diagram of the process for determining the first mapping rule provided in an embodiment of this application; Figure 4 A flowchart illustrating the process of determining a second general field that matches each target field, as provided in an embodiment of this application; Figure 5 This is a flowchart illustrating the process of writing the field value of each second general field to the matching target field, as provided in an embodiment of this application. Figure 6 This is a schematic diagram of the structure of the data mapping device provided in the embodiments of this application; Figure 7 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application.

[0023] Explanation of reference numerals in the attached figures: 10—Data mapping device; 101—Data source acquisition module; 102—Field value writing module; 103—Information acquisition module; 104—Field matching module; 105 - Data writing module; 20—Computer equipment; 201—Processor; 202—Memory. Detailed Implementation

[0024] To facilitate understanding of the embodiments of this application, a more comprehensive description of the embodiments of this application will be provided below with reference to the accompanying drawings. Preferred embodiments of the embodiments of this application are shown in the drawings. However, the embodiments of this application can be implemented in many different forms and are not limited to the embodiments described herein. Rather, these embodiments are provided so that the disclosure of the embodiments of this application will be more thorough and complete.

[0025] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which embodiments of this application belong. The terminology used herein in the description of embodiments of this application is for the purpose of describing particular embodiments only and is not intended to be limiting of the embodiments of this application.

[0026] The terms “first,” “second,” etc., used in this document are used to distinguish different objects, not to describe a specific order. The term “includes / contains” specifies the presence of the stated feature, whole, step, operation, part, or combination thereof, but does not exclude the possibility of the presence or addition of one or more other features, wholes, steps, operations, parts, or combinations thereof.

[0027] Example 1 This application provides a data mapping method that can improve data mapping efficiency. Before discussing the data mapping method in detail, the execution subject and hardware architecture of this method will be introduced first.

[0028] Exemplarily, the data mapping method provided in this application embodiment is implemented by a server running a computer program. The computer program includes, but is not limited to, application software deployed on the server's operating system. Exemplarily, this application embodiment refers to the computer program as a "data middleware". In this application embodiment, the server executing the data mapping method is also referred to as a data mapping device or computer equipment. The server described in this application embodiment is a single server or a server cluster.

[0029] For example, please see Figure 1 , Figure 1 This is a hardware architecture diagram of the data mapping method provided in an embodiment of this application. Figure 1As shown, the server includes application software, operating system, and other software, as well as hardware such as a central processing unit (CPU), memory, display screen, and other peripherals. The data middle platform is the application software deployed on the server. The server can execute the data mapping method in this application embodiment by running the data middle platform. The data mapping method described in this application embodiment is used to realize data mapping between the data source in the source system and the target system. Specifically, the data middle platform includes a software module for a general source table. The server maps the data from the data source in the source system to the general source table in the server, and maps the data from the data source stored in the general source table to the target system according to the needs of the target system, thereby realizing data mapping between the data source and the target system. Here, the source system refers to the system where the data is originally generated or stored; it is the starting point of the data flow. Examples of source systems include Enterprise Resource Planning (ERP), Customer Relationship Management (CRM), and Supply Chain Management (SCM). There can be more than one source system. The target system is the system where the data ultimately needs to be written; it is the endpoint of the data flow. Examples of target systems include cloud data warehouses (Snowflake) and full-text search engines (Elasticsearch). There can be more than one target system. A source system includes one or more data sources, such as message queues (e.g., Apache Kafka), files (e.g., Excel, databases, logs), etc. A data source includes multiple source fields, and the data in the data source refers to the field values ​​of these multiple source fields. This application embodiment uses the implementation of data mapping between a source system's data source and a target system as an example. The server belonging to the source system can be the server executing this data mapping method, or it can be different from the server executing this data mapping method. The same applies to the server belonging to the target system. This application embodiment uses an example where both the source system's server and the target system's server are different from the server executing this data mapping method.

[0030] Please see Figure 2 , Figure 2 This is a flowchart illustrating the data mapping method provided in an embodiment of this application. Figure 2 As shown, this data mapping method includes the following steps S100 to S500. This data mapping method is applied to, for example, Figure 1 On the hardware architecture shown.

[0031] S100: Retrieves the field values ​​of multiple first source fields from one or more target data sources.

[0032] Here, one or more target data sources refer to the data sources that need to be mapped, such as: Excel spreadsheets or message queues in the source system. The first source field refers to the source fields of one or more target data sources, such as the header of an Excel spreadsheet, like customer number or order date.

[0033] In some cases, the method for obtaining the field value corresponding to each first source field is as follows: the server receives target parsing documents from one or more target data sources, extracts interface information and the first source fields from the target parsing documents, and then obtains the field value corresponding to the first source field based on the interface information. The target parsing document can be considered a specification for one or more target data sources, used to parse these data sources. For example, for API-type data sources that need to be obtained through an Application Programming Interface (API), the target parsing document is a Swagger document. The server extracts the API interface information and the first source fields from the Swagger document, and then calls the specific API interface to obtain the field value of the first source field.

[0034] In other cases, the method for obtaining the field value corresponding to each first source field is for the server to obtain one or more target data sources, which include multiple first source fields and their values. For example, for an Excel data source, the server can extract the field values ​​of the first source fields from the cells of the Excel file upon receiving the Excel file.

[0035] In some cases, the method for obtaining the field value corresponding to each first source field is as follows: the server reads the first source field from the service components of one or more target data sources, obtains one or more target data sources, and then extracts the field value of the first source field from the one or more target data sources based on the first source field. For example, for a message queue type data source, the server connects to the Apache Kafka message service component, such as SchemaRegistry, and reads the first source field from the service component, then extracts the field value of the first source field from the message content based on the first source field.

[0036] In one feasible implementation, multiple adapters run on the server, serving as data exchange points between various data sources and a common source table. Each adapter corresponds to one type of data source. The adapters exist on the server as software programs or components. For example, an API access adapter matches an API-type data source; a message queue access adapter matches a message queue (Kafka)-type data source; and a file access adapter matches a file-type data source. The server retrieves the field values ​​of multiple first source fields from one or more target data sources by using the target adapter that matches the type of the target data source. The target adapter is one of multiple adapters.

[0037] In enterprise IT infrastructure development, the need for data migration between source and target systems is increasingly common, as is the need for integrating heterogeneous data from multiple sources. This includes integrating and migrating data from different data sources to the target system. Heterogeneous data from multiple sources refers to data with different data structures from various data sources. Traditional data migration techniques involve developing dedicated data access modules for different data sources to map data between them and the target system, achieving the integration of data with different structures. For example, when e-commerce platform A integrates data from supplier systems, due to differences in data structure among the various supplier systems, platform A develops a dedicated data access module for each system. Then, by predefined mapping rules, data mapping between supplier systems is achieved. However, traditional methods lack a unified access module for diverse data sources. Developing dedicated data access modules suffers from high access costs, long integration cycles, and inconsistent data structures.

[0038] In this data mapping method, multiple adapters are run on the server for data interaction between various types of data sources and a common source table. This can be understood as building standardized access components on the server, including API access adapters, message queue access adapters, and file access adapters, serving as a unified access module for diverse data sources. Based on the type of one or more target data sources, a matching target adapter is determined from the standardized access components. Then, data interaction between one or more target data sources and the common source table is performed through the target adapter, such as retrieving the first source field and its value. This data mapping method, through adapters matched to the data source type—that is, through standardized access components—enables unified access to different types of data sources, achieving data integration across multiple data sources. This avoids developing dedicated data access modules for different types of data sources, resulting in lower access costs and shorter integration cycles.

[0039] In some cases, the server obtains the field values ​​of the first source field from one or more target data sources through a target adapter, and then converts the data structure of the field values ​​into a target structure using the target adapter before executing subsequent steps of this data mapping method. Here, the target structure refers to the data structure when writing the field values ​​of one or more target data sources into a general source table. For example, the target structure is the data structure required by the general source table for data storage. When storing data in a general source table, the data structure should be uniform. In this case, an adapter is used to convert the data structure to achieve unified storage of heterogeneous data from different types of data sources. For example, if the target structure required for storing data in a general source table is in Excel format, the server uses an API access adapter to convert the data from an API-type data source from JSON format to Excel format.

[0040] Different types of data sources may have different data structures. This data mapping method not only enables unified access to data from multiple data sources, but also transforms the data structures of different data sources from their initial, differentiated structures into a unified target structure, thereby achieving the integration of multi-source heterogeneous data. For example, the data structures of different types of primary and secondary data sources can be transformed into the target structure before data integration. By using a common source table as an intermediate carrier, unified storage and processing of multi-source heterogeneous data can be achieved, which also helps to decouple the differences in data structures between the data source and the target system.

[0041] S200, according to the first mapping rule, write the field values ​​of multiple first source fields into multiple first general fields.

[0042] Among them, the multiple first general fields are at least some of the multiple general fields of the general source table; the first mapping rule is the mapping relationship between the multiple first source fields, the semantic labels of the multiple first source fields and the multiple first general fields of the general source table.

[0043] The general source table is a standardized data carrier, including general fields and metadata storage fields. In this embodiment, the general fields are used to store data from the data source, specifically the field values ​​of the source fields. The metadata storage fields are used to link the mapping rules between the data source and the general source table. For example, the general source table in this embodiment has the following characteristics: First, the general source table includes 200 general fields, such as field1 to field200, to integrate and store large-scale data. Optionally, the number of general fields can be expanded to 500 or more to adapt to more complex industrial IoT data integration scenarios. The type of the general fields is uniformly VARCHAR (4000) to achieve compatibility with multiple data types. Second, the general source table adopts a partitioned table design, supporting physical partitioning storage according to the type of data source to reduce data query overhead and improve data query efficiency. The data sources can be categorized into at least three types: API-type data sources (e.g., those obtained via the RepresentationalStateTransferApplicationProgrammingInterface (RESTAPI)); message queues (e.g., Apache Kafka); and files (e.g., Excel, databases, logs). Different data source types have different data structures. For example, API data sources might use JSON format; message queue data sources might use Avro format; and file data sources might use Excel format. The general source table also supports unified storage and management of data from multiple data sources with different structures. Furthermore, the metadata dictionary records the semantic tags (hereinafter referred to as "semantic tags") of each source field in the data source stored in the general source table, enabling functions such as field querying and field matching. The metadata dictionary is a centralized rule base and specification for the general source table, which acts as an intermediary. It systematically defines the business meaning (preset semantics) and mapping rules of the general fields in the general source table, and can be used to interpret the data stored in the general source table. The semantic tags of the source fields indicate their business meaning. Finally, the universal source table supports data storage from multilingual data sources to adapt to data mapping scenarios involving complex languages.

[0044] In one feasible implementation, please refer to Figure 3 , Figure 3This is a flowchart illustrating the process of determining the first mapping rule provided in an embodiment of this application. Before obtaining the field values ​​of multiple first source fields from one or more target data sources, the server first determines the first mapping rule between one or more target data sources and a general source table through the following steps S210 to S230. Based on the determination of the first mapping rule, the field values ​​of the first source fields are then obtained, and the field values ​​of the first source fields are written into the first general field according to the first mapping rule.

[0045] S210 retrieves multiple first source fields from one or more target data sources.

[0046] The server can obtain each first source field in the same way as described above for obtaining the field value of each first source field. Optionally, it can extract the first source field from the target parsed document of one or more target data sources, extract it directly from one or more target data sources, or read it from the service component of one or more target data sources. These methods will not be elaborated further here. It should be understood that if the server has already obtained the first source field, it does not need to repeatedly obtain the first source field when obtaining its field value.

[0047] In some cases, retrieving multiple first-source fields from one or more target data sources includes retrieving the field names and field descriptions of multiple first-source fields from one or more target data sources. The field description is information used to describe the meaning of the corresponding first-source field in the business context.

[0048] Specifically, when the server obtains each first source field using the above method, it reads or extracts the field name and field description of the first source field. For example, the field name is "Time," and the field description is "Order completion timestamp." Thus, the server obtains the "Time" first source field from one or more target data sources, which refers to the order completion timestamp.

[0049] In some cases, the server obtains multiple first source fields from one or more target data sources through the target adapter to achieve unified access to data from multiple data sources.

[0050] S220, Based on multiple first source fields, determine the semantic labels for the multiple first source fields. The semantic labels are used to indicate the meaning of the corresponding first source field in the business context.

[0051] Semantic tags not only embody the meaning of the first source field but also serve as structured tags describing its business meaning. Semantic tags define the business dimensions of the first source field through hierarchical naming. For example, a semantic tag for the first source field might be "customer|name," indicating that the first source field means not only "name" but also "customer's name." Another example is "order|customer|name," where the semantic tag further adds the system module level "order," indicating that the first source field means "customer's name" within the order system module. Determining the semantic tags for the first source field accurately describes its business meaning, thus providing a precise semantic description. For instance, knowledge graph technology can be introduced into the semantic tagging system to enhance the semantic understanding of cross-industry data, using more precise semantic tags to describe the semantics of the first source field.

[0052] In some cases, semantic labels for multiple first source fields are determined based on multiple first source fields, including determining semantic labels for multiple first source fields based on their field names and field descriptions.

[0053] As mentioned earlier, the server retrieves the field names and descriptions of multiple primary source fields to achieve the acquisition of multiple primary source fields. The field description refers to information that describes the meaning of the primary source field in the business context; it can be understood as a supplementary explanation of the meaning of the primary source field based on its name. Based on this, the server determines the semantic label of the primary source field according to its field name and description. For example, if the field name is "Time" and the field description is "order completion timestamp in the business context," the semantic label of the primary source field determined based on its field name and description is "order|time." This determined semantic label more accurately indicates the business meaning of the primary source field.

[0054] In some cases, the server determines the semantic labels of multiple first source fields through the target adapter.

[0055] S230, determine the first mapping rule based on multiple first source fields, the semantic labels of multiple first source fields, and multiple first general fields of the general source table.

[0056] The server determines multiple first general fields, the same number as multiple first source fields, and establishes a mapping relationship between the multiple first source fields, their semantic labels, and the multiple first general fields, thus obtaining a first mapping rule. In some cases, the server determines multiple first general fields without stored field values ​​according to the numbering order of the multiple general fields in the general source table or randomly. In some cases, among at least one general field in the general source table that already stores the field values ​​of the source fields, there exists a general field whose preset semantic similarity to the semantic label of the target first source field meets a preset requirement; this general field that meets the requirement is then determined as the first general field mapped to the target first source field. The preset semantic of at least one general field is determined based on the semantic label of the corresponding source field.

[0057] In the first mapping rule, there is a one-to-one correspondence between the first source field, semantic label, and first general field. On the one hand, establishing the first mapping rule facilitates the efficient writing of field values ​​from multiple first source fields into corresponding first general fields, and facilitates the efficient mapping of field values ​​from one or more target data sources to the general source table. On the other hand, based on the first mapping rule, the semantic label of the first source field can also provide a preset semantic for the corresponding first general field, which is beneficial for the server to calculate the semantic similarity between fields based on the first general field. Specifically, if the first general field mapped to the first source field of the target already has a preset semantic before the first mapping rule is established, since the similarity between this preset semantic and the semantic label of the target's first source field meets the preset requirements, i.e., the two are highly similar, from a specific meaning perspective, this preset semantic can also be considered to be determined based on the semantic label of the target's first source field.

[0058] Alternatively, the server can also obtain the field values ​​of multiple first source fields during the process of acquiring multiple first source fields from one or more target data sources and determining a first mapping rule based on the multiple first source fields. For example, the server synchronously acquires multiple first source fields and their values ​​from one or more target data sources, and then determines a first mapping rule based on the multiple first source fields, so as to write the field values ​​of the multiple first source fields into multiple first general fields according to the first mapping rule.

[0059] In some cases, after determining the first mapping rule based on multiple first source fields, the semantic labels of the multiple first source fields, and multiple first general fields of the general source table, the server records the first mapping rule in the metadata dictionary of the general source table.

[0060] Recording the first mapping rule in the metadata dictionary can be used to explain the preset semantics of multiple first general fields in the general source table and the source of the preset semantics. When the field value of the first source field is written into the first general field, it can also be used to explain the source of the field values ​​stored in multiple first general fields, which facilitates the backtracking of the data mapping process in the later stage.

[0061] As mentioned earlier when introducing the general source table, the metadata storage fields included in the general source table are at least used to link mapping rules between data sources and the general source table. Based on this, optionally, the field value of the metadata storage field is a storage address, which is used to link mapping rules recorded in the metadata dictionary, such as the first mapping rule corresponding to one or more target data sources. Optionally, the field value of the metadata storage field is based on the mapping rule linked to by the storage address, such as the field value being the first mapping rule. By linking the first mapping rule in the metadata storage field, it is beneficial to efficiently and conveniently view the first source field to which the field values ​​stored in each first general field belong, and to understand the preset semantics of each first general field.

[0062] For example, the first mapping rule is recorded in the mapping rule table, which is a pre-defined table in the metadata dictionary of the general source table. The mapping rule table is used to record different versions of the mapping rule.

[0063] Each version of the mapping rule is recorded in the mapping rule table, including the first mapping rule. Recording each version of the mapping rule in the form of a preset table is used to record and manage changes to the mapping rule, which facilitates easier access to rules, detection of conflicts, and tracing back the data mapping process.

[0064] For example, each mapping rule recorded in the mapping rule table contains the following information: version number, data source, mapping system, mapping rule, rule effective time, and whether it is the currently used version. The version number is a unique identifier for the mapping rule, which can be used to search for and invoke the mapping rule. The data source and mapping system are the input and output of the data mapped by the mapping rule, respectively. The mapping rule is the core information recorded, such as the specific mapping relationship. The rule effective time refers to the time when the corresponding mapping rule was generated; for example, the time when the first mapping rule was determined is the effective time of the first mapping rule. When a new version of the mapping rule becomes effective, it is marked as the currently used version by default, and historical versions of the mapping rule become invalid. Marking the currently used version helps ensure the consistency of data mapping rules across multiple systems. By recording the above-mentioned information for the mapping rules, it is beneficial to comprehensively and clearly trace back and audit the data mapping process.

[0065] In one feasible implementation, writing the field values ​​of multiple first source fields into multiple first general fields according to a first mapping rule includes: the server generating a data mapping script according to the first mapping rule, the data mapping script being used to write the field values ​​of multiple first source fields into multiple first general fields, and executing the data mapping script to realize the writing of the field values ​​of multiple first source fields into multiple first general fields.

[0066] For example, after generating the above data mapping script, if data from one or more target data sources is received and multiple first source fields of the target data sources remain unchanged, the received data can be efficiently and conveniently written into the general source table using the above data mapping script. Similarly, the first mapping version remains the currently used version. When receiving one or more target data sources, the server does not need to redetermine the mapping rules between the one or more target data sources and the general source table, nor does it need to generate a new data mapping script.

[0067] In one feasible implementation, after writing the field values ​​of multiple first source fields into multiple first general fields according to the first mapping rule, the server also corrects the field values ​​in the multiple first general fields according to the preset industry data template.

[0068] The semantics of the template fields in the preset industry data template are related to the preset semantics of the first general field. The preset industry data template is used to provide a data representation format. Specifically, the preset industry data template is a predefined representation template for data in common business scenarios across various industries. For example, a representation template for automotive industry-related data, including data such as car brands and models.

[0069] During the generation of one or more target data sources, the representation format of field values ​​in these data sources may vary. For example, the representation format of certain data within an industry might be the full name, but one or more target data sources might only record the abbreviation of this data. The semantics of template fields in the preset industry data template are related to the business scenario, and the preset semantics of the first general field are determined based on semantic tags with mapping relationships, which also refer to the business-level meaning. The semantic relationship between the two indicates that the business scenario involved in the preset industry data template is related to the business to which the field values ​​of multiple first general fields belong. Therefore, by modifying the field values ​​of multiple first general fields in the general source table according to the preset industry data template, the data representation format can be unified and business standardized, which helps to reduce misunderstandings.

[0070] In some cases, the server modifies the field values ​​of multiple primary general fields according to a preset industry data template in the following specific ways.

[0071] First, each of the multiple first general fields has a preset semantic meaning, and at least two template fields of the preset industry data template also have semantic meaning. The server determines the template field that matches the target general field among the multiple first general fields based on at least two semantic similarities between each first general field and at least two template fields. Among the at least two semantic similarities corresponding to the target general field, there exists a semantic similarity greater than a second threshold.

[0072] Multiple primary general fields store the values ​​of multiple primary source fields, including industry-related data and non-industry-related data. Taking the automotive industry as an example, the values ​​of primary source fields such as vehicle model and vehicle brand are industry-related data. The values ​​of primary source fields such as customer name and delivery address are not industry-related. It can be seen that whether a field value is industry-related depends on whether the business meaning (i.e., semantic label) of the corresponding primary source field is industry-related. The semantic label of the primary source field assigns a preset semantic to the primary general field. The server corrects the field values ​​in multiple primary general fields according to the preset industry data template. It must first determine the industry-related data for correction; that is, it first determines the target general field with the preset semantic related to the industry from the multiple primary general fields to correct the expression format of the target general field's field value.

[0073] By calculating the similarity between the preset semantics of each first general field and the semantics of at least two template fields, at least two semantic similarities for each first general field can be obtained. For any first general field, if one of the at least two semantic similarities is greater than a second threshold, it indicates that at least two template fields of the preset industry data template contain template fields with business meanings similar to those of the first general field. Therefore, this first general field can be identified as the target general field, and the field values ​​stored in the target general field are industry-related data. Template fields with business meanings similar to the target general field are matching template fields. The field values ​​of the matching template fields provide a data representation format and can be used as a basis for correcting the field values ​​of the target general field.

[0074] For example, if the semantic similarity between the first general field with the preset semantics of "customer|name" and all template fields in the automotive industry data template is less than or equal to the second threshold, it means that there are no template fields in the automotive industry data template with business meanings similar to "customer|name". If the semantic similarity between the first general field with the preset semantics of "car|brand" and all template fields in the automotive industry data template is greater than the second threshold, it means that there are template fields in the automotive industry data template with business meanings similar to "car|brand". The field values ​​of these similar template fields can be used to correct the field value of the first general field (target general field) with the preset semantics of "car|brand".

[0075] Furthermore, the server corrects the field value of the target general field based on the field value of the matching template field. The field value of the template field that matches the target general field provides a data representation format, which is used to correct the representation format of the target general field's field value. For example, the template field has at least two field values. The similarity between the matching target general field value and the at least two field values ​​of the template field is calculated to obtain the similarity score of at least two field values. The field value of the template field corresponding to the highest field value similarity is determined as the correction basis, and the representation format of the target general field's field value is corrected based on this correction basis. Thus, the standardization of the representation format of the data stored in the general source table can be achieved.

[0076] S300 retrieves multiple target fields from the target system.

[0077] After storing the data from one or more target data sources in a general source table, the field values ​​stored in multiple first general fields of the general source table are written to the target system according to the needs of the target system, thereby achieving data mapping between one or more target data sources and the target system. Specifically, multiple target fields of the target system are first obtained. Each target field has semantic meaning.

[0078] S400, based on the similarity between the semantics of each target field and the preset semantics of multiple first general fields, determine multiple second general fields from the multiple first general fields that match the multiple target fields respectively. As described above, the preset semantics of the first general fields can be determined based on the semantic tags mapped to the first general fields in the first mapping rule.

[0079] Specifically, based on the semantic similarity between fields, the server can automatically determine multiple second general fields from multiple first general fields that match multiple target fields respectively, that is, determine the second general field that matches each target field. For example, the second general field is the first general field among multiple first general fields whose preset semantics are most similar to the corresponding target field. This facilitates the efficient establishment of mapping relationships between multiple second general fields and multiple target fields, that is, it facilitates the efficient establishment of mapping rules between the general source table and the target system.

[0080] In one feasible implementation, such as Figure 4 As shown, Figure 4 The flowchart provided in this application embodiment illustrates the process of determining the second general field that matches each target field. The server can also determine the second general field that matches each target field through the following steps S410 to S420 or the following steps S410 and S430.

[0081] S410, calculate the similarity between the semantics of each target field and the preset semantics of multiple first general fields, and obtain multiple semantic similarities for each target field.

[0082] For example, one or more methods such as vectorization, deep learning, and regular expressions are used to calculate the similarity between the semantics of each target field and the preset semantics of multiple first general fields, thereby obtaining multiple semantic similarities for each target field. The vectorization method may be a text vectorization method that combines Term Frequency-Inverse Document Frequency (TF-IDF) and Word to Vector (Word2Vec). The deep learning method may be a deep learning model such as Bidirectional Encoder Representations from Transformers (BERT). This implementation does not limit the method used to calculate the semantic similarity.

[0083] The following example demonstrates a feasible process for calculating the similarity between the semantics of a target field and the preset semantics of a first general field using regular expressions. The semantics of the target field are directly matched with the preset semantics of the first general field (e.g., the semantic tags of the first source field with which a mapping relationship exists) using regular expressions. If they match directly, the similarity is considered to be 100%. If they do not match directly, the following steps are taken: First, the expression form with the highest semantic similarity to the target field is obtained from the thesaurus associated with the target field in the expression template library. The expression template library is a preset library containing multiple expression forms with various semantics. For example, for the semantic "customer name," the expression template library includes multiple expressions such as "customer name," "customer name," "customer name," and "company." For instance, if the semantics of the target field is "customer name," the expression form with the highest similarity obtained from the expression template library is "customer name." In this context, a semantic expression may have multiple forms, such as Chinese and English synonyms. Since the corresponding general field is locked in the general source table through the target field, if the target field and its semantics are in English, the expression with the highest similarity to the target field's semantics can be efficiently determined directly from multiple English synonyms. Next, the determined expression with the highest similarity is matched with the preset semantics of the first general field using regular expressions to obtain the similarity between the determined expression and the preset semantics. For example, the number of characters in the expression that are identical to the preset semantics is determined, and the total number of characters in the preset semantics is determined. Then, the number of identical characters divided by the total number of characters in the preset semantics is used as the similarity between the determined expression and the preset semantics, that is, as the similarity between the semantics of the target field and the preset semantics of the first general field. For example, the number of identical characters refers to the number of identical characters obtained after preprocessing the expression and the preset semantics. Preprocessing could include converting Chinese characters into the first letters of their pinyin (e.g., Customer Name—KHMC) or converting English characters into the first letters of their words (e.g., Customer Name—CN). For example, the determined total number of characters refers to the total number of characters after preprocessing and deduplication. Furthermore, if the expression with the highest similarity among multiple English synonyms also fails to match the preset semantics of the general field, the expression with the highest similarity among multiple Chinese synonyms can be further determined and matched using regular expressions.

[0084] When the server determines the matching second general field for each target field from multiple first general fields based on multiple semantic similarities corresponding to each target field, in some cases, the second general field is configured to be the first general field corresponding to the highest semantic similarity among multiple semantic similarities; in other cases, the second general field is configured to be the first general field corresponding to the highest semantic similarity among multiple semantic similarities that is greater than or equal to a first threshold. In the second case, since multiple semantic similarities corresponding to the target field may all be less than the first threshold, it is difficult to directly determine the matching second general field for the target field based solely on multiple semantic similarities. For the specific handling method of the second case, please refer to the following steps S420 or S430.

[0085] S420, if there is a first similarity among the multiple semantic similarities of the target field that is greater than or equal to the first threshold, then based on the first similarity, determine the second general field that matches the target field from the multiple first general fields.

[0086] The existence of a first similarity greater than or equal to a preset first threshold indicates that among multiple first general fields, there exists a first general field whose business meaning is similar to that of the target field, and the degree of similarity has reached the preset requirement. Subsequently, the server automatically determines a second general field that matches the target field based on the first similarity. For example, if one first similarity exists, the first general field corresponding to the first similarity is determined as the matching second general field. If at least two first similarities exist, the first general field corresponding to the largest of the at least two first similarities is determined as the matching second general field.

[0087] S430, if multiple semantic similarities of the target field are all less than a first threshold, a field writing instruction is received to determine a second general field that matches the target field. The field writing instruction includes the following information: the correspondence between the target field and the matching second general field.

[0088] If multiple semantic similarities are all less than the first threshold, it indicates that the similarity in business meaning between multiple first general fields of the general source table and the target fields does not meet the preset requirements. At this time, the server receives a field write instruction, which includes the following information: the correspondence between the target field and the matching second general field. Based on the instruction, the server determines the second general field that matches the target field. For example, the server outputs a prompt message on the graphical user interface to inform the user which target fields do not match the first general fields, and then the user inputs a field write instruction based on the graphical user interface.

[0089] In this data mapping method, the second general field that matches the target field is automatically determined by the semantic similarity standard. If the semantic similarity does not meet the standard, a field writing instruction is received, and the second general field that matches the target field is determined based on the field writing instruction. In this way, the second general field that matches each target field of the target system can be determined from multiple first general fields.

[0090] In one feasible implementation, after determining the second general field that matches each target field of the target system from multiple first general fields, the server determines and records the mapping rules between the general source table and the target system in the metadata dictionary. The mapping rules between the general source table and the target system include the matching relationships between multiple target fields of the target system and multiple second general fields of the general source table. Recording these mapping rules in the metadata dictionary facilitates later backtracking of the data mapping process. For example, uniformly recording the mapping rules between the general source table and the target system in a rule mapping table in the metadata dictionary facilitates easier viewing, calling, and detection of the mapping rules.

[0091] S500 writes the field value of each second general field to the matching target field.

[0092] For example, depending on the structure of the data to be written to the target system, the server completes the data writing by calling the target system's REST API interface or through direct database connection. Optionally, if the writing fails, retry and alarm instructions are triggered.

[0093] During the data mapping process, after obtaining the field values ​​of multiple first source fields from one or more target data sources, the server writes these values ​​into multiple first general fields of a general source table according to pre-established first mapping rules. This automatically and efficiently maps the field values ​​of one or more target data sources to the general source table. Further, after obtaining multiple target fields from the target system, the server determines the matching second general fields for each target field in the target system from the multiple first general fields with preset semantics in the general source table, based on the semantic similarity between the fields. It should be understood that a target field matching a second general field based on semantic similarity means that the target field also matches the first source field mapped by the second general field. Based on this, the server precisely writes the field value of the first source field in each second general field into the matching target field, i.e., into the target system. Thus, the automatic and efficient mapping of field values ​​from one or more target data sources stored in the general source table to the target system is achieved, i.e., efficient data mapping between one or more target data sources and the target system is realized.

[0094] The server executes this data mapping method without requiring manual configuration of mapping rules between each first source field of one or more target data sources and each target field of the target system. This effectively avoids one-to-one hard-coded mappings between the first source fields of one or more target data sources and the target fields of the target system, thus improving data mapping efficiency. For example, using a general source table as an intermediate carrier, the data mapping method in this embodiment can be used to achieve a smooth and efficient data transition between legacy systems and cloud-native systems.

[0095] In one feasible implementation, when obtaining the target field of the target system, the server also obtains the data format conditions of the target system. The data format conditions refer to the format conditions that data written to the target system must meet. Please refer to... Figure 5 , Figure 5 This is a schematic diagram of the process of writing the field value of each second general field to the matching target field according to an embodiment of this application. The server writes the field value of each second general field to the matching target field through the following steps S510 to S520.

[0096] S510, validate each piece of data to be written according to the target rules to obtain the corresponding target data. The data to be written is the field value of the second general field that matches the target field. The target rules include at least one of the following: data format conditions, business rules of the industries associated with multiple target fields, a first mapping rule, and the matching relationship between multiple target fields and the second general field.

[0097] Specifically, after performing pre-processing on each piece of data to be written, the corresponding target data is obtained, and then each target data is written into the matching target field. The pre-processing involves validating the data to be written according to the target rules; the validation includes verification and correction.

[0098] For example, data format requirements include: requiring the date format to be "yyyy-MM-dd", requiring the length of the target data to be within a preset range, requiring the data structure of the target data to be a preset structure, requiring the vehicle unit to be "unit", requiring the business serial number to be completed, etc. Performing data validation based on the target system's data format requirements to obtain the target data helps ensure that the target data written to the target system conforms to the target system's data format requirements, and improves the accuracy and standardization of data mapping. For example, the server, based on the target data's data format requirements, converts the date format in the data to be written from "yyyyMMdd" to "yyyy-MM-dd"; unifies the vehicle unit in the data to be written from "vehicle" to "unit"; and automatically completes the business serial number based on the data to be written to obtain the target data.

[0099] For example, the business rules of the industry associated with the target field may include requirements such as the order quantity being greater than or equal to 1, and the vehicle model being within an enumerated range. Performing data validation based on these industry business rules to obtain the target data helps reduce data errors and improves the accuracy of the target data.

[0100] For example, the first mapping rule, the matching relationship between multiple target fields and the second general field can be used to verify the consistency of data mapping between one or more target data sources and the target system, so as to ensure that the target data written to the target system matches the field values ​​of one or more target data sources.

[0101] In some cases, when the second general field matching the target field stores field values ​​from multiple data sources, the aforementioned validation of the data to be written also includes data fusion. The following example illustrates this with multiple data sources, including a first data source and a second data source, and one or more target data sources being either the first or second data source. For instance, if the second general field with the preset semantics of "order|quantity" stores the field value "2 vehicles" from the first data source and the field value "2 units" from the second data source, the server will validate the data to be written, including "2 vehicles" and "2 units," according to the target rules. This validation includes: fusing the two data points "2 vehicles" and "2 units," and correcting the data according to the target system's requirement that the unit for vehicles be "unit," to ultimately obtain the target data including "2 units."

[0102] S520 writes each target data to the matching target field.

[0103] In one feasible implementation, the server also writes mapping information between one or more target data sources and the target system into the target system. The mapping information includes at least one of the following: a first mapping rule, and matching relationships between multiple target fields and multiple second general fields.

[0104] In addition to writing the values ​​of the second general fields into the target system to achieve efficient data mapping between one or more target data sources and the target system, the server also writes the mapping information between one or more target data sources and the target system into the target system to enable traceable writing of field values. Specifically, at least one of the first mapping rules used in the data mapping process and the matching relationships between multiple target fields and multiple second general fields is written into the target system. This enables the target system to perform data reverse lookups, trace back the original data and mapping process in the general source table or one or more target data sources, which is beneficial for meeting auditing and rule rollback requirements.

[0105] In one feasible implementation, the server writes the information set from the data mapping task into a log to facilitate information backtracking and retrieval. The information set includes at least one of the following: one or more target data sources and multiple first source fields of the target data sources; a target system and multiple target fields of the target system; mapping rules between one or more target data sources and a general source table, and version numbers of the mapping rules; and mapping rules between the general source table and the target system, and version numbers of the mapping rules.

[0106] In one feasible implementation, during the execution of the data mapping method provided in this application embodiment, the server periodically identifies whether one or more target data sources have added a second source field or removed a third source field, where the third source field is one of multiple first source fields. Specifically, when the server obtains data from one or more target data sources through a target adapter, the server performs periodic identification through the target adapter. If one or more target data sources have not added a second source field or removed a third source field, the server executes the data mapping method as described above. If one or more target data sources have added a second source field or removed a third source field, based on the specific execution process of the data mapping method described above, the server performs corresponding processing operations to adapt to changes in the field values ​​and other business data of one or more target data sources. This will be described in detail below.

[0107] In some cases, if a second source field is added to one or more target data sources, the server determines a second mapping rule. The second mapping rule is obtained by adding a mapping relationship between the second source field, the semantic label of the second source field, and the third general field based on the first mapping rule. The third general field is a general field that is not part of the first general field.

[0108] The specific implementation methods for the server to obtain the second source field, determine the semantic label of the second source field, determine the mapping relationship corresponding to the second source field, and obtain the field value of the second source field can be referred to the relevant introduction about the first source field in the previous text, and will not be repeated here.

[0109] It should be understood that if a second source field is added to one or more target data sources, the mapping rules used to interpret the mapping relationships between fields and to generate data mapping scripts to write the field values ​​from one or more target data sources into the general source table need to be updated. Specifically, the server adds a second source field, semantic tags for the second source field, and mapping relationships between the third general field to the existing first mapping rule to obtain the second mapping rule.

[0110] In traditional data migration or mapping methods, when business data changes, such as adding source fields and their corresponding values ​​to a data source, the mapping rules for all source fields in the data source must be manually modified—essentially reconfiguring all rules. This traditional method is time-consuming and labor-intensive, and the complexity of rule management increases exponentially as the size of the data source or source fields grows. Furthermore, reconfiguring the mapping rules for all source fields is difficult to handle scenarios with frequent changes in source fields, resulting in high maintenance costs and a high risk of errors. In this data mapping method, the server does not need to redefine the mapping relationships between the second source field and each of the first source fields in one or more target data sources. It only needs to update the changed parts to obtain the latest version of the second mapping rules, thus improving the efficiency of updating mapping rules. Moreover, the server only needs to dynamically adjust the established first mapping rules to obtain the second mapping rules, resulting in faster and lower maintenance costs, and reducing the risk of errors when updating mapping rules.

[0111] Furthermore, based on the established second mapping rules and the time points at which the second source field is added to one or more target data sources, the server performs additional processing operations. The specific server operations are described below when the second source field is added to one or more target data sources at three different time points.

[0112] At time point one, before the server writes the values ​​of multiple first source fields into multiple first general fields according to the first mapping rule, a second source field is added. In this case, it is equivalent to one or more target data sources adding a second source field at the beginning of the data mapping. Correspondingly, the server obtains the values ​​of multiple first source fields and second source fields from one or more target data sources, and then, according to the second mapping rule, writes the values ​​of multiple first source fields and second source fields into multiple first general fields and third general fields. If the values ​​of the first source fields have already been obtained, the server only needs to obtain the newly added value of the second source field.

[0113] The execution process for other aspects of the data mapping method can be referenced from the execution process described above when no second data source is added. The difference is that the server needs to use the second mapping rule as the latest and currently used version of the mapping rule to perform the corresponding operations. For example, a data mapping script is generated based on the second mapping rule, and the data mapping script is executed to write the field values ​​of one or more target data sources into the general source table. Another example is determining the second general field that matches each target field based on the semantics of each target field and its similarity to the preset semantics of multiple first and third general fields.

[0114] At time point two, after writing the field values ​​of multiple first source fields into multiple first general fields according to the first mapping rule, and before determining the second general field that matches each target field from the multiple first general fields, a new implementation is added. In this case, to write all field values ​​from one or more target data sources into the general source table to more comprehensively and accurately realize the data mapping between one or more target data sources and the target system, an optional implementation is that the server generates a new mapping script for writing the field values ​​of the second source fields into the third general field according to the second mapping rule, and executes the new mapping script. In some cases, the server also updates the data mapping script based on the new mapping script, and the updated data mapping script is also used to write the second source fields. Based on this, if the server subsequently receives data from one or more target data sources, and the multiple first source fields and second source fields of one or more target data sources have not changed, the server can efficiently and conveniently write the newly received data into the general source table through the updated data mapping script. Another optional implementation is that the server updates the data mapping script according to the second mapping rule, and executes the updated data mapping script, the updated data mapping script being used to write the field values ​​of multiple first source fields and second source fields into multiple first general fields and third general fields, respectively. It should be understood that through the above processing operations, the server dynamically updates the data stored in the general source table, which can adapt to changes in business data such as field values ​​of one or more target data sources.

[0115] The execution process for other data mapping methods can be referenced from the process described above when no second data source is added. The difference is that the server needs to use the second mapping rule as the latest and currently used version of the mapping rule to perform the corresponding operations. For example, based on the semantics of each target field and the similarity between it and the preset semantics of multiple first and third general fields, the second general field that matches each target field is determined.

[0116] At time point three, after determining the second general field that matches each target field from multiple first general fields, and writing the value of each second general field to the matching target field, a new field is added. In this case, the server has completed data mapping from one or more target data sources to the target system. Therefore, the second mapping rule, as a preset, currently used version of the mapping rule, is used by the server in the next process of receiving data from one or more target data sources and performing data mapping. In some cases, when writing field values ​​from one or more target data sources to the general source table next time, the server needs to update the data mapping script based on the second mapping rule to enable the writing of the field values ​​of the second source fields.

[0117] In some cases, if a third source field is removed from one or more target data sources, the server also determines a second mapping rule. Here, the third source field is one of multiple first source fields, and the second mapping rule is obtained by marking the mapping relationship between the third source field, its semantic label, and the corresponding first general field as invalid, based on the first mapping rule.

[0118] Similarly, if a second source field is removed from one or more target data sources, the mapping rules used to interpret the mapping relationships between fields and to generate data mapping scripts to write the field values ​​from one or more target data sources into the general source table also need to be updated. The server determines the second mapping rule. The server only needs to make dynamic adjustments based on the determined first mapping rule. By marking the changed parts as invalid to obtain the latest version of the second mapping rule, the efficiency of updating the mapping rule can be improved. Moreover, the maintenance of the mapping rule is fast and the maintenance cost is low, which helps to reduce the risk of errors when updating the mapping rule.

[0119] Similarly, based on the determined second mapping rule and the time point at which the third source field is removed from one or more target data sources, the server performs other processing operations. For a detailed description of the server's specific operations when the third source field is removed from one or more target data sources at three different time points, please refer to the description above regarding the addition of the second source field. The following section only describes the differences in the server's operations in the removal scenario.

[0120] Specifically, for time node one, for example, the server writes the field values ​​of multiple first source fields into multiple first general fields according to the second mapping rule. These multiple first source fields do not include the third source field. If one or more target data sources delete the third source field before the server has obtained the field values ​​of one or more target data sources, the server only needs to obtain the field values ​​of one or more target data sources that do not include the third source field. Another example is that the server determines the matching second general field for each target field based on the similarity between the semantics of each target field and the preset semantics of multiple first general fields. These multiple first general fields do not include the first general field corresponding to the third source field. For time node two, for example, the server updates the data mapping script according to the second mapping rule. The updated data mapping script is used to write the field values ​​of multiple first source fields from one or more target data sources into a general source table. These multiple first source fields do not include the third source field. Based on this, if the server subsequently receives data from one or more target data sources, and the multiple first source fields of one or more target data sources have not changed, the server can efficiently and conveniently write the newly received data into the general source table using the updated data mapping script. For example, the server determines the second general field that matches each target field based on the similarity between the semantics of each target field and the preset semantics of multiple first general fields. These multiple first general fields do not include the first general field corresponding to the third source field.

[0121] In one feasible implementation, after the server determines the second mapping rule by adding a second source field or removing a third source field from one or more target data sources, it also records the second mapping rule in the metadata dictionary of the general source table. The metadata dictionary also records the first mapping rule, and the second mapping rule is marked as being in use.

[0122] When a second mapping rule is determined, the server records it in the metadata dictionary to explain the source of data stored in the common fields of the common source table, facilitating backtracking and auditing of the data mapping process. At this point, the metadata dictionary records at least the first and second mapping rules. When the second mapping rule is determined and recorded, it automatically takes effect and is marked as in use, indicating that the latest second mapping rule is the currently used version, and the field values ​​of one or more target data sources in the common source table are stored based on the second mapping rule. The server records each version of the mapping rule in the metadata dictionary and marks the currently used version, which helps users or the system to more easily view each version of the mapping rule, backtrack the data mapping process, and detect mapping conflicts. Furthermore, in this data mapping method, the metadata dictionary supports incremental updates of the mapping rules. For example, based on the recorded first mapping rule, the server can perform incremental updates based on the first mapping rule to determine the second mapping rule.

[0123] For example, the first and second mapping rules are uniformly recorded in the mapping rule table of the metadata dictionary, and the second mapping rule is marked as being in use. Recording the various versions of mapping rules through the mapping rule table allows for the recording and management of rule changes, facilitating easier rule retrieval, conflict detection, and data mapping process review. This data mapping method, based on the mapping rule table, supports version updates and incremental updates of mapping rules, thus enabling full lifecycle management of mapping rules.

[0124] Based on the detailed description of the data mapping method in the aforementioned embodiments, and for ease of understanding, a feasible example is provided below to provide a clear and comprehensive analysis of the data mapping method.

[0125] Business scenario background: System #1 is a Customer Relationship Management (CRM) system, storing data from data source #1, which contains customer order data. Data source #1 is an API-type data source, accessible via a REST API interface.

[0126] Data source #1 includes multiple source fields with mixed Chinese and English names: Customer Identitydocument (Customer ID) (e.g., CUST2025001), Order Time (e.g., 2025-08-01 09:30), Vehicle Model (e.g., Model Y), Order Quantity (e.g., 2 vehicles), and Customer Phone Number (e.g., 138xxxx5678).

[0127] Source system #2 is an Enterprise Resource Planning (ERP) system, which stores data source #2, which contains production planning data. Data source #2 is a file-based data source accessed via a direct database connection.

[0128] The data source includes multiple source fields with a mix of Chinese characters and code names: customer number (e.g., KH2025001), order date (e.g., 2025-08-01), vehicle model (e.g., Model Y 2025), quantity purchased (e.g., 2 units), and contact number (e.g., 138xxxx5678).

[0129] Business scenario goals: Integrate the data from data sources 1 and 2 into the enterprise data platform (target system) for linked analysis of "order-production".

[0130] Step 1: The server uniformly receives data from data source 1 and data source 2. 1) Receive JSON data from data source #1 via the REST API of source system #1 (example below): { Customer ID: CUST2025001 Order Time: 2025-08-01 09:30 Vehicle type: "ModelY" Order Quantity: 2 vehicles Customer Phone Number: 138xxxx5678 } 2) Receive Excel-formatted data from the MySQL database of source system #2 (example below):

[0131] Step 2: The server performs data standardization processing. 1) Data standardization processing for data source #1: The server parses the Swagger document corresponding to data source 1 through the API access adapter, extracts the field information of the source fields (such as "Customer ID: unique customer identifier"; "Order Time: transaction timestamp"), obtains multiple source fields, and extracts the interface information of the API. The server calls a semantic tag extraction tool to generate semantic tags for each source field (e.g., "Customer ID" is customer|id; "Order Time" is order|time). The server calls the API interface to extract the field value of the source field; The server converts JSON data into Excel data.

[0132] 2) Data standardization processing for data source #2: The server reads the field information of the table corresponding to data source #2 (such as "Customer ID: Customer Unique Code"; "Order Date: Order Creation Date") through the database access adapter, obtains multiple source fields, and reads the field values ​​of each source field in the table; The server calls a semantic tag extraction tool to generate semantic tags for each source field (e.g., "Customer ID" is customer|id; "Order Date" is order|date).

[0133] The standardized source field values ​​and semantic tags (examples are provided below, and the same applies to other source fields):

[0134] Step 3: The server stores the general source table. First, the server receives data: the output of step two, "the field values ​​corresponding to the source fields and the semantic tags of the source fields." In the table above, the source system corresponds to the data source.

[0135] Secondly, the server calls the universalsourcetable to process the data: 1) Determine the mapping rules between data source #1 and the general source table. The mapping rules should include at least the following mapping relationships: Mapping between the source field "Customer ID", the semantic tag "customer|id", and the common field "field5" in the common source table; Mapping between the source field "order time", the semantic tag "order|time", and the common field "field12" in the common source table; Mapping between the source field "vehicle model", the semantic tag "product|model", and the common field "field23" in the common source table; Mapping between the source field "order quantity", the semantic tag "order|quantity", and the common field "field24" in the common source table.

[0136] 2) Determine the mapping rules between data source 2 and the general source table. In cases where the preset semantics of a general field in the general source table (determined according to the semantic labels of the mapping) are similar to the semantic labels of the source fields in data source 2, a mapping relationship between the source fields and the general fields with similar semantics can be established.

[0137] For example, suppose the default semantics of field5, which maps to "Customer ID", are "customer|id", and the semantic label of the source field "Customer Number" in data source 2 is also "customer|id". Since the two semantics are similar, it can be determined that both "Customer Number" and "Customer ID" refer to customers. Therefore, a mapping relationship between the source field "Customer Number" and field5 is also established.

[0138] For example, suppose the default semantics of field12, which maps to "order time", are "order|time", and the semantic label of the source field "order placement time" in data source 2 is "order|date". Since the two semantics are similar, it can be determined that both "order time" and "order placement time" refer to time. Therefore, a mapping relationship between the source field "order placement time" and field12 is also established.

[0139] The mapping rules for data source #2 should include at least the following mapping relationships: Mapping between the source field "Customer ID", the semantic tag "customer|id", and the common field "field5" in the common source table; Mapping between the source field "order time", the semantic tag "order|date", and the common field "field12" in the common source table; Mapping between the source field "vehicle model", the semantic tag "product|model", and the common field "field23" in the common source table; Mapping between the source field "Purchase Quantity", the semantic tag "order|quantity", and the common field "field24" in the common source table.

[0140] In this example, the default semantics of field5 are determined based on the semantic tag "customer|id"; The default semantics of field12 are determined based on the semantic tags “order|time” and / or “order|date”; The default semantics of field23 are determined based on the semantic tag "product|model"; The default semantics of field24 are determined based on the semantic tag "order|quantity".

[0141] Finally, the server generates and executes a data mapping script based on the mapping rules. This script is used to write the field values ​​corresponding to the source fields into the mapped common fields. Additionally, the server records the mapping rules corresponding to the data source in the metadata dictionary of the common source table, specifically in the mapping rule table (mappingversion) of the metadata dictionary.

[0142] Thus, the server obtains data from the generic source table. The metadata storage field of the generic source table is used to link the mapping rules (between the data source and the generic source table) stored in the metadata dictionary. When the generic source table stores field values ​​from the data source, the corresponding field value in the metadata storage field represents the mapping rule of the linked data source.

[0143] The following is an example of general source table data (the field values ​​for other source fields are similar):

[0144] Optionally, the server can also automatically supplement the time precision. For example, it can complete the field value "2025-08-01" corresponding to data source No. 2 in the ERP source system to "2025-08-01 00:00".

[0145] Step 4: The server performs dynamic mapping processing. The server receives data: the "General Source Table Data" output in step three. The source systems in the table correspond to the data sources.

[0146] The server will process this: The server retrieves the automotive industry-related data format template, "Automotive Industry Order Template." It determines that the value of the generic field field23, with the preset semantics of "product|model," is associated with the automotive industry. Based on the "ModelY" format template in the "Automotive Industry Order Template," the server automatically corrects the value of field23. Specifically, both "ModelY" and "ModelY2025" match product|model=ModelY, thus automatically correcting the format of "ModelY2025" to "ModelY."

[0147] Step 5: The server outputs the data from the general source table to the target system. The server receives the data: the "General Source Table Data" output in step four.

[0148] The server will process this: First, the server determines the data to be written from the general source table based on the target fields required by the enterprise data platform. For example, the target fields required by the enterprise data platform are unified customer ID, unified order time, vehicle model, and order quantity. Based on the semantic similarity between each target field and multiple general fields with preset semantics in the general source table, the server determines the general fields that match each target field. Then, the data stored in data source 1 and data source 2 containing the matching general fields are determined as the data to be written for the target fields.

[0149] Secondly, the server performs checks, corrections, and integrations on the data to be written based on the data format requirements of the enterprise data platform. For example, the target data required by the enterprise data platform may be in JSON format, with a time format of "yyyy-MM-ddHH:mm:ss", and the data may not include units.

[0150] Since data source 1 and data source 2 are two data sources for the same business, the data in data source 1 and data source 2 stored in the same common fields may be redundant. The server first performs data fusion on the data to be written in the target field to obtain a standardized mapping result (example):

[0151] The server then converts the merged data into a format that meets the data format requirements. For example, it automatically completes the time format, converting "2025-08-01 09:30" to "2025-08-01 09:30:00", and automatically removes the units of quantity, converting "2 vehicles" to "2", thus obtaining the target data for each target field.

[0152] Finally, the server calls the enterprise data platform API to write each target data into the matching target field.

[0153] For example, the server also converts "Unified Customer ID" to "customerunifiedid"; "Unified Order Date" to "orderunifieddate"; "Vehicle" to "vehicle_model"; "Order Quantity" to "order_quantity"; and provides the field "data_source" to obtain the final mapping result in JSON format. The server calls the enterprise data platform API to write the final mapping result.

[0154] The integrated data received by the enterprise data platform (examples below): { "customer_unified_id": "U2025001", "order_unified_date": "2025-08-0109:30:00", "vehicle_model": "ModelY", "order_quantity": "2", "data_source": "integrated" } For example, during or after data mapping by the server, the data source may involve changes to its source fields. In this case, incremental updates are used to update and maintain the mapping rules. The following example illustrates a change in the source fields of data source 1 after the target data is written to the target system.

[0155] The server receives data from data source 1 after the source fields have been modified. For example, data source 1 has added a new source field "urgency level", with the data example being: {"urgency level": "high"}.

[0156] The server will process this: First, the newly added source field "urgency level" in data source 1 is automatically detected. The server obtains the field value of data source 1 and extracts the semantic label "order|priority" of the newly added source field. Secondly, the server establishes a mapping relationship between the newly added source field "urgency level", the semantic tag "order|priority", and the general field field45; Secondly, based on the original mapping rules between data source 1 and the general source table recorded in the metadata dictionary, a new mapping relationship corresponding to the "urgency level" source field is added, and the new mapping rules are updated. The new mapping rules are recorded in the mapping rule table of the metadata dictionary and marked as being in use (the version number of the mapping rule is updated from V1.0 to V1.1, and V1.1 adds the mapping relationship corresponding to "urgency level" based on V1.0). Finally, the server updates and executes the data mapping script according to the new mapping rules. The updated data mapping script can also be used to map the field value corresponding to the source field "urgency" to field45.

[0157] At this point, the field value of the metadata storage field is the new mapping rule that is linked to (that is, the mapping relationship corresponding to the "urgency" source field is added on the basis of the original mapping rule).

[0158] The updated general source table and metadata storage fields now include the following information:

[0159] Example 2 Figure 6 Please refer to the schematic diagram of the data mapping device provided in the embodiments of this application. Figure 6The data mapping device 10 is used to implement the data mapping method in the aforementioned embodiment one. Exemplarily, the data mapping device 10 includes a data source acquisition module 101, a field value writing module 102, an information acquisition module 103, a field matching module 104, and a data writing module 105. It should be understood that the module division and naming here are for illustrative purposes only. In specific implementations, some modules may be combined together, or one module may be split into multiple modules.

[0160] In one feasible implementation, a data source acquisition module 101 is used to acquire field values ​​of multiple first source fields from one or more target data sources; a field value writing module 102 is used to write the field values ​​of the multiple first source fields into multiple first general fields according to a first mapping rule; the multiple first general fields are at least a portion of multiple general fields in a general source table; the first mapping rule is a mapping relationship between the multiple first source fields, the semantic labels of the multiple first source fields, and the multiple first general fields of the general source table; an information acquisition module 103 is used to acquire multiple target fields of a target system; a field matching module 104 is used to determine multiple second general fields that match the multiple target fields from the multiple first general fields according to the similarity between the semantics of each target field and the preset semantics of the multiple first general fields; the preset semantics of the first general fields are determined according to the semantic labels mapped to the first general fields in the first mapping rule; and a data writing module 105 is used to write the field value of each second general field into the matching target field.

[0161] In one feasible implementation, the field matching module 104 is used to calculate the similarity between the semantics of each target field and the preset semantics of multiple first general fields, thereby obtaining multiple semantic similarities for each target field; if there is a first similarity among the multiple semantic similarities of the target field that is greater than or equal to a first threshold, then based on the first similarity, a second general field matching the target field is determined from the multiple first general fields; if all the multiple semantic similarities of the target field are less than the first threshold, then a field writing instruction is received to determine the second general field matching the target field, wherein the field writing instruction includes the following information: the correspondence between the target field and the matching second general field.

[0162] In one feasible implementation, the data writing module 105 is used to verify each piece of data to be written according to the target rules to obtain the corresponding target data; wherein, the data to be written is the field value of the second general field that matches the target field, and the target rules include at least one of the following: data format conditions, business rules of the industries associated with multiple target fields, a first mapping rule, and the matching relationship between multiple target fields and the second general field, wherein the data format conditions refer to the format conditions that the data written to the target system must meet; and each piece of target data is written to the matching target field.

[0163] In one feasible implementation, the data writing module 105 is used to write mapping information between one or more target data sources and the target system into the target system; the mapping information includes at least one of the following: a first mapping rule, and a matching relationship between multiple target fields and multiple second general fields. In one feasible implementation, the data source acquisition module 101 is used to acquire multiple first source fields from one or more target data sources; and determine semantic labels for the multiple first source fields based on the multiple first source fields, wherein the semantic labels are used to indicate the meaning of the corresponding first source fields in the business. The data mapping device 10 further includes a rule determination module, used to determine a first mapping rule based on the multiple first source fields, the semantic labels of the multiple first source fields, and multiple first general fields of the general source table.

[0164] In one feasible implementation, the data mapping device 10 further includes a rule recording module for recording a first mapping rule in the metadata dictionary of the general source table.

[0165] In one feasible implementation, the data mapping device 10 is applied to a server, on which multiple adapters run. These adapters are used for data interaction between various types of data sources and a common source table, with each adapter corresponding to one type of data source. The data source acquisition module 101 is used to acquire multiple first source fields of one or more target data sources through a target adapter that matches the type of one or more target data sources. The target adapter is one of the multiple adapters. Through the target adapter, the semantic labels of the multiple first source fields are determined based on the multiple first source fields.

[0166] In one feasible implementation, the data source acquisition module 101 is used to acquire the field names and field descriptions of multiple first source fields of one or more target data sources; the field description is information used to describe the meaning of the corresponding first source field in the business; and the semantic tags of the multiple first source fields are determined based on the field names and field descriptions of the multiple first source fields.

[0167] In one feasible implementation, the data writing module 105 is used to generate a data mapping script according to a first mapping rule, the data mapping script being used to write the field values ​​of multiple first source fields into multiple first general fields; and to execute the data mapping script.

[0168] In one feasible implementation, the data mapping device 10 further includes a data correction module, which is used to correct the field values ​​in multiple first general fields according to a preset industry data template; wherein the semantics of the template fields in the preset industry data template are associated with the preset semantics of the first general fields, and the preset industry data template is used to provide a data representation format.

[0169] In one feasible implementation, the rule determination module is used to determine a second mapping rule if a second source field is added to one or more target data sources. The second mapping rule is obtained by adding a mapping relationship between the second source field, the semantic label of the second source field, and a third general field based on the first mapping rule. The third general field is a general field other than the first general field.

[0170] In one feasible implementation, the rule determination module is used to determine a second mapping rule if a third source field is deleted from one or more target data sources, wherein the third source field is one of a plurality of first source fields, and the second mapping rule is obtained by marking the mapping relationship between the third source field, the semantic label of the third source field, and the first general field corresponding to the third source field as invalid based on the first mapping rule.

[0171] In one feasible implementation, a rule recording module is used to record a second mapping rule in the metadata dictionary of a general source table, wherein the metadata dictionary also records a first mapping rule, and the second mapping rule is marked as being in use.

[0172] The explanations of the terms can be found in the relevant descriptions in the foregoing data mapping method embodiments, and will not be elaborated upon here. The specific execution process of the data mapping device 10 can be found in the detailed description of the foregoing data mapping method embodiments, and will not be repeated here.

[0173] Each module of the aforementioned data mapping device 10 can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0174] Example 3 Figure 7 Please refer to the schematic diagram of the computer device provided in the embodiments of this application. Figure 7 For example, the computer device 20 includes a processor 201 coupled to a memory 202 for temporarily storing computer programs or instructions and / or data. The processor 201 is used to execute the computer programs or instructions stored in the memory 202 or to read the data stored in the memory 202 to perform the methods in the aforementioned data mapping method embodiments.

[0175] Optionally, there may be one or more processors 201.

[0176] Optionally, the memory 202 may be one or more.

[0177] Alternatively, the memory 202 can be integrated with the processor 201, or it can be set separately.

[0178] Optionally, the computer device 20 also includes a transceiver for receiving and / or transmitting signals. For example, the processor 201 controls the transceiver to receive and / or transmit signals.

[0179] It should be understood that the processor in the embodiments of this application can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. A general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in the embodiments of this application can be directly implemented by a hardware encoding processor, or implemented by a combination of hardware and software modules in the encoding processor. The software modules can reside in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. This storage medium is located in memory, and the processor reads information from the memory and, in conjunction with its hardware, completes the steps of the above method.

[0180] The memory in the embodiments of this application can be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. Non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory can be random access memory (RAM), which serves as an external cache. By way of example, but not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), and synchronous link dynamic random access memory (SLDRAM).

[0181] It should be noted that when the processor is a general-purpose processor, DSP, ASIC, FPGA, or other programmable logic device, discrete gate or transistor logic device, or discrete hardware component, the memory can be integrated into the processor.

[0182] It should also be noted that the memory described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0183] Example 4 This application also provides a computer-readable storage medium that can store multiple computer program instructions. These computer program instructions are adapted to be loaded by a processor and executed by the method steps in the aforementioned data mapping method embodiments. For details of the execution process, please refer to... Figures 1 to 5 The specific details of the illustrated embodiments will not be elaborated here.

[0184] Example 5 In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. This computer program product includes one or more computer program codes or instructions. When the computer program code or instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer program code or instructions can be stored in a computer-readable storage medium or transferred from one computer-readable storage medium to another. For example, the computer program code or instructions can be transferred from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium accessible to a computer or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state drive), etc.

[0185] Those skilled in the art will understand that implementing all or part of the processes in the above method embodiments can be accomplished by hardware related to computer program code or instructions. The computer program code or instructions can be stored in a computer-readable storage medium, and when executed, can include the processes described in the above method embodiments. The aforementioned storage medium includes various media capable of storing program code, such as ROM or random access memory (RAM), magnetic disks, or optical disks.

[0186] The technical features in the above embodiments can be arbitrarily combined. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as the combination of these technical features does not contradict each other, it should be considered within the scope of this specification. The above embodiments only illustrate several feasible implementation methods of this application, and their descriptions are relatively specific and detailed, but they should not be construed as limiting the scope of this application's patent. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A data mapping method, characterized in that, The method includes: Retrieves the values ​​of multiple first source fields from one or more target data sources; According to the first mapping rule, the field values ​​of the plurality of first source fields are written into a plurality of first general fields; the plurality of first general fields are at least a portion of the plurality of general fields in the general source table; the first mapping rule is the mapping relationship between the plurality of first source fields, the semantic tags of the plurality of first source fields, and the plurality of first general fields in the general source table; Retrieve multiple target fields from the target system; Based on the similarity between the semantics of each target field and the preset semantics of the plurality of first general fields, a plurality of second general fields matching the plurality of target fields are determined from the plurality of first general fields; the preset semantics of the first general fields are determined based on the semantic tags mapped to the first general fields in the first mapping rules; and, Write the field value of each of the second general fields to the matching target field.

2. The method according to claim 1, characterized in that, The step of determining multiple second general fields that match the multiple target fields from the multiple first general fields based on the similarity between the semantics of each target field and the preset semantics of the multiple first general fields includes: Calculate the similarity between the semantics of each target field and the preset semantics of the plurality of first general fields to obtain multiple semantic similarities for each target field; If there is a first similarity greater than or equal to a first threshold among the multiple semantic similarities of the target field, then based on the first similarity, a second general field matching the target field is determined from the multiple first general fields; If multiple semantic similarities of the target field are all less than the first threshold, a field writing instruction is received to determine a second general field that matches the target field. The field writing instruction includes the following information: the correspondence between the target field and the matching second general field.

3. The method according to claim 1 or 2, characterized in that, The step of writing the field value of each of the second general fields into the matching target field includes: Each piece of data to be written is validated according to the target rules to obtain the corresponding target data; wherein, the data to be written is the field value of the second general field that matches the target field, and the target rules include at least one of the following: data format conditions, business rules of the industries associated with the multiple target fields, the first mapping rule, and the matching relationship between the multiple target fields and the second general field, and the data format conditions refer to the format conditions that the data written to the target system must meet; Write each of the target data into the matching target field.

4. The method according to any one of claims 1 to 3, characterized in that, The method further includes: Write the mapping information between the one or more target data sources and the target system into the target system; The mapping information includes at least one of the following: the first mapping rule, and the matching relationship between the plurality of target fields and the plurality of second general fields.

5. The method according to any one of claims 1 to 4, characterized in that, Before obtaining the field values ​​of multiple first source fields from one or more target data sources, the method further includes: Obtain the plurality of first source fields from the one or more target data sources; Based on the plurality of first source fields, semantic labels are determined for the plurality of first source fields, and the semantic labels are used to indicate the meaning of the corresponding first source fields in the business. The first mapping rule is determined based on the plurality of first source fields, the semantic labels of the plurality of first source fields, and the plurality of first general fields of the general source table.

6. The method according to claim 5, characterized in that, After determining the first mapping rule based on the plurality of first source fields, the semantic tags of the plurality of first source fields, and the plurality of first general fields of the general source table, the method further includes: The first mapping rule is recorded in the metadata dictionary of the general source table.

7. The method according to claim 5 or 6, characterized in that, This method is applied to a server, on which multiple adapters run. These multiple adapters are used for data interaction between various types of data sources and the general source table, with each adapter corresponding to one type of data source. The step of obtaining the plurality of first source fields from the one or more target data sources includes: The plurality of first source fields of the one or more target data sources are obtained by a target adapter that matches the type of the one or more target data sources, wherein the target adapter is one of the plurality of adapters; The step of determining the semantic tags of the plurality of first source fields based on the plurality of first source fields includes: The target adapter determines the semantic tags of the plurality of first source fields based on the plurality of first source fields.

8. The method according to any one of claims 5 to 7, characterized in that, The step of obtaining the plurality of first source fields from the one or more target data sources includes: Obtain the field names and field descriptions of the plurality of first source fields from the one or more target data sources; the field descriptions are information used to describe the meaning of the corresponding first source fields in the business context. The step of determining the semantic tags of the plurality of first source fields based on the plurality of first source fields includes: Based on the field names and field descriptions of the plurality of first source fields, determine the semantic tags of the plurality of first source fields.

9. The method according to any one of claims 1 to 8, characterized in that, The step of writing the field values ​​of the plurality of first source fields into a plurality of first general fields according to the first mapping rule includes: According to the first mapping rule, a data mapping script is generated, which is used to write the field values ​​of the plurality of first source fields into the plurality of first general fields; Execute the data mapping script.

10. The method according to any one of claims 1 to 9, characterized in that, After writing the field values ​​of the plurality of first source fields into the plurality of first general fields according to the first mapping rule, the method further includes: Based on the preset industry data template, correct the field values ​​in the plurality of first general fields; The semantics of the template fields in the preset industry data template are associated with the preset semantics of the first general field, and the preset industry data template is used to provide a data representation format.

11. The method according to claim 1, characterized in that, The method further includes: If a second source field is added to one or more target data sources, a second mapping rule is determined. The second mapping rule is obtained by adding a mapping relationship between the second source field, the semantic label of the second source field, and the third general field based on the first mapping rule. The third general field is a general field other than the first general field among the multiple general fields.

12. The method according to claim 1, characterized in that, The method further includes: If a third source field is deleted from one or more target data sources, a second mapping rule is determined, wherein the third source field is one of the plurality of first source fields, and the second mapping rule is obtained by marking the mapping relationship between the third source field, the semantic label of the third source field, and the first general field corresponding to the third source field as invalid based on the first mapping rule.

13. The method according to claim 11 or 12, characterized in that, After determining the second mapping rule, the method further includes: The second mapping rule is recorded in the metadata dictionary of the general source table, wherein the first mapping rule is also recorded in the metadata dictionary, and the second mapping rule is marked as being in use.

14. A data mapping device, characterized in that, The device includes: The data source acquisition module is used to obtain the field values ​​of multiple first source fields from one or more target data sources; The field value writing module is used to write the field values ​​of the plurality of first source fields into a plurality of first general fields according to a first mapping rule; the plurality of first general fields are at least a portion of the plurality of general fields of the general source table; the first mapping rule is the mapping relationship between the plurality of first source fields, the semantic tags of the plurality of first source fields and the plurality of first general fields of the general source table; The information acquisition module is used to acquire multiple target fields from the target system; A field matching module is used to determine, based on the similarity between the semantics of each target field and the preset semantics of the plurality of first general fields, a plurality of second general fields that respectively match the plurality of target fields; the preset semantics of the first general fields are determined based on the semantic tags mapped to the first general fields in the first mapping rules; and, The data writing module is used to write the field value of each of the second general fields to the matching target field.

15. A computer device, characterized in that, The device includes a processor and a memory, the memory being used to store computer program instructions, and the processor being used to invoke the computer program instructions stored in the memory to cause the computer device to perform the method according to any one of claims 1 to 13.

16. A computer-readable storage medium, characterized in that, The computer-readable storage medium is used to store computer program instructions that, when executed by a processor, cause an apparatus including a processor to perform the method according to any one of claims 1 to 13.

Images