Data type mapping method, related device and storage medium

Abstract

The embodiment of the invention relates to the field of data processing, and provides a data type mapping method, a related device and a storage medium, and the method comprises the steps: obtaining source end data and target end data source types; wherein the source end data comprises data source types, annotations and contents; converting source end data and target end data source types into feature vectors; wherein the feature vectors comprise feature vectors of semantic features and feature vectors of statistical features; and inputting the feature vector into a data type recommendation model, generating a target end data type and a value range corresponding to the to-be-mapped field in the source end data, and outputting the target end data type and the value range. According to the embodiment of the invention, the data type recommendation model predicts the target data type according to the field context of the to-be-mapped field, the annotation of the source end data, the field name of the to-be-mapped field, the sample value of the to-be-mapped field and the like, and understands the real semantics and use of the to-be-mapped field; therefore, the predicted target data type has higher accuracy.

Description

TECHNICAL FIELD

[0001] Embodiments of the present application relate to the field of data processing, and more particularly to a data type mapping method, related apparatus and storage medium. BACKGROUND

[0002] In a big data synchronization task, field mapping between different data sources needs to be performed. The existing mapping method is to manually judge the data type (such as integer type INT, variable length string VARCHAR, date type DATE) of the source field, and determine the data type of the target field. Facing data sources with different structures and non-uniform naming (such as MySQL, Oracle, Kafka, CSV, etc.), manual judgment of the data type of the field is difficult, and errors such as type conflict and field truncation are prone to occur, which is difficult to meet the efficiency and accuracy requirements of complex data synchronization tasks. SUMMARY

[0003] Embodiments of the present application provide a data type mapping method, related apparatus and storage medium, which can predict the data type of the target field and improve the accuracy of field type judgment.

[0004] In a first aspect, the embodiments of the present application provide a data type mapping method, which comprises: obtaining source data and target data source type; wherein the source data comprises a data source type of the source data, an annotation of the source data and content of the source data; converting the source data and the target data source type into a feature vector; wherein the feature vector comprises a feature vector of semantic features of the source data, a feature vector of semantic features of the target data source type, and a feature vector of statistical features of the source data; inputting the feature vector into a data type recommendation model to generate a target data type corresponding to a field to be mapped in the source data and a value range of the target data type; and outputting the target data type and the value range of the target data type.

[0005] In a possible implementation, if the data source type of the source data is structured data, the content of the source data comprises a plurality of fields to be mapped and at least one of the following: field context of the plurality of fields to be mapped, data type of the field, field annotation and sample value of each field to be mapped; if the data source type of the source data is semi-structured data, the content of the source data comprises a plurality of fields to be mapped and at least one of the following: field context of the plurality of fields to be mapped, field annotation and sample value of each field to be mapped; if the data source type of the source data is unstructured data, the content of the source data comprises a plurality of fields to be mapped and at least one of the following: field context of the plurality of fields to be mapped, field annotation and sample value of each field to be mapped.

[0006] In a possible implementation, before the feature vector is input into the data type recommendation model, the method further includes: obtaining a correspondence between the data type of the original source data and the data type of the source data; obtaining a correspondence between the value range of the data type of the original source data and the value range of the data type of the source data; converting the correspondence between the data type of the original source data and the data type of the source data and the correspondence between the value range of the data type of the original source data and the value range of the data type of the source data into a supplementary feature vector; inputting the feature vector and the supplementary feature vector into the data type recommendation model to generate the target end data type corresponding to the to-be-mapped field in the source end data and the value range of the target end data type; and the value range of the target end data type corresponding to the to-be-mapped field is the same as the value range of the data type of the original source data.

[0007] In a possible implementation, the data type recommendation model is obtained by the following method: obtaining basic information of a data type and cross-data-source type mapping samples, and constructing a knowledge base of the data type; the basic information of the data type includes at least one of the following: attributes, constraints, coding rules, and operation characteristics of the data type; obtaining a feature vector of the data type based on the knowledge base of the data type; determining a label based on the cross-data-source type mapping sample data; the label includes a main label and an auxiliary label, the main label is used to mark a task that needs to be predicted by the data type recommendation model, and the auxiliary label is used to describe a data source; and training the data type recommendation model by using the feature vector of the data type and the label.

[0008] In a possible implementation, the feature vector of the data type is obtained based on the knowledge base of the data type, and includes: constructing a feature vector of data type semantic features based on the basic information of the data type and the cross-data-source type mapping samples; constructing a feature vector of data type statistical features based on the cross-data-source type mapping samples; and splicing the feature vector of the data type semantic features and the feature vector of the data type statistical features to obtain the feature vector of the data type.

[0009] In a possible implementation, after the data type recommendation model is trained by using the feature vector of the data type and the label, the method further includes: constructing a feature vector of a newly added data type statistical feature based on historical synchronization task records in the enterprise; performing incremental training on an output layer of the data type recommendation model by using the feature vector of the newly added data type statistical feature, to obtain a new data type recommendation model; and wherein the incremental training parameters include: the optimizer is AdamW, the learning rate is set to 1e-5, that is, 0.00001, and the training rounds are 3, 4, or 5 rounds.

[0010] In a possible implementation, the basic information of the data type includes at least one of the following: an attribute, a constraint, a coding rule, or an operation characteristic of the data type, and the cross-data-source type mapping sample includes: a source field name, a source data type, a target data type, a source data source type, a target data source type, a target data type length or precision, a source data type constraint condition, a mapping reason, a source field annotation, or data of multiple dimensions of a source sample value.

[0011] In a second aspect, an embodiment of the present application provides a data type mapping device having a function of implementing the data type mapping method provided in the first aspect. The function can be implemented by hardware, or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the above functions, which can be software and / or hardware.

[0012] In an embodiment, the device includes: an input and output module configured to obtain source data and a target data source type; wherein the source data includes a data source type of the source data, an annotation of the source data, and content of the source data; a processing module configured to convert the source data and the target data source type into a feature vector; wherein the feature vector includes a feature vector of semantic features of the source data, a feature vector of semantic features of the target data source type, and a feature vector of statistical features of the source data; the processing module is further configured to input the feature vector into a data type recommendation model to generate a target data type corresponding to a field to be mapped in the source data and a value range of the target data type; and the input and output module is further configured to output the target data type and the value range of the target data type.

[0013] In a third aspect, an embodiment of the present application provides a computer-readable storage medium including instructions that, when executed on a computer, cause the computer to perform the data type mapping method of the first aspect.

[0014] In a fourth aspect, an embodiment of the present application provides a computing device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the data type mapping method of the first aspect when executing the computer program.

[0015] In a fifth aspect, an embodiment of the present application provides a chip, which comprises a processor coupled with a transceiver, and is configured to implement the technical solution of the first aspect. In a possible design, the chip can also be a special hardware structure for implementing the technical solution of the first aspect, for example, the processing related to the neural network model can be implemented by a special neural network processor or a graphic processor.

[0016] In a sixth aspect, an embodiment of the present application provides a chip system, which comprises a processor configured to implement the functions related to the first aspect, for example, to generate or process the information related to the data type mapping method of the first aspect.

[0017] In a possible design, the chip system further comprises a memory connected to the processor by a circuit structure. The memory is configured to store program instructions and data necessary for the terminal. The chip system can be composed of a chip, or can include the chip and other discrete devices. Further optionally, the chip further comprises a communication interface connected to the processor. The communication interface is configured to receive data and / or information to be processed, and the processor obtains the data and / or information from the communication interface and processes the data and / or information, and outputs the processing result through the communication interface. The communication interface can be an input / output interface.

[0018] In a seventh aspect, an embodiment of the present application provides a computer program product comprising instructions, which, when executed on a computer, cause the computer to perform the data type mapping method of the first aspect.

[0019] By the method of the embodiments of the present application, the mapping relationship between data types of different data sources (such as structured data, semi-structured data, unstructured data, etc.) can be obtained through the data type recommendation model, so that the data type of the target end field does not need to be determined manually, saving the time of the developer. In addition, the data type recommendation model can predict the target data type according to the field context of the field to be mapped, the annotation of the source end data, the field name of the field to be mapped, and the sample value of the field to be mapped, etc., by understanding the real semantic and purpose of the field to be mapped, so that the predicted target data type has higher accuracy. For example, the field name is name, if the field names of other fields in Table 1 are mostly related to the parameters of the commodity, it can be inferred that the data in the name indicates the name of the commodity, and the name of the commodity occupies more space, if the field names of other fields in Table 2 are mostly related to the information of the person, it can be inferred that the data in the name field indicates the name of the person, and the name of the person occupies less space, so that a more suitable data type is recommended. BRIEF DESCRIPTION OF DRAWINGS

[0020] The purposes, features and advantages of the embodiments of the present application will become easy to understand by reading the detailed description of the embodiments of the present application with reference to the accompanying drawings. Among them: Figure 1 The flowchart of the data type mapping method of the embodiments of the present application is shown in the figure; Figure 2 The training flowchart of the data type recommendation model of the embodiments of the present application is shown in the figure; Figure 3 The structure diagram of the data type mapping device of the embodiments of the present application is shown in the figure; Figure 4 The structure diagram of the computing device of the embodiments of the present application is shown in the figure; Figure 5 The structure diagram of the server in the embodiments of the present application is shown in the figure.

[0021] In the drawings, the same or corresponding reference numerals represent the same or corresponding parts. DETAILED DESCRIPTION

[0022] The terms "first", "second", etc. in the specification and claims of the present application and in the above drawings are used to distinguish similar objects (for example, the first xx and the second xx are respectively represented as different xx, and other similar objects) and do not necessarily have to be used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged under appropriate circumstances, so that the embodiments described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device that includes a series of steps or modules does not necessarily have to be limited to those steps or modules clearly listed, but can include other steps or modules that are not clearly listed or inherent to these processes, methods, products or devices. The division of modules in the embodiments of the present application is only a logical division, and in actual application, other division methods can be used, for example, a plurality of modules can be combined or integrated into another system, or some features can be omitted or not executed. In addition, the coupling or direct coupling or communication connection between the displayed or discussed modules can be through some interface, indirect coupling between modules, and communication connection can be electrical or other similar forms, which are not limited in the embodiments of the present application. In addition, the modules or sub-modules described as separate components can or can not be physically separate, can or can not be physical modules, or can be distributed in a plurality of circuit modules, and some or all of the modules can be selected according to actual needs to achieve the purpose of the embodiments of the present application.

[0023] The scheme provided by the embodiments of the present application relates to technologies such as artificial intelligence (AI), computer vision (CV), machine learning (ML), etc. The specific embodiments are described as follows: AI is a theory, method, technology and application system that uses a digital computer or a machine controlled by a digital computer to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology of computer science, which tries to understand the essence of intelligence and produce a new intelligent machine that can react in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines, so that the machine has the functions of perception, reasoning and decision-making.

[0024] AI technology is a comprehensive discipline, involving a wide range of fields, both hardware and software technologies. Artificial intelligence basic technologies generally include technologies such as sensors, special artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction system, mechatronics, etc. Artificial intelligence software technology mainly includes computer vision technology, speech processing technology, natural language processing technology, and machine learning / deep learning, etc.

[0025] CV is a science that studies how to make machines "see". Further, it refers to using cameras and computers to replace human eyes to identify, track and measure targets, etc. Machine vision, and further image processing, so that computer processing becomes more suitable for human eye observation or image transmission to instrument detection. As a scientific discipline, computer vision researches related theories and technologies, trying to establish artificial intelligence systems that can obtain information from images or multi-dimensional data. Computer vision technology usually includes anti-disturbance generation, image recognition, image semantic understanding, image retrieval, OCR, video processing, video semantic understanding, video content / behavior recognition, three-dimensional object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous localization and mapping, etc.

[0026] Multi-source heterogeneous data synchronization refers to mapping data in the source data source to different types of target data sources. Among them, the data source includes database, file (such as JSON, PDF, etc.), multimedia data (such as pictures, videos, audio), etc. For example, data in MySQL, Oracle or PostgreSQL is synchronized to Hive, Flink, ClickHouse, etc. Data lake or data warehouse system.

[0027] In the multi-source heterogeneous data synchronization scenario, the correct judgment and mapping of field types are the premise of correct data entry into the lake / warehouse. Its importance is reflected in the following aspects: 1. Field type determines data parsing and storage method. For example, the source is VARCHAR(32), and the target is incorrectly mapped to INT, which will cause parsing failure or data anomaly. 2. Field type affects downstream computing logic. For example, DECIMAL(10,2) is incorrectly mapped to STRING, which will affect the accuracy of numerical aggregation and statistical analysis. 3. Field type affects compression and query performance. For example, incorrectly mapping low cardinality fields to wide character types will significantly increase storage costs and reduce query performance. 4. Different system field types have large compatibility differences. For example, PostgreSQL's TEXT, Oracle's NUMBER, and MySQL's TINYINT have different mapping rules in Hive, and manual maintenance is complex and error-prone.

[0028] Existing field data type prediction methods include rule template method and method of classification by shallow machine learning classifier. For example, the rule template method can be implemented as: if the field name contains id, it is inferred to be an integer type, but it depends on naming conventions and has poor generalization ability. The method of classification by shallow machine learning classifier can predict the field based on field sample values (or field example data), but cannot understand field semantics and context information, and has low accuracy. When facing complex fields and multi-semantic fields (such as field name user_status, created_flag, etc.), the two methods are prone to misjudgment and are difficult to apply to large-scale practical engineering. Therefore, in large-scale synchronization tasks, the system needs to automatically analyze the field original type and field sample value information, intelligently judge the target type, so as to improve the synchronization efficiency and accuracy.

[0029] Based on this, the embodiment of the application provides a data type mapping method, related device and storage medium, the data source type of the source end data, the annotation of the source end data, the content of the source end data and the target end data source type are analyzed by the data type recommendation model, and the target end data type of the field is recommended in combination with the field context. The embodiment recommends the target end data type by understanding the real semantics and use of the field to be mapped, so that the recommended target data type has higher adaptability to the data.

[0030] The embodiment of the application provides a computing device, which can implement the data type mapping method provided in the embodiment of the application. The computing device can be a server and / or a terminal device.

[0031] The server related to the embodiment of the application can be an independent physical server, a server cluster or a distributed system composed of multiple physical servers, or a cloud server providing cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDNs, and basic cloud computing services such as big data and artificial intelligence platforms.

[0032] The terminal device referred to in the embodiments of the present application can be a device providing voice and / or data connectivity for a user in a device having wireless connection functionality, or other processing device connected to a wireless modem. For example, a mobile phone (also known as a "cellular" phone) and a computer with mobile termination, for example, can be a portable, pocket, hand-held, computer-embedded or car-mounted mobile apparatus that exchanges voice and / or data with a wireless access network. For example, Personal Communication Service (PCS) phones, cordless phones, Session Initiation Protocol (SIP) phones, Wireless Local Loop (WLL) stations, Personal Digital Assistants (PDAs), and the like.

[0033] Reference Figure 1 , Figure 1 A flowchart of a data type mapping method of the embodiments of the present application is shown. The data type mapping method provided by the embodiments of the present application is described below, which includes steps 101 to 104.

[0034] In step 101, source data and target data source types are obtained.

[0035] The source data includes a data source type of the source data, an annotation of the source data, and content of the source data.

[0036] The data source type of the source data can be three categories of structured data, semi-structured data, and unstructured data.

[0037] For example, the structured data includes a database (such as a relational database such as MySQL, Oracle, PostgreSQL, SQL Server), a data warehouse, or a data lake, etc. If the data source type of the source data is structured data, the content of the source data includes a plurality of fields to be mapped and at least one of the following: a field context of the plurality of fields to be mapped, a data type of the field, a field annotation, and a sample value of each field to be mapped; the annotation of the source data can be an annotation provided by the data source type itself, for example, when the data type of the source data is MySQL, the MySQL data includes a table and an annotation of the MySQL data, and the annotation of the MySQL data is used to describe the purpose of the table.

[0038] As the comment of the student table is: the table is used to count the information of students, and the comment of the cargo information table is: the table is used to count the cargo information. When the source data is the student table, the system can obtain, according to the comment of the student table, that the name in the student name table indicates the name of the student, and the length of the name field should not be too long, such as only 10 characters. When the source data is the cargo information table, the system can obtain, according to the comment of the cargo information table, that the name of the cargo can be determined according to multiple information such as batch and time, and the name can be long, so 100 characters can be needed. Therefore, the system can determine the length of the target data type according to the comment of the source data.

[0039] Especially, the data type of the source data does not have a length, and the data type of the target data has a length. For example, from HIVE to Mysql, the name type in HIVE does not have a field length, and the Mysql type after conversion has a field length setting. The length and other parameters of the target data type can be determined by referring to the comment of the table.

[0040] Taking the source data as an example of MySQL type data, part of the to-be-mapped fields and sample values corresponding to the to-be-mapped fields are shown in Table 1.

[0041] Table 1: Example of to-be-mapped fields

[0042] For example, semi-structured data includes MongoDB, Elasticsearch, etc. If the data source type of the source data is semi-structured data, the content of the source data includes multiple to-be-mapped fields and at least one of the following: field context, field annotation, and sample value of each to-be-mapped field. The comment of the source data can be realized as human-added metadata, such as recording the purpose of the source data through a topic table.

[0043] For example, unstructured data includes Kafka, Pulsar, or RabbitMQ, etc. If the data source type of the source data is unstructured data, the content of the source data includes multiple to-be-mapped fields and at least one of the following: field context, field annotation, and sample value of each to-be-mapped field. The comment of the source data can be realized as human-added metadata, such as recording the purpose of the source data through a topic table.

[0044] Step 102: converting the source data and the target data source type into a feature vector.

[0045] The feature vector includes the feature vector of the semantic feature of the source data, the feature vector of the semantic feature of the target data source type, and the feature vector of the statistical feature of the source data.

[0046] In some embodiments, the feature vectors of the semantic features of the data type source data of different types include different contents.

[0047] For numerical data types, the data type semantic features include the source data type, the source data source type, and at least one of the following: the source field name, the field context, the source data type constraint condition, the mapping reason, the source field annotation, and the source sample value.

[0048] For text data types, the data type semantic features include the source data type, the source data source type, and at least one of the following: the source field name, the field context, the source data type constraint condition, the mapping reason, the source field annotation, and the source sample value.

[0049] For time data types, the data type semantic features include the source data type, the source data source type, and at least one of the following: the source field name, the target data source type, the target data type length or precision, the field context, the source data type constraint condition, the mapping reason, the source field annotation, the source sample value, or the time format (such as “yyyy-MM-dd”).

[0050] By distinguishing that the field belongs to a numerical type, a text type, or a time type, and constructing the feature vector of the data type semantic features according to the type to which each data type belongs, the feature vector of each data type to be mapped has characteristics that can better reflect the data type, so that the model recommendation for the target data type is more appropriate, and space waste or character truncation is reduced.

[0051] Similarly, the feature vectors of the statistical features of the data type source data of different types include different contents.

[0052] The statistical features of the source data include the statistical features of each field to be mapped in the source data, and the statistical features of each field to be mapped are determined according to the sample value of the field to be mapped.

[0053] For text fields, the data type statistical features include at least one of the following: the maximum length, the average length, the 90% quantile length, the 95% quantile length, the Chinese proportion, or the null value proportion. The above statistical features in turn refer to the maximum length, the average length, the 90% quantile length, the 95% quantile length, the Chinese proportion, or the null value proportion of the sample of the field to be mapped.

[0054] For the numerical type field, the data type statistical features include at least one of the following: maximum value, minimum value, maximum length of integer part, maximum length of decimal part, numerical range, number of unique values, or standard deviation of numerical distribution. The above statistical features in turn refer to the maximum value, minimum value, maximum length of integer part, maximum length of decimal part, numerical range, number of unique values, or standard deviation of numerical distribution of the sample of the field to be mapped.

[0055] For the numerical type field, the data type statistical features include at least one of the following: maximum length, average length, or null value ratio. The above statistical features in turn refer to the maximum length, average length, or null value ratio of the sample of the field to be mapped.

[0056] By distinguishing that the field belongs to a numerical type, a text type, or a time type, etc., a feature vector of data type statistical features is constructed according to the type to which each data type belongs, so that the feature vector of each data type to be mapped has features that can better reflect the data type, thereby making the model's recommendation of the target end data type more appropriate and reducing space waste or character truncation.

[0057] In step 103, the feature vector is input into the data type recommendation model to generate the target end data type corresponding to the field to be mapped in the source end data and the value range of the target end data type.

[0058] Some numerical type data types and text type data types can be set to have a value range, such as a set length, precision, or scale, etc. The data type recommendation model can determine the appropriate length of the target end data type according to the feature vector, according to the type, length, etc. of the field to be mapped in the source end data. By setting the value range, the data type can occupy less space while meeting the use requirements, saving computing resources.

[0059] Taking the target end data source MySQL as an example, if the data type of the field is integer, and the data length of the field is less than 1 byte, then the TINYINT data type is selected, if the data length of the field is not less than 1 byte and less than 2 bytes, then the SMALLINT data type is selected, if the data length of the field is not less than 2 bytes and less than 3 bytes, then the MEDIUMINT data type is selected, and if the data length of the field is not less than 3 bytes and less than 4 bytes, then the INT data type is selected.

[0060] Still taking the target end data source MySQL as an example, the numerical data types include: VARCHAR(n), DECIMAL(p, s), etc., and the text data types include: CHAR(n), etc. If it is determined that the data type is VARCHAR(n), the maximum number of characters n that can be stored in the field can be set. If it is determined that the data type is DECIMAL(p, s), the length p and precision s of the decimal in the field can be set. If it is determined that the data type is CHAR(n), the maximum number of characters n that can be stored in the field can be set.

[0061] In step 104, the target end data type and the value range of the target end data type are output.

[0062] For example, the above steps 102 to 104 can be implemented by a pre-trained data type recommendation model, so as to obtain the target end data type and the value range of the target end data type output by the data type recommendation model. The data type recommendation model for executing step 102 and the data type recommendation model for executing step 104 can be the same model or different models.

[0063] Through the above method, the mapping relationship between data types of different data sources (such as structured data, semi-structured data, unstructured data, etc.) can be obtained, so that the data type of the target field does not need to be determined manually, and the time of the developer is saved. In addition, the data type recommendation model can predict the target data type according to the field context of the field to be mapped, the annotation of the source end data, the field name of the field to be mapped, and the sample value of the field to be mapped, etc. By understanding the real semantics and purpose of the field to be mapped, the accuracy of the predicted target data type is higher. For example, the field name is the same as name, if the field names of other fields in Table 1 are mostly related to the parameters of the product, it can be inferred that the data in the name indicates the name of the product, and the name of the product occupies more space. If the field names of other fields in Table 2 are mostly related to the information of the person, it can be inferred that the data in the name field indicates the name of the person, and the name of the person occupies less space, so a more suitable data type is recommended.

[0064] The following is an example of an embodiment of the present application.

[0065] Example one: mapping of structured data to structured data (MySQL→Hive).

[0066] Source field includes: name (data type of field: VARCHAR (32), sample value: "Zhang San", "Alice", "Li Si"), according to the content of the source field, the feature vector of semantic characteristics ("name" + "username" + sample value encoding, etc.) + statistical characteristics (maximum length 4, Chinese proportion 67%) are obtained and input into the data type recommendation model.

[0067] Source field includes: email (data type of field: VARCHAR (100), sample value: "user@example.com", "contact@company.org", "support@service.net", field annotation: user email address), according to the content of the source field, the feature vector of semantic characteristics ("email" + "user email address" + sample value encoding (such as "contains_at_symbol", "contains_dot", "has_domain")) + statistical characteristics (maximum length 100, average length: 18.5, unique value proportion: 95%, null value rate: 0.1%) are obtained and input into the data type recommendation model.

[0068] The above name and email are contents in the MySQL data source, and the two are in a context relationship.

[0069] The data type recommendation model outputs: the data type STRING of Hive, where STRING has no length limit.

[0070] Example two: semi-structured data MongoDB to structured data Oracle mapping.

[0071] Source field includes: order_info (sample value: {"order_id":"12345","amount":199.99}), according to the content of the source field, the feature vector of semantic characteristics ("order_info" + "order information" + sample value encoding + data type, etc.) + statistical characteristics (maximum length 50, containing key-value pairs) are obtained and input into the data type recommendation model. The data type recommendation model outputs: the data type VARCHAR2 (100) of Oracle.

[0072] Example three: structured data MySQL to structured data Oracle mapping, but the source field is named by enterprise-specific naming.

[0073] The source field includes: je (data type of the field: DECIMAL (10,2), sample value: “100.50” “2000.00”, field annotation: no annotation, annotation of source data: the use scenario of the data source is a financial scenario), according to the content of the source field, the semantic feature vector (“je”+“enterprise financial field”+sample value coding+data type, etc.)+statistical feature (decimal places 2, maximum integer part 5 bits) is obtained, and is input into the data type recommendation model. The output of the data type recommendation model is: the data type DECIMAL (12,2) of Oracle, the precision is equal to the maximum length of the integer part 5+the number of decimal places 2+5 bits of buffer, which is equal to 12 bits. The 5-bit buffer is: the length added to the length of the data sampled from the source field, for example, there are actually 4 bits of length, but the sampling does not extract data with 4 bits of length, and only 2 bits of decimal may be extracted during sampling. Therefore, a few bits of data length can be reserved to reduce the risk of data truncation and ensure data integrity.

[0074] In the scheme of Example Three, in the case where there is no field annotation, the data type recommendation model can determine the field meaning according to the business attribute sorted in advance, according to the business dictionary (including the named annotation-internal data standard dictionary), and determine the data type corresponding to the field, so as to recommend a more accurate data type.

[0075] For example, if the data type recommendation model does not predict the data type according to the dictionary, but only according to the sampling data, the data type recommendation model may prefer to recommend a floating point type. However, for the data of the field, the floating point type is not accurate, and the DECIMAL (10,2) is accurate.

[0076] In actual production, the processing of business data often needs to synchronize the business data from a first business system to a data warehouse, process a large amount of data from multiple business systems in the data warehouse, and then synchronize the processed data to a second business system. That is, the data source type of the business data needs to be converted from the data source type (such as MySQL) of the first business system to the data source type (such as Hive) uniformly used by the data warehouse, so as to unify the data types of various business data systems, and then process the business data. After the data warehouse processes the business data, each business system converts the data from the data type of the data warehouse to the data type corresponding to the data source required by each business system. When the business data is exported from the data warehouse, the data source type of the second business system is the same as that of the first business system, such as both being MySQL systems, and the data source type (such as Hive) of the data warehouse is different from that of the first business system and the second business system. In this case, when converting from the data type of the data warehouse to the data type of the second business system, the prior art does not consider the data type of the first business system, but directly judges according to the field name or field sample value, thereby causing the selected data type to be unsuitable for the field. Alternatively, in some cases, the data source of the first business system and the data source of the second business system both have the same data type 1 (such as the MEDIUMINT type of MySQL), but the data source of the data warehouse does not have the data type 1, so when the first business system synchronizes data to the data warehouse, the data type 1 needs to be modified to a data type 2 (such as the INT type of Hive), and if the data source of the second business system also has the data type 2 (such as the first business system and the second business system are both MySQL, and MySQL includes the INT type), the prior art may directly use the data type 2 as the data type of the field, which also causes the selected data type to be unsuitable for the field, and INT occupies more storage space than MEDIUMINT.

[0077] To solve the problem that the determination manner of the data type of the data from the data warehouse to the second business system is not reasonable, the embodiments of the present application further include the following steps 1011 to 1013 before step 103, and step 103 can be implemented as step 1014.

[0078] Step 1011: Obtain the correspondence between the data type of the original source data and the data type of the source data.

[0079] Step 1012: Obtain the correspondence between the value range of the data type of the original source data and the value range of the data type of the source data.

[0080] The correspondence between the data type of the original source data and the data type of the source data and the correspondence between the value range of the data type of the original source data and the value range of the data type of the source data can be pre-stored data. For example, after the data type in the first business system is converted into the data type in the data warehouse, the correspondence of the data type can be recorded and stored. The data type in the data warehouse needs to be converted into the data type in the second business system subsequently, and the pre-stored data can be obtained at this time. In the data type conversion process, the data source in the first business system is the original source data source, the data in the first business system is the original source data, the data source in the data warehouse is the source data source, the data in the data warehouse is the source data, the data source in the second business system is the target data source, and the data in the second business system is the target data.

[0081] For example, the data type DECIMAL (10, 4) of the field price of the original source MySQL corresponds to the NUMBER (10, 4) of Oracle, the data type INT of the field page_views of MySQL corresponds to the NUMBER (10, 0) of Oracle, the data type VARCHAR2 (10) of the field status of MySQL corresponds to the VARCHAR2 (10) of Oracle, and the like.

[0082] In step 1013, the correspondence between the data type of the original source data and the data type of the source data and the correspondence between the value range of the data type of the original source data and the value range of the data type of the source data are converted into a supplementary feature vector.

[0083] The supplementary feature vector includes the semantic feature of the correspondence between the data type of the original source data and the data type of the source data and the correspondence between the value range of the data type of the original source data and the value range of the data type of the source data.

[0084] By converting the correspondence between the data type of the original source data and the data type of the source data into a supplementary feature vector in step 1013, the data type recommendation model can read the correspondence between the data source type of the original source data and the data source type of the source data, so as to determine the data source type of the target data subsequently.

[0085] In step 1014, the feature vector and the supplementary feature vector are input into the data type recommendation model to generate the target data type corresponding to the field to be mapped in the source data and the value range of the target data type.

[0086] The target end data type corresponding to the to-be-mapped field is the same as the data type of the original source end data, and the value range of the target end data type corresponding to the to-be-mapped field is the same as the value range of the data type of the original source end data.

[0087] The data type recommendation model can determine the data type of the original source end data corresponding to the to-be-mapped field in the source end data through the data source type of the target end data, the data source type of the source end data, the data source type of the original source end data, and the supplementary feature vector, and take the data type of the original source end data as the target end data type of the field. For example, the original source end data source is of the MySQL type, the data type of the field is DECIMAL (10, 4), the source end data source is of the Oracle type, the data type of the field is NUMBER (10, 4), and the target end data source is of the MySQL type. According to the corresponding relationship between the source end data type NUMBER (10, 4) and the original source end data type DECIMAL (10, 4), the data type of the target end field is DECIMAL (10, 4).

[0088] Through step 1014, the target end data type and the value range of the target end data type are determined more accurately, and the situation of data truncation or storage space waste caused by inappropriate data types is reduced, thereby improving the data processing efficiency.

[0089] The training process of the data type recommendation model will be introduced below. Referring to Figure 2 , the training process of the data type recommendation model can be implemented as the following steps 201 to 204.

[0090] Step 201, obtain basic information of data types and cross-data-source-type mapping samples, and construct a knowledge base of data types.

[0091] The data source includes at least one of the following: structured data, semi-structured data, and unstructured data.

[0092] The basic information of the data type includes at least one of the following: attributes, constraints, coding rules, and operation characteristics of the data type.

[0093] For example, the attributes of the data type include the data type name (such as VARCHAR, TIMESTAMP, INT, etc.) and the value range (such as length, precision, scale, etc.). For example, in MySQL, the data type name is VARCHAR, and the value range is 1 to 65535 bytes, and the maximum number of characters is 16383 under utf8mb4 coding. For another example, the maximum precision of the DECIMAL data type in Hive is 38.

[0094] Exemplarily, the constraints of the data type include index compatibility, indicating the index mode supported by the data type. For example, the TEXT type and the BLOB type do not support the primary key index. When the model predicts the data type of the target field, the field with the index requirement should be avoided to be mapped to the TEXT or BLOB type.

[0095] In some examples, the constraints of the data type further include one or more of the default value support, the null or non-null constraint. For example, the declaration syntax of the data type DECIMAL(M, D) is the rule of the total number of digits M and the number of decimal digits D, supports setting a default value, and allows to be empty.

[0096] Exemplarily, the coding rule of the data type includes that the field containing Chinese or visual emotional symbols (Emoji) needs to be predicted as the UTF-8 encoding. For example, the field type is NVARCHAR in Oracle, and the field type is STRING in Hive.

[0097] Exemplarily, the operation characteristics of the data type include whether to allow automatic increment, time zone processing conversion mode of time type, etc. For example, neither the TIMESTAMP nor the DATETIME in MySQL allows automatic increment, the TIMESTAMP converts the time zone, and the DATETIME does not convert the time zone.

[0098] Exemplarily, the cross-data-source type mapping sample includes multiple dimensions of data such as the source field name, the source data type, the target data type, the source data source type, the target data source type, the target data type length or precision, the source data type constraint condition, the mapping reason, the source field annotation, or the source sample value. Some examples are shown in Tables 2 to 4. Tables 2 and 3 show sample examples of structured data on both the source data and the target data, including the source data type, the target data type, the target data type length or precision, and the source data type constraint condition. Table 4 shows a sample example of unstructured data on the source data and structured data on the target data.

[0099] Table 2: Cross-data-source type mapping sample example

[0100] Table 3: Cross-data-source type mapping sample example

[0101] Table 4: Cross-data-source type mapping sample example

[0102] The field context refers to the field context of each field in each cross-data-source type mapping sample. When reading the cross-data-source type mapping sample, the model can read the field context of each field in the sample. For example, referring to Table 4, the field context of the timestamp includes age, ip, and level. For another example, referring to Table 2, assuming that the field names of the MySQL source data type are age, ip, level, and date in sequence, the field context of the ip includes [age (INT), level DECIMAL(p, s), date (DATETIME)].

[0103] The model can enhance the understanding of the field according to the field context of the field, thereby facilitating the understanding of the correspondence between the data type of the field and the target data type, so that the target data type recommended by the trained model is more accurate.

[0104] It can be understood that each source field name can correspond to multiple source sample values, and only part of the source sample values are shown in Table 4 for example.

[0105] For example, the basic information of the data type can be obtained based on public technical materials, such as the official website documents of a data manufacturer. The cross-data-source type mapping sample can be obtained based on public technical materials or through user feedback and the like.

[0106] The user feedback data can be data for modifying the data type predicted by the model, for example, the model predicts that the target type is STRING, and the user corrects it to BIGINT. The user feedback data includes the source type, the target type predicted by the model, and the target type corrected by the user, and can also include the correction reason. The model can learn the reason why the user corrects the data type according to the correction reason, and by inputting the user feedback data into the model, the model can be incrementally fine-tuned, which can improve the accuracy of the data type predicted by the model and improve the effect of the model.

[0107] In step 202, a feature vector of a data type is obtained based on a knowledge base of data types.

[0108] For example, step 202 can be implemented as steps 2021 to 2023.

[0109] In step 2021, a feature vector of a data type semantic feature is constructed based on the basic information of the data type and the cross-data-source type mapping sample.

[0110] For example, taking the feature vector of the constraint of the data type as an example, the data type supporting the primary key index is mapped to 1, and the data type not supporting the primary key index is mapped to 0. The feature vector of the data type supporting the setting of the default value is mapped to 1, and the feature vector of the data type not supporting the setting of the default value is mapped to 0. The feature vector of the data type allowing null is mapped to 1, and the feature vector of the data type not allowing null is mapped to 0. Taking the feature vector of the coding rule of the data type as an example, the feature vector of the data type including Chinese or visual emotional symbols is mapped to 1, and the feature vector of the data type not including Chinese or visual emotional symbols is mapped to 0. Taking the feature vector of the operation characteristic of the data type as an example, the feature vector of the data type allowing automatic increment is mapped to 1, and the feature vector of the data type not allowing automatic increment is mapped to 0. The time zone processing conversion mode of the time type is: the time zone is converted, which is mapped to 1, and the time zone processing conversion mode of the time type is: the time zone is not converted, which is mapped to 0.

[0111] In some embodiments, the feature vectors of different types of data type semantic features include different contents.

[0112] For the numerical data type, the data type semantic feature includes the source data type, the source data source type, and at least one of the following: the source field name, the field context, the source data type constraint condition, the mapping reason, the source field annotation, and the source sample value.

[0113] For the text data type, the data type semantic feature includes the source data type, the source data source type, and at least one of the following: the source field name, the field context, the source data type constraint condition, the mapping reason, the source field annotation, and the source sample value.

[0114] For the time data type, the data type semantic feature includes the source data type, the source data source type, and at least one of the following: the source field name, the target data source type, the target data type length or precision, the field context, the source data type constraint condition, the mapping reason, the source field annotation, the source sample value, or the time format (such as “yyyy-MM-dd”).

[0115] The source data type constraint condition of the numerical data type, the source data type constraint condition of the text data type, and the source data type constraint condition of the time data type are not the same. For example, the numerical data type constraint condition includes the value range of the data, the source data type constraint condition of the text data type includes the data format, and the source data type constraint condition of the time data type includes the time range.

[0116] By distinguishing that the field belongs to the numerical type, the text type, or the time type, and constructing the feature vector of the data type semantic feature according to the type to which each data type belongs, the model can be trained according to each data type, so that the model is more suitable for recommending the target data type, and space waste or character truncation is reduced.

[0117] Exemplarily, the feature vector conversion of the semantic feature adopts a Chinese fine-tuned BERT-base model as the encoding tool for vector conversion.

[0118] Exemplarily, the feature vector conversion of the semantic feature can also adopt an LLaMA-7B model.

[0119] In step 2022, a feature vector of a data type statistical feature is constructed based on the cross-data source type mapping sample.

[0120] The feature vector of the data type statistical feature can be determined according to the source-end sample value.

[0121] In some embodiments, the feature vectors of different types of data type statistical features include different contents.

[0122] For a text type field, the data type statistical feature includes at least one of the following: maximum length, average length, 90% quantile length, 95% quantile length, Chinese proportion, or null value proportion. The above statistical features refer to the maximum length, average length, 90% quantile length, 95% quantile length, Chinese proportion, or null value proportion of the sample in the source-end sample value, respectively.

[0123] For a numerical type field, the data type statistical feature includes at least one of the following: maximum value, minimum value, integer part maximum length, decimal part maximum length, numerical range, unique value number, or numerical distribution standard deviation. The above statistical features refer to the maximum value, minimum value, integer part maximum length, decimal part maximum length, numerical range, unique value number, or numerical distribution standard deviation of the sample in the source-end sample value, respectively.

[0124] The feature vector can be normalized by the maximum value-minimum value to the interval [0, 1].

[0125] By establishing the feature vector for the above statistical features, the model can determine the length, precision, etc. of the target-end data type. In addition, by converting fields of different forms into different feature vectors, the feature vector can be more suitable for the data type, so that the subsequent model can predict the data type more accurately.

[0126] Exemplarily, the feature vector text format is as follows: [CLS] source field name: {field_name} source field comment: {field_comment} context: {table_fields} source sample value: {sample_values} source data source type: {source_db_type} [SEP]. The format after conversion to the feature vector is: [CLS] source field name: sjhm source field comment: user mobile phone number context: [username (STRING), gender (VARCHAR), age (INT)] source sample value: ["139****1234", "188****5678"] source data source type: MySQL [SEP].

[0127] In step 2023, the feature vector of the data type semantic feature and the feature vector of the data type statistical feature are spliced to obtain the feature vector of the data type.

[0128] Splicing the feature vector of the data type semantic feature and the feature vector of the data type statistical feature can be implemented by splicing the feature vector of the data type semantic feature and the feature vector of the data type statistical feature of the source data type and the source data source type. For example, the feature vector of the data type statistical feature is arranged after the feature vector of the data type semantic feature.

[0129] By inputting the content in the knowledge base into the model, when the model performs data type matching, the information of the data type in the knowledge base can be used to determine the suitable data type to be matched and determine the output range of the data type to be matched.

[0130] Splicing the feature vector of the data type semantic feature and the feature vector of the data type statistical feature can make the feature vector of the data type have both basic information data related to the data type rule and sample data of the data type mapping, so that the prediction of the model on the data type is more accurate.

[0131] In step 203, a label is determined based on the cross-data source type mapping sample data.

[0132] The label includes a main label and an auxiliary label. The main label is used to mark the task that needs to be predicted by the model, for example, the target end data type and the value range. By setting the main label, the prediction accuracy of the target end data type by the model can be evaluated according to the main label.

[0133] The auxiliary label is used to describe the data source. For example, if the source data is structured data, the auxiliary label is the source data type; if the source data is semi-structured data or unstructured data, the auxiliary label is the source field name and the source sample value. By setting the data type of the source data or the field name and sample value of the source data as the auxiliary label, the source of the model can be obtained when verifying and testing the training effect of the model. If the prediction effect of a certain target type is low, the training of this data type can be optimized to improve the prediction effect of the model.

[0134] In step 204, the data type recommendation model is trained using the feature vector of the data type and the label.

[0135] For example, the model structure is as follows: the data of the input layer of the model is the feature vector of the data type; the hidden layer includes at least two fully connected networks, the first layer is a ReLU activation function, and the second layer is a GELU activation function, and a dropout of 0.3 probability is added to each layer to reduce overfitting; a cross-entropy loss function is used as the main loss function for field type classification; the output layer includes a main output and an auxiliary output, the main output content is the probability distribution of multiple data types, and a Softmax function is used; the auxiliary output is divided into a text field and a numerical field, if the data type is a text field, the output length prediction is output, a linear activation function is used, and the output data is an integer; if the data type is a numerical field, the output precision prediction is output, a linear activation is used, and the output data includes two integers p and s.

[0136] For another example, the data type recommendation model structure can be a model structure in the prior art, such as Baichuan, BERT, ChatGLM, etc.

[0137] The data type recommendation model can recommend multiple target data types in sequence.

[0138] In some embodiments, the top data type recommended by the model is used as the target data type.

[0139] In other embodiments, the target data type is determined according to the remaining storage space of the business platform where the target data source is located. For example, after the data is calculated from the data warehouse, the data is synchronized to the user's business platform, and the remaining storage space of the user's business platform reaches a preset value such as 10GB, the data type with the smallest space occupied among the recommended data types is selected as the preferred data type. Thus, when the user's storage space is insufficient, the user can perform the data synchronization task as much as possible, rather than failing to synchronize the data due to the large space occupied by the data type, thereby improving the robustness of the data synchronization system.

[0140] In some embodiments, the model can determine the value range of the target end data type according to the value range of the sample, for example, 2 times the maximum length of the sample. Thus, the occurrence of data truncation due to insufficient sample data can be reduced.

[0141] In some embodiments, the method of the embodiments of the present application can also improve the model according to the personalized needs of the user.

[0142] The above knowledge base can also include enterprise internal historical synchronization task records. Referring to Table 5, the enterprise internal historical synchronization task records include the field name, field annotation, field sample value, source data type, source data source type, target end data type, etc. of the enterprise internal data. The field name of the enterprise internal data can reflect the naming habits specific to the enterprise.

[0143] Table 5 Enterprise internal historical synchronization task record data

[0144] After obtaining the trained model in step 204, the model can also be improved according to the personalized needs of the user. After step 204, steps 205 and 206 are also included.

[0145] Step 205, based on the enterprise internal historical synchronization task records, constructing a feature vector of the newly added data type statistical features.

[0146] Step 206, using the feature vector of the newly added data type statistical features to perform incremental training on the output layer of the data type recommendation model, to obtain a new data type recommendation model.

[0147] Among them, the incremental training parameters include: the optimizer is AdamW, the learning rate is set to 1e-5, and the training rounds are 3, 4 or 5 rounds.

[0148] In some examples, the loss function only retains the data type classification loss and the value range loss, and the weight is set to 0.7:0.3.

[0149] By setting a lower learning rate in incremental training, parameter oscillation is reduced. By setting the training rounds to 3-5 rounds, small sample fine-tuning is achieved, and overfitting is reduced. By setting the weight ratio of the data type classification loss and the value range loss to 0.7:0.3, the data type that is more concerned in the enterprise scenario has higher prediction accuracy.

[0150] Through the above incremental training for the personalized needs of the user, the prediction of the data type has better accuracy. For example, the sample value of the enterprise "je" field is 100.50, 2000.00, etc. The model recommends FLOAT before fine-tuning, and recommends DECIMAL(10,2) after fine-tuning, which meets the accuracy requirements of the financial scenario.

[0151] In some embodiments, to improve the understanding of multi-semantics such as Chinese, pinyin, abbreviations, synonyms, etc., a multi-language / multi-word dictionary is also included in the data for training the classifier. The multi-language / multi-word dictionary includes: Chinese corresponding pinyin, abbreviations, synonyms, foreign language (such as English, French) and the like. Thus, the model can replace the original content in the training data with the content in the multi-language / multi-word dictionary (such as replacing age with age_user, replacing firstname with name1, etc.) when training, so that the trained model can adapt to source data in multiple languages and has a wider data type prediction capability.

[0152] The above describes a data type mapping method in the embodiments of the present application, and the following describes a data type mapping device (for example, a server) for executing the data type mapping method. The data type mapping device in the embodiments of the present application can implement the steps of the data type mapping method executed in the embodiments corresponding to the above Figure 1 The functions implemented by the data type mapping device can be implemented by hardware or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the above functions, which can be software and / or hardware.

[0153] In some examples, the data type mapping device provided by the embodiments of the present application includes a field information acquisition module, a vector representation module, a type prediction length estimation module, and a mapping rule output module.

[0154] The field information acquisition module is configured to obtain source data and target data source types.

[0155] The source data includes a data source type of the source data, an annotation of the source data, and content of the source data.

[0156] The vector representation module is configured to convert the source data and the target data source types into feature vectors. The feature vectors include a feature vector of semantic features of the source data, a feature vector of semantic features of the target data source types, and a feature vector of statistical features of the source data.

[0157] The type prediction length estimation module is configured to input the feature vectors into a data type recommendation model to generate a target data type corresponding to a field to be mapped in the source data and a value range of the target data type. The mapping rule output module is configured to output the target data type and the value range of the target data type.

[0158] The above modules are also configured to perform other steps in the above data type mapping method embodiments, which will not be described here.

[0159] In other examples, see Figure 3 ,like Figure 3 The diagram shows a structural schematic of a data type mapping device. The data type mapping device 60 may include an input / output module 601 and a processing module 602. The functional implementation of the processing module 602 and the input / output module 601 can be found in [reference needed]. Figure 1 The operations performed in the corresponding embodiments will not be described in detail here. For example, the processing module 602 can be used to control the sending, receiving, and acquiring operations of the input / output module 601.

[0160] The input / output module 601 is configured to acquire source data and target data source type; and output target data type and the range of values ​​for the target data type.

[0161] The processing module 602 is configured to convert source data and target data source type into feature vectors; input the feature vectors into a data type recommendation model to generate the target data type and the value range of the target data type corresponding to the field to be mapped in the source data.

[0162] The processing module 602 and the input / output module 601 can also be configured to implement other steps of the above data type mapping method, which will not be described in detail here.

[0163] The data type mapping device 60 in this application embodiment has been described above from the perspective of modular functional entities. The data type mapping device in this application embodiment will be described below from the perspective of hardware processing.

[0164] It should be noted that, Figure 3 The physical device corresponding to the input / output module 601 shown can be a transceiver, radio frequency circuit, communication module, and input / output (I / O) interface, etc., and the physical device corresponding to the processing module 602 can be a processor.

[0165] Figure 3 The devices shown can all have the following characteristics: Figure 4 The structure shown, when Figure 3 The data type mapping device 60 shown has, for example, Figure 4 When the structure shown is used, Figure 4 The processor and transceiver in the device can perform the same or similar functions as the processing module 602 and input / output module 601 provided in the aforementioned device embodiments. Figure 4 The memory storage processor in the memory needs to call the computer program when executing the above data type mapping method.

[0166] The embodiments of the present application also relate to a chip system, which comprises at least one processor and an interface circuit, the processor comprises a plurality of vector storage units, and the processor is configured to perform the interaction of instructions and / or data through the interface circuit, so that the chip system performs the method of any of the above embodiments. In a possible implementation, the chip system can also directly comprise a memory in which a computer program or computer instructions are stored. By way of example, the memory can be a volatile memory or a non-volatile memory, or can include both volatile and non-volatile memories. Among them, the non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically EPROM (EEPROM), or a flash memory. The volatile memory can be a random access memory (RAM) used as an external cache. By way of example but not limitation, many forms of RAM are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchlink DRAM (SLDRAM), and direct rambus RAM (DRRAM). The embodiments of the present application also relate to a processor comprising a plurality of storage units, configured to invoke a computer program or computer instructions stored in a memory, so that the processor performs the method of any of the above embodiments. For example, in the embodiments of the present application, the processor is an integrated circuit chip with processing capability of signals. For example, the processor can be an FPGA, a general-purpose processor, a DSP, an ASIC, or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, SoC, CPU, network processor (NP), micro controller unit (MCU), PLD, or other integrated chips, which can implement or execute the disclosed methods, steps and logic block diagrams in the embodiments of the present application. In one possible implementation, the embodiments of the present application also provide a computer readable storage medium, which stores program codes, when the program codes are executed on the computer, the computer executes the above-mentioned method embodiments.

[0167] In the above embodiments, the description of each embodiment has its own focus, and the parts not described in detail in a certain embodiment can be referred to the related description of other embodiments.

[0168] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the above-described system, device and module can refer to the corresponding process in the foregoing method embodiments, which will not be repeated here.

[0169] In several embodiments provided by the embodiments of the present application, it should be understood that the disclosed system, device and method can be implemented by other means. For example, the above-described device embodiments are only schematic, for example, the division of the modules is only a logical function division, and actual implementation can have another division manner, for example, a plurality of modules or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the shown or discussed modules can be indirect coupling or communication connection through some interfaces, devices or modules, which can be electrical, mechanical or other forms.

[0170] The modules described as separate components can or can not be physically separated, and the components shown as modules can or can not be physical modules, that is, they can be located in one place, or can be distributed on a plurality of network modules. According to actual needs, part or all of the modules can be selected to achieve the purpose of the embodiments of the present application.

[0171] In addition, each of the functional modules in each of the embodiments of the present application can be integrated in one processing module, or each of the modules can exist physically independently, or two or more of the modules can be integrated in one module. The integrated module can be realized in the form of hardware or in the form of a software functional module. When the integrated module is realized in the form of a software functional module and sold or used as an independent product, the integrated module can be stored in a computer readable storage medium.

[0172] In the above embodiments, all or part of the embodiments can be realized by software, hardware, firmware, or any combination thereof. When realized by software, all or part of the embodiments can be realized in the form of a computer program product.

[0173] The computer program product includes one or more computer instructions. When the computer program is loaded and executed on a computer, all or part of the processes or functions described in the embodiments of the present application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer readable storage medium or transferred from one computer readable storage medium to another computer readable storage medium, for example, the computer instructions can be transferred from one website, computer, server or data center to another website, computer, server or data center through wired (such as coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (such as infrared, wireless, microwave, etc.) mode. The computer readable storage medium can be any available medium that can be stored by a computer or a data storage device such as a server, data center, etc. integrated with one or more available media. The available media can be magnetic media (for example, floppy disk, hard disk, magnetic tape), optical media (for example, DVD), or semiconductor media (for example, solid state disk (SSD)) and the like.

[0174] The above describes the technical solutions provided by the embodiments of the present application in detail. The principles and implementation manners of the embodiments of the present application are described by using specific examples. The above descriptions are only used to help understand the method and core idea of the embodiments of the present application; meanwhile, for those skilled in the art, according to the ideas of the embodiments of the present application, the specific implementation manners and application ranges will be changed. Therefore, the content of the present application should not be understood as a limitation of the embodiments of the present application.

Claims

1. A data type mapping method, characterized by, The method comprises: acquiring source data and target data source type; wherein the source data comprises data source type of the source data, annotation of the source data and content of the source data; converting the source data and the target data source type into feature vectors; wherein the feature vectors comprise feature vectors of semantic features of the source data, feature vectors of semantic features of the target data source type and feature vectors of statistical features of the source data; inputting the feature vectors into a data type recommendation model to generate target data type corresponding to a field to be mapped in the source data and a value range of the target data type; outputting the target data type and the value range of the target data type.

2. The method of claim 1, wherein, if the data source type of the source data is structured data, the content of the source data comprises a plurality of fields to be mapped and at least one of the following: field context of the plurality of fields to be mapped, data type of the field, field annotation and sample value of each field to be mapped; if the data source type of the source data is semi-structured data, the content of the source data comprises a plurality of fields to be mapped and at least one of the following: field context of the plurality of fields to be mapped, field annotation and sample value of each field to be mapped; if the data source type of the source data is unstructured data, the content of the source data comprises a plurality of fields to be mapped and at least one of the following: field context of the plurality of fields to be mapped, field annotation and sample value of each field to be mapped.

3. The method according to claim 1 or 2, characterized in that, Before the feature vectors are input into the data type recommendation model, the method further comprises: acquiring a corresponding relationship between data type of original source data and data type of the source data; acquiring a corresponding relationship between value range of the data type of the original source data and value range of the data type of the source data; converting the corresponding relationship between the data type of the original source data and the data type of the source data and the corresponding relationship between the value range of the data type of the original source data and the value range of the data type of the source data into supplementary feature vectors; inputting the feature vectors and the supplementary feature vectors into the data type recommendation model to generate target data type corresponding to a field to be mapped in the source data and a value range of the target data type; wherein the target data type corresponding to the field to be mapped is the same as the data type of the original source data, and the value range of the target data type corresponding to the field to be mapped is the same as the value range of the data type of the original source data.

4. The method of claim 1, wherein, The data type recommendation model is obtained by training in the following manner: acquiring basic information of data type and cross-data source type mapping samples to construct a knowledge base of data type; the basic information of data type comprises at least one of the following: attribute, constraint, coding rule and operation characteristic of data type; obtain a feature vector of the data type based on the knowledge base of the data type; determine a label based on the cross-data-source-type mapping sample data; the label includes a main label and an auxiliary label, the main label is used to mark a task that the data type recommendation model needs to predict, and the auxiliary label is used to describe a data source; train the data type recommendation model by using the feature vector of the data type and the label.

5. The method of claim 4, wherein, The obtaining of the feature vector of the data type based on the knowledge base of the data type comprises: constructing a feature vector of a data type semantic feature based on the basic information of the data type and the cross-data-source-type mapping sample; constructing a feature vector of a data type statistical feature based on the cross-data-source-type mapping sample; splicing the feature vector of the data type semantic feature and the feature vector of the data type statistical feature to obtain the feature vector of the data type.

6. The method of claim 4, wherein, After the training of the data type recommendation model by using the feature vector of the data type and the label, the method further comprises: constructing a feature vector of a newly added data type statistical feature based on enterprise internal historical synchronization task records; incrementally training an output layer of the data type recommendation model by using the feature vector of the newly added data type statistical feature to obtain a new data type recommendation model; wherein the incremental training parameters include: the optimizer is AdamW, the learning rate is set to 1e-5, and the training rounds are 3, 4 or 5 rounds.

7. The method of claim 4, wherein, The basic information of the data type includes at least one of the following: attribute, constraint, coding rule or operation characteristic of the data type, and the cross-data-source-type mapping sample includes multiple dimensions of data in source field name, source data type, target data type, source data source type, target data source type, target data type length or precision, source data type constraint condition, mapping reason, source field annotation or source sample value.

8. A data type mapping apparatus, characterized by, The device comprises: an input and output module configured to obtain source data and target data source type; wherein the source data includes data source type, annotation and content of the source data; a processing module configured to convert the source data and the target data source type into a feature vector; wherein the feature vector includes a feature vector of semantic features of the source data, a feature vector of semantic features of the target data source type, and a feature vector of statistical features of the source data; the processing module is further configured to input the feature vector into a data type recommendation model to generate a target data type corresponding to a to-be-mapped field in the source data and a value range of the target data type; the input and output module is further configured to output the target data type and the value range of the target data type.

9. A computer-readable storage medium, characterized in that, The computer program product comprises instructions which, when executed on a computer, cause the computer to perform the method of any one of claims 1-7.

10. A computer program product comprising computer instructions, characterized in that, The computer program product comprises instructions which, when executed on a computer, cause the computer to perform the method of any one of claims 1-7. The computer program product comprises instructions which, when executed on a computer, cause the computer to perform the method of any one of claims 1-7.

Images