Data table merging and splitting method and device, equipment, medium and program product

Abstract

The invention discloses a data table merging and splitting method and device, equipment, a medium and a program product. Capturing target data meeting data acquisition requirements from a data source and filling the target data into an initial table; determining data features of the target data and determining an association rule between the target data; based on the data features and the association rules, setting a dynamic model according to business requirements; and combining the initial table based on the dynamic model to obtain a combined table and / or splitting the initial table based on the dynamic model to obtain a split table. According to the scheme, a specific mode of intelligent merging and splitting is provided, the complex problem in data table merging and splitting can be effectively solved, a fixed merging and splitting mode is solved, complex and changeable services and data environments can be flexibly coped, and intelligent merging and splitting of the data tables are achieved.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a data table merging and splitting method, device, equipment, medium and program product. Background Art

[0002] A data table is a structure that organizes and stores data in a tabular form, usually composed of rows and columns, and is used to effectively manage and process data. With the rapid development of big data technology, the management and processing of data tables has become a key link in restricting data analysis and application.

[0003] Traditional data table merging and splitting methods have many shortcomings. They mainly rely on manual judgment or simple rule matching and are difficult to adapt to complex and changing data environments and diverse processing needs.

[0004] Therefore, in view of the shortcomings of the existing technology, it is an urgent need to develop a method that can automatically adapt to data characteristics and achieve intelligent merging and splitting. Summary of the invention

[0005] The present invention provides a data table merging and splitting method, device, equipment, medium and program product to realize intelligent merging and splitting of data tables.

[0006] According to one aspect of the present invention, a method for merging and splitting data tables is provided, comprising:

[0007] Capture target data that meets data collection requirements from the data source and fill it into the initial table;

[0008] Determine data features of the target data and determine association rules between the target data;

[0009] Based on the data features and the association rules, a dynamic model is set according to business needs;

[0010] The initial tables are merged based on the dynamic model to obtain a merged table and / or the initial tables are split based on the dynamic model to obtain split tables.

[0011] Optionally, after the target data meeting the data collection requirements is captured from the data source, one or more of the following steps are further included:

[0012] Identifying and deleting duplicate data in the target data;

[0013] Filling or deleting missing values ​​in the target data;

[0014] The data format of the target data is uniformly converted into a standard format.

[0015] Optionally, the data features include field correlation and data distribution rules; accordingly, determining the data features of the target data and determining association rules between the target data include:

[0016] Determine the field correlation by constructing a split node through a decision tree or integrate multiple decision trees through a random forest and determine the field correlation through features in each decision tree;

[0017] Using a clustering algorithm, the data is divided into different groups according to different characteristics of the data, and the data distribution law is determined according to the clustering results;

[0018] At least one associated topic is determined, target data corresponding to each associated topic is determined according to the field correlation and the data distribution law, and the association rule is obtained according to the associated topic.

[0019] Optionally, setting a dynamic model according to business requirements includes:

[0020] Determine the related topic that has the highest similarity to the business requirement;

[0021] The target data corresponding to the associated subject is determined, and the dynamic model is set according to data features of the corresponding target data.

[0022] Optionally, merging the initial tables based on the dynamic model to obtain a merged table includes:

[0023] According to the association rule, at least one merging sub-rule is set for the association rule, and a merging rule is obtained according to the association rule and the merging sub-rule;

[0024] The data to be merged are extracted from the initial table through the dynamic model and the merge rule, and the data to be merged are merged into a preset merge table.

[0025] Optionally, splitting the initial table based on the dynamic model to obtain split tables includes:

[0026] According to the association rule, at least one splitting sub-rule is set for the association rule, and a splitting rule is obtained according to the association rule and the splitting sub-rule;

[0027] The initial table is split using the dynamic model and the splitting rule to obtain a split table.

[0028] Optionally, the method further comprises: performing a data consistency check and a data integrity check on the merged table and / or the split table, and when there is data inconsistency or incomplete data, adjusting the merge rule and / or the split rule.

[0029] According to another aspect of the present invention, a data table merging and splitting device is provided, comprising:

[0030] A data processing unit, used to capture target data that meets data collection requirements from a data source and fill it into an initial table;

[0031] A data analysis unit, used to determine the data features of the target data and determine the association rules between the target data;

[0032] A model setting unit, used to set a dynamic model according to business needs based on the data features and the association rules;

[0033] A merging and splitting unit is used to merge the initial tables based on the dynamic model to obtain a merged table and / or split the initial table based on the dynamic model to obtain a split table.

[0034] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising:

[0035] At least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the data table merging and splitting method described in any embodiment of the present invention.

[0036] According to another aspect of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the data table merging and splitting method described in any embodiment of the present invention when executed.

[0037] According to another aspect of the present invention, a computer program product is provided. The computer program product includes a computer program. When the computer program is executed by a processor, the data table merging and splitting method described in any embodiment of the present invention is implemented.

[0038] The technical solution of the embodiment of the present invention provides a specific method of intelligent merging and splitting, which can effectively solve the complex problems in merging and splitting of data tables, solve the fixed merging and splitting mode, and can flexibly respond to complex and changeable business and data environments to achieve intelligent merging and splitting of data tables.

[0039] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0041] Figure 1 This is a flow chart of a data table merging and splitting method provided in the first embodiment of the present invention;

[0042] Figure 2 It is a structural schematic diagram of a data table merging and splitting device provided in Embodiment 2 of the present invention;

[0043] Figure 3 It is a structural schematic diagram of an electronic device for implementing the data table merging and splitting method according to an embodiment of the present invention. DETAILED DESCRIPTION

[0044] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0045] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0046] Embodiment 1

[0047] Figure 1 This is a flow chart of a data table merging and splitting method provided in the first embodiment of the present invention. This embodiment is applicable to the case of intelligently merging and splitting data tables. The method can be executed by a data table merging and splitting device. The data table merging and splitting device can be implemented in the form of hardware and / or software. The data table merging and splitting device can be configured in an electronic device. Figure 1 As shown, the method includes:

[0048] S110, capturing target data that meets data collection requirements from the data source and filling it into the initial table.

[0049] To automatically crawl the required data from multiple data sources through data crawling tools, you need to determine the target URL (URL) of the data to be crawled. The target URL can come from different data sources such as websites and web applications. For example, if you want to crawl product prices and evaluation information on an e-commerce platform, you need to determine the URL list of each product detail page, or first find the URL of the platform's category list page, and then further obtain the specific product page link from it.

[0050] In the embodiment of the present invention, after the target data that meets the data collection requirements is captured from the data source, one or more of the following steps are also included:

[0051] Identify and remove duplicate data from target data;

[0052] Fill or delete missing values ​​in the target data;

[0053] Convert the data format of the target data into a standard format.

[0054] For duplicate data, it is usually necessary to directly delete duplicate records to avoid redundant data interfering with subsequent analysis and increasing unnecessary calculations. For example, in an e-commerce order data table, duplicate order records may be generated due to system failures and other reasons. These duplicates need to be removed and only one valid record is retained for analysis. However, in some special cases, if the duplicate records have additional meaningful identification information such as different timestamps, further analysis and processing may be required based on specific business needs, rather than simple deletion.

[0055] For numerical data, missing values ​​may appear in the form of null values, specific placeholders, etc. For categorical data, missing values ​​may also appear as null values, or special marks that do not conform to normal classification categories. Missing values ​​can be identified by viewing basic statistical information of the data, visualizing data distribution, etc.

[0056] When the missing values ​​in a record (row) account for a small proportion, and the missing data is randomly distributed and has little impact on the overall analysis, the entire row of data containing missing values ​​can be directly deleted. If the proportion of missing values ​​in a field (column) is too high (such as more than 50%), and the field is not a key factor for the analysis target, consider deleting the column. For missing values ​​in numeric fields, if the data distribution is relatively uniform, the mean or median can be used to fill. For categorical fields, the mode is often used for filling. When missing values ​​are related to other fields, the data of other complete fields can be used to predict missing values ​​by building a suitable machine learning model (such as a regression model, decision tree model, etc.).

[0057] Converting data formats refers to the process of converting raw data from one form to another form that better meets specific needs. Its purpose is to make the data better suited to subsequent data analysis, modeling, storage, and presentation, such as unifying the data format to facilitate data interaction between different systems, optimizing the data structure to meet the requirements of specific algorithms for input data, or converting complex and difficult-to-process data into a form that is easier to understand and operate.

[0058] The processed data is stored using a diversified and distributed storage architecture to ensure data security and availability.

[0059] S120: Determine data features of target data and determine association rules between target data.

[0060] Data features refer to elements that can describe the attributes or characteristics of a certain aspect of data. They are key information obtained after abstracting and refining the original data and used for further analysis, modeling, and understanding the internal laws of the data. Preferably, the target data is analyzed through NLP natural language processing and machine learning to extract data features. NLP is used to process analysis tasks related to text data, including lexical analysis: word segmentation, part-of-speech tagging, named entity recognition, etc.; syntactic analysis: analyzing the grammatical structure of sentences, constructing the syntactic tree of sentences, and clarifying the grammatical relationship of each word in the sentence; semantic analysis: mining deeper semantic information of the text, including lexical semantics, sentence semantics, and paragraph semantics. Association rules are used to reflect the association relationship between data. Machine learning technology and big data mining methods are used to discover association rules between data, providing a basis for merging and splitting.

[0061] In the embodiment of the present invention, data features include field correlation and data distribution rules; accordingly, determining the data features of target data and determining association rules between target data include:

[0062] Determine field relevance by constructing split nodes through decision trees or integrate multiple decision trees through random forests and determine field relevance through features in each decision tree;

[0063] Through clustering algorithms, data is divided into different groups according to different characteristics of the data, and the data distribution pattern is determined based on the clustering results;

[0064] At least one associated topic is determined, target data corresponding to each associated topic is determined according to field relevance and data distribution rules, and association rules are obtained according to the associated topics.

[0065] Field correlation refers to the degree of association between different fields (variables) in a data set. It reflects how changes in one field affect changes in other fields. By analyzing field correlation, we can gain a deep understanding of the structure and rules within the data, and discover valuable information hidden behind the data, thus providing an important basis for data analysis, modeling, and decision-making.

[0066] In the process of building a decision tree, the features (fields) that are most discriminatory for the classification results are selected for splitting based on information gain. The features that are preferentially selected as splitting nodes are often highly correlated with the target variable, and the relative degree of correlation between each feature and the target variable can be judged based on this.

[0067] Random forest is an integrated learning model composed of multiple decision trees. It evaluates the importance of features by comprehensively considering the average number of times a feature is selected as a split node in each decision tree and its contribution to reducing the model prediction error.

[0068] Data distribution law Data distribution law refers to the specific pattern, form and frequency characteristics of data within the value range, which reflects the internal structure and change trend of data. Clustering algorithms (such as K-means, DBSCAN, etc.) can discover the natural aggregation pattern of data in multidimensional space, thereby revealing the distribution law of data.

[0069] An associated topic refers to a category of target data with relevance. For example, if the associated topic is the identity information of employees of Company X, the target data under this associated topic may be the employee's name, employee's home address, employee's contact information, etc. After determining the associated topic, the data corresponding to the associated topic can be determined through data features. After determining multiple associated topics and the data corresponding to each associated topic, association rules can be obtained, which reflect the relationship between target data through associated topics.

[0070] S130. Based on data features and association rules, a dynamic model is set according to business needs.

[0071] The dynamic model represents the data merge / splitting corresponding to the business needs, and is used to predict the data change trend and the timing of merge / splitting. Since the actual business needs do not necessarily correspond completely to the related topics, the most similar related topics can be determined through fuzzy search. For example, if the business need is the employee information of X company, the most similar related topic is the personal information of the employees of X company. Therefore, the dynamic model is set according to the data characteristics of "personal information of employees of X company". The dynamic model is used to carry out targeted processing of "personal information of employees of X company" when merging / splitting. If there are multiple business needs, different dynamic models are set according to different business needs to carry out corresponding merge / splitting for each business need.

[0072] S140: Merge the initial tables based on the dynamic model to obtain a merged table and / or split the initial table based on the dynamic model to obtain a split table.

[0073] In an embodiment of the present invention, the initial tables are merged based on the dynamic model to obtain a merged table, including:

[0074] According to the association rule, at least one merging sub-rule is set for the association rule, and a merging rule is obtained according to the association rule and the merging sub-rule;

[0075] Through dynamic models and merging rules, the data to be merged is extracted from the initial table and merged into the preset merged table.

[0076] Before merging, the target data is first subjected to similarity measurement and context awareness. Similarity measurement uses a variety of similarity calculation methods (such as cosine similarity, Jaccard similarity, etc.) to evaluate the similarity between different data tables or fields. Accurate similarity measurement can help to mine the association relationship in the data. Context awareness is to make the merger more in line with the actual situation and improve accuracy and effectiveness based on the context information of the data table (such as time, location, business scenario, etc.). To improve the accuracy of the merger.

[0077] The merge sub-rule is to make more detailed restrictions on the merge rules under the association rules. For example, to obtain the male employee numbers of Company X, all employee numbers of Company X can be obtained according to the association rules. At this time, it is necessary to add a merge sub-rule, so that the merge rule only merges the employee numbers of male employees on the basis of all employees. The dynamic model is set based on data features. Therefore, the dynamic model and merge rules can filter out the data that meets the requirements in the initial table and merge them into a new table.

[0078] In an embodiment of the present invention, the initial table is split based on the dynamic model to obtain a split table, including:

[0079] According to the association rule, at least one splitting sub-rule is set for the association rule, and a splitting rule is obtained according to the association rule and the splitting sub-rule;

[0080] The initial table is split by using the dynamic model and splitting rules to obtain a split table.

[0081] Before splitting, you can use clustering algorithms (such as K-means, DBSCAN, etc.) to divide the records in the data table into multiple subsets, each with similar characteristics. By analyzing association rules, you can identify fields or records with strong associations to achieve refined splitting.

[0082] The split sub-rule is a more detailed restriction on the split rules under the association rules. For example, the initial table contains the target data of the names and corresponding numbers of all employees of Company X. The business requirement is to split the initial table into a data table of the names and corresponding numbers of male employees and a data table of the names and corresponding numbers of female employees. Therefore, the target data of the names and corresponding numbers of all employees of Company X are determined from the initial table through the association rules, and then further split requirements are set by setting the split sub-rule. The split operation is performed in combination with the dynamic model to split the initial table into multiple sub-tables.

[0083] In an embodiment of the present invention, the method may further include:

[0084] Perform data consistency check and data integrity check on the merged table and / or split table, and adjust the merge rules and / or split rules when there is data inconsistency or incomplete data.

[0085] Data consistency verification refers to verifying whether data in different storage locations, different data sources or different data copies follow the same rules, logic and semantics to ensure that the meaning of data expression is unified and matches each other. In actual data application scenarios, since data may come from multiple channels, undergo multiple conversions or be stored in distributed systems, inconsistencies are likely to occur. Data consistency mainly includes format consistency verification and logical consistency verification.

[0086] Data integrity verification is to confirm whether the data is complete, that is, whether the data covers all the required information, whether the records are complete, and whether there are missing values. For example, in a data table in the database, according to business logic, there should be sales order records every day. If the data of a certain day is missing, it means that the record is incomplete.

[0087] Performing data consistency and data integrity checks on merged or split data tables can ensure the availability of the data tables.

[0088] The technical solution of the embodiment of the invention provides a specific method of intelligent merging and splitting, which can effectively solve the complex problems in merging and splitting of data tables, solve the fixed merging and splitting mode, and can flexibly respond to complex and changing business and data environments, thereby realizing the intelligent merging and splitting of data tables.

[0089] Embodiment 2

[0090] Figure 2 This is a schematic diagram of the structure of a data table merging and splitting device provided in the second embodiment of the present invention. Figure 2 As shown, the device comprises:

[0091] The data processing unit 210 is used to capture target data that meets the data collection requirements from the data source and fill it into the initial table.

[0092] The data analysis unit 220 is used to determine the data features of the target data and determine the association rules between the target data.

[0093] The model setting unit 230 is used to set a dynamic model according to business requirements based on the data features and the association rules.

[0094] The merging and splitting unit 240 is configured to merge the initial tables based on the dynamic model to obtain a merged table and / or split the initial tables based on the dynamic model to obtain split tables.

[0095] The data table merging and splitting device provided in the embodiment of the present invention can execute the data table merging and splitting method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0096] Embodiment 3

[0097] Figure 3 A schematic diagram of the structure of an electronic device 10 that can be used to implement an embodiment of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.

[0098] like Figure 3As shown, the electronic device 10 includes at least one processor 11, and a memory connected to the at least one processor 11, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., wherein the memory stores a computer program that can be executed by at least one processor, and the processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage unit 18 to the random access memory (RAM) 13. In the RAM 13, various programs and data required for the operation of the electronic device 10 can also be stored. The processor 11, the ROM 12, and the RAM 13 are connected to each other through a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0099] A number of components in the electronic device 10 are connected to the I / O interface 15, including: an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0100] The processor 11 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 11 executes the various methods and processes described above, such as a data table merging and splitting method.

[0101] In some embodiments, the data table merging and splitting method can be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 18. In some embodiments, part or all of the computer program can be loaded and / or installed on the electronic device 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded into the RAM 13 and executed by the processor 11, one or more steps of the data table merging and splitting method described above can be performed. Alternatively, in other embodiments, the processor 11 can be configured to perform the data table merging and splitting method in any other appropriate manner (for example, by means of firmware).

[0102] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0103] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the computer program is executed by the processor, the functions / operations specified in the flow chart and / or block diagram are implemented. The computer program may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0104] In the context of the present invention, a computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by or in combination with an instruction execution system, device or equipment. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0105] To provide interaction with a user, the systems and techniques described herein may be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).

[0106] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0107] A computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The client and server relationship is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services.

[0108] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution of the present invention can be achieved, and this document does not limit this.

[0109] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A data table merging and splitting method, characterized in that: include: Capture target data that meets data collection requirements from the data source and fill it into the initial table; Determine data features of the target data and determine association rules between the target data; Based on the data features and the association rules, a dynamic model is set according to business needs; The initial tables are merged based on the dynamic model to obtain a merged table and / or the initial tables are split based on the dynamic model to obtain split tables.

2. The method according to claim 1, characterized in that After the target data that meets the data collection requirements is captured from the data source, one or more of the following steps are also included: Identifying and deleting duplicate data in the target data; Filling or deleting missing values ​​in the target data; The data format of the target data is uniformly converted into a standard format.

3. The method according to claim 1, characterized in that: The data features include field correlation and data distribution rules; accordingly, determining the data features of the target data and determining the association rules between the target data include: Determine the field correlation by constructing a split node through a decision tree or integrate multiple decision trees through a random forest and determine the field correlation through features in each decision tree; Using a clustering algorithm, the data is divided into different groups according to different characteristics of the data, and the data distribution law is determined according to the clustering results; At least one associated topic is determined, target data corresponding to each associated topic is determined according to the field correlation and the data distribution law, and the association rule is obtained according to the associated topic.

4. The method according to claim 3, characterized in that The step of setting a dynamic model according to business requirements includes: Determine the related topic that has the highest similarity to the business requirement; The target data corresponding to the associated subject is determined, and the dynamic model is set according to data features of the corresponding target data.

5. The method according to claim 1, characterized in that: The step of merging the initial tables based on the dynamic model to obtain a merged table includes: According to the association rule, at least one merging sub-rule is set for the association rule, and a merging rule is obtained according to the association rule and the merging sub-rule; The data to be merged are extracted from the initial table through the dynamic model and the merge rule, and the data to be merged are merged into a preset merge table.

6. The method according to claim 1, characterized in that The step of splitting the initial table based on the dynamic model to obtain a split table includes: According to the association rule, at least one splitting sub-rule is set for the association rule, and a splitting rule is obtained according to the association rule and the splitting sub-rule; The initial table is split using the dynamic model and the splitting rule to obtain a split table.

7. The method according to any one of claims 1 to 6, characterized in that: Further including: A data consistency check and a data integrity check are performed on the merged table and / or the split table. When there is data inconsistency or incomplete data, the merge rule and / or the split rule are adjusted.

8. A data table merging and splitting device, characterized in that: include: A data processing unit, used to capture target data that meets data collection requirements from a data source and fill it into an initial table; A data analysis unit, used to determine the data features of the target data and determine the association rules between the target data; A model setting unit, used to set a dynamic model according to business needs based on the data features and the association rules; A merging and splitting unit is used to merge the initial tables based on the dynamic model to obtain a merged table and / or split the initial table based on the dynamic model to obtain a split table.

9. An electronic device, characterized in that: The electronic device comprises: At least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the data table merging and splitting method described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the data table merging and splitting method according to any one of claims 1 to 7 when executed.

11. A computer program product, characterized in that The computer program product comprises a computer program, and when the computer program is executed by a processor, the computer program implements the data table merging and splitting method according to any one of claims 1 to 7.

Images