Data standardization method and device, computer equipment and storage medium

Abstract

The invention discloses a data standardization method and device, computer equipment and a storage medium, and belongs to the technical field of big data. The method comprises the steps of obtaining to-be-processed data, and identifying a data structure of the to-be-processed data; determining an adaptive data table template and a data processing rule according to the data structure of the to-be-processed data; constructing a data standardization processing engine based on the data processing rule; performing standardization processing on the to-be-processed data by using a data standardization processing engine to obtain standardized data; and importing the standardized data into the data table template to obtain a standardized data table, and storing and outputting the standardized data table. The invention further relates to the technical field of block chains, and the to-be-processed data is stored on the nodes of the block chain network. According to the method, the corresponding processing rule is matched according to the data structure of the to-be-processed data, so that the data standardization processing engine is dynamically constructed, the tedious process of traversing a huge rule set is avoided, and the mass data processing efficiency is greatly improved.

Description

Technical Field

[0001] The present application belongs to the field of big data technology, and specifically relates to a data standardization method, apparatus, computer equipment and storage medium. Background Art

[0002] In today's data-driven era, data modeling has become an important part of the core competitiveness of enterprises. Data modeling not only significantly improves the consistency and accuracy of data by defining data structures, relationships and rules, but also ensures that data quality meets the standards required for decision-making. More importantly, accurate data models can provide insight into the deep logic of business operations, transform complex data into intuitive infographics, and help decision makers quickly capture market trends, identify potential risks and opportunities, and thus develop more scientific and efficient business strategies to promote the sustainable development of enterprises.

[0003] At present, existing data standardization technologies often use pre-configured processing engines when processing multi-source data. Each data needs to be processed through a huge set of rules to match the best processing solution. This method is inefficient when processing massive amounts of data, especially in a high-concurrency environment, where performance bottlenecks are particularly prominent. This not only limits the speed of data processing, but also affects the real-time nature of data standardization and the overall responsiveness of the system. Summary of the invention

[0004] The purpose of the embodiments of the present application is to propose a data standardization method, apparatus, computer equipment and storage medium, and to provide a data standardization solution, which aims to improve rule matching efficiency and concurrent processing capabilities, and ensure rapid and accurate standardization of large-scale data.

[0005] In order to solve the above technical problems, the present application embodiment provides a data standardization method, which adopts the following technical solution:

[0006] A data normalization method, comprising:

[0007] Acquire the data to be processed and identify the data structure of the data to be processed;

[0008] Determine an adapted data table template and data processing rules according to the data structure of the data to be processed;

[0009] Building a data standardization processing engine based on the data processing rules;

[0010] Using the data standardization processing engine to perform standardization processing on the data to be processed to obtain standardized data;

[0011] The standardized data is imported into the data table template to obtain a standardized data table, and the standardized data table is stored and output.

[0012] Furthermore, the data processing rules include data cleaning rules, data extraction rules and data normalization processing rules. The step of using the data normalization processing engine to perform normalization processing on the data to be processed to obtain normalized data specifically includes:

[0013] Using the data cleaning rules to clean the data to be processed;

[0014] Extracting data entities and entity relationships from the cleaned data to be processed using the data extraction rules;

[0015] The data normalization processing rules are used to perform data normalization processing on the extracted data entities and entity relationships to obtain the standardized data.

[0016] Furthermore, the data table template includes a data output table template and a data storage table template, and the steps of importing the standardized data into the data table template to obtain a standardized data table, and storing and outputting the standardized data table specifically include:

[0017] Importing the standardized data table into the data output table template to obtain a standardized data output table;

[0018] Outputting the standardized data output table to a user terminal;

[0019] Importing the standardized data table into the data storage table template to obtain a standardized data storage table;

[0020] The standardized data storage table is stored in a preset standardized data storage library.

[0021] Furthermore, the step of constructing a data standardization processing engine based on the data processing rules specifically includes:

[0022] The data cleaning rule, the data extraction rule and the data normalization processing rule are respectively modularly packaged to obtain a data cleaning module, a data extraction module and a data normalization processing module;

[0023] A processing engine environment is constructed, and the data cleaning module, the data extraction module and the data normalization processing module are installed into the processing engine environment to obtain the data standardization processing engine.

[0024] Furthermore, before the step of obtaining the data to be processed and identifying the data structure of the data to be processed, the method further includes:

[0025] Acquire historical data from a multi-source database to obtain first historical data, wherein the first historical data includes data obtained from different preset data sources;

[0026] Acquire first historical standardized data corresponding to the first historical data, and identify data information of the first historical standardized data, the data information including standardized field name, field comment, field type, field length, field precision, field code table, and field security level;

[0027] A data table template for the first historical data is defined according to the data information of the first historical standardized data.

[0028] Further, after the step of defining the data table template of the first historical data according to the data information of the first historical standardized data, the method further includes:

[0029] Acquire a standardization processing program corresponding to the first historical standardized data to obtain a first standardization processing program, wherein the first standardization processing program includes a data cleaning program, a data extraction program, and a data normalization processing program;

[0030] Obtaining historical data cleaning rules from the data cleaning program;

[0031] Obtaining historical data extraction rules from the data extraction program;

[0032] Obtaining historical data normalization processing rules from the data normalization processing program;

[0033] The historical data cleaning rule, the historical data extraction rule and the historical data normalization processing rule are integrated to obtain the data processing rule.

[0034] Further, after the step of defining the data table template of the first historical data according to the data information of the first historical standardized data, the method further includes:

[0035] Monitoring the data connection interface of the multi-source database;

[0036] When the data docking interface is connected to a new data source, historical data of the new data source is acquired to obtain second historical data;

[0037] Acquire a standardized processing program corresponding to the second historical data to obtain a second standardized processing program;

[0038] executing the second standardization processing program to process the second historical data to obtain second historical standardized data;

[0039] The data information of the second historical standardized data is identified, and a data table template of the second historical data is defined according to the data information of the second historical standardized data.

[0040] In order to solve the above technical problems, the embodiment of the present application further provides a data standardization device, which adopts the following technical solution:

[0041] A data standardization device, comprising:

[0042] A structure recognition module is used to obtain the data to be processed and identify the data structure of the data to be processed;

[0043] A template rule module, used to determine an adapted data table template and data processing rules according to the data structure of the data to be processed;

[0044] A processing engine module, used to construct a data standardization processing engine based on the data processing rules;

[0045] A standardization processing module, used to use the data standardization processing engine to perform standardization processing on the data to be processed to obtain standardized data;

[0046] The storage and output module is used to import the standardized data into the data table template to obtain a standardized data table, and store and output the standardized data table.

[0047] In order to solve the above technical problems, the embodiment of the present application further provides a computer device, which adopts the following technical solution:

[0048] A computer device comprises a memory and a processor, wherein the memory stores computer-readable instructions, and the processor implements the steps of any of the above-mentioned data standardization methods when executing the computer-readable instructions.

[0049] In order to solve the above technical problems, the embodiment of the present application further provides a computer-readable storage medium, which adopts the following technical solution:

[0050] A computer-readable storage medium stores computer-readable instructions, which, when executed by a processor, implement the steps of any of the data standardization methods described above.

[0051] Compared with the prior art, the embodiments of the present application have the following beneficial effects:

[0052] The present application discloses a data standardization method, device, computer equipment and storage medium, which belongs to the field of big data technology. The present application obtains the data to be processed and identifies the data structure of the data to be processed; determines the adapted data table template and data processing rules according to the data structure of the data to be processed; constructs a data standardization processing engine based on the data processing rules; uses the data standardization processing engine to perform standardization processing on the data to be processed to obtain standardized data; imports the standardized data into the data table template to obtain a standardized data table, and stores and outputs the standardized data table. The present application matches the corresponding processing rules according to the data structure of the data to be processed to dynamically construct a data standardization processing engine, avoiding the tedious process of traversing a huge set of rules, and significantly improving the efficiency of massive data processing, especially in high-concurrency scenarios, effectively alleviating performance bottlenecks, and ensuring the real-time nature of data processing and the high responsiveness of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] In order to more clearly illustrate the scheme in the present application, a brief introduction is given below to the drawings required for use in the description of the embodiments of the present application. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0054] Figure 1 An exemplary system architecture diagram is shown in which the present application can be applied;

[0055] Figure 2 A flow chart showing an embodiment of a data standardization method according to the present application;

[0056] Figure 3 A schematic structural diagram of an embodiment of a data standardization device according to the present application is shown;

[0057] Figure 4 A schematic structural diagram of an embodiment of a computer device according to the present application is shown. DETAILED DESCRIPTION

[0058] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by technicians in the technical field of the present application; the terms used in the specification of the application herein are only for the purpose of describing specific embodiments and are not intended to limit the present application; the terms "including" and "having" and any variations thereof in the specification and claims of the present application and the above-mentioned drawings are intended to cover non-exclusive inclusions. The terms "first", "second", etc. in the specification and claims of the present application or the above-mentioned drawings are used to distinguish different objects, not to describe a specific order.

[0059] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0060] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the accompanying drawings.

[0061] like Figure 1 As shown, the system architecture 100 may include terminal devices 101, 102, 103, a network 104 and a server 105. The network 104 is used to provide a medium for communication links between the terminal devices 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0062] Users can use terminal devices 101, 102, 103 to interact with server 105 through network 104 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 101, 102, 103, such as web browser applications, shopping applications, search applications, instant messaging tools, email clients, social platform software, etc.

[0063] Terminal devices 101, 102, 103 can be various electronic devices with display screens and supporting web browsing, including but not limited to smart phones, tablet computers, e-book readers, MP3 players (Moving Picture Experts Group Audio Layer III), MP4 (Moving Picture Experts Group Audio Layer IV) players, laptop computers, desktop computers, etc.

[0064] Server 105 can be a server that provides various services, such as a background server that provides support for pages displayed on terminal devices 101, 102, and 103. The server can be an independent server or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms.

[0065] It should be noted that the data standardization method provided in the embodiment of the present application is generally executed by a server, and accordingly, the data standardization device is generally set in the server.

[0066] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0067] Continue to refer Figure 2 , shows a flow chart of an embodiment of the data standardization method according to the present application. The embodiment of the present application can acquire and process relevant data based on artificial intelligence technology. Among them, artificial intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0068] The basic technologies of artificial intelligence generally include sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics and other technologies. Artificial intelligence software technology mainly includes computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning. The data standardization method includes the following steps:

[0069] S201, obtaining data to be processed and identifying the data structure of the data to be processed.

[0070] Specifically, we first obtain the data sets to be processed. Then, we conduct in-depth analysis of these data sets to identify their internal data structures. Data structure is the organizational form of data, which determines how data is stored, accessed, and modified. Identifying data structure is a key step in data processing.

[0071] In data processing, different data structures will directly affect the processing strategy. For example, text data is often represented as a sequence of characters, which is suitable for text processing methods such as word segmentation and stop word removal; relational data exists in the form of tables, containing multiple fields and records, which is suitable for SQL queries and JOIN operations; and graph data, such as user relationships in social networks, is represented by nodes and edges, which is more suitable for graph algorithms such as path finding and community detection.

[0072] S202: Determine an adapted data table template and data processing rules according to the data structure of the data to be processed.

[0073] Specifically, the system selects appropriate data table templates and data processing rules based on the data structure of the data to be processed. The data structure determines the way the data is organized, which in turn affects the efficiency of data storage, query, and analysis. Therefore, the system designs adaptive data table templates for different data structures, which can optimize the data storage structure and improve data access efficiency. At the same time, formulating targeted data processing rules can ensure that the data remains accurate, consistent, and complete during the processing process.

[0074] S203, constructing a data standardization processing engine based on data processing rules.

[0075] Specifically, data standardization is a key step to ensure data quality and improve data availability. By building a data standardization processing engine based on data processing rules, the system can automatically execute preset data processing rules, such as data cleaning, conversion, and verification, to achieve normalization and unified processing of raw data. The flexibility and scalability of the data standardization processing engine also enable it to adapt to changes in different scenarios and needs. By dynamically building a data standardization processing engine, the tedious process of traversing a large set of rules is avoided, which greatly improves the efficiency of massive data processing, especially in high-concurrency scenarios, effectively alleviating performance bottlenecks and ensuring the real-time nature of data processing and the high responsiveness of the system.

[0076] S204, using a data standardization processing engine to perform standardization processing on the data to be processed to obtain standardized data.

[0077] Specifically, after obtaining the data standardization processing engine corresponding to the data to be processed, the data to be processed is input into the data standardization processing engine, which will perform standardization operations such as cleaning, conversion, verification, and normalization on the data according to the preset data processing rules. This process aims to eliminate redundancy, errors, and inconsistencies in the data to ensure the accuracy and availability of the data. Ultimately, the data processed by the engine will become standardized data.

[0078] S205, importing the standardized data into the data table template to obtain a standardized data table, and storing and outputting the standardized data table.

[0079] Specifically, the system standardized data is imported into the data table template determined according to the data structure of the data to be processed, forming a structured and standardized data table. This step not only makes the data more orderly and efficient in physical storage, but also facilitates subsequent data query and analysis. After the import is completed, the standardized data table will be stored in the corresponding database and can be output as needed for use by other systems or applications.

[0080] In the above embodiments, the present application matches corresponding processing rules according to the data structure of the data to be processed to dynamically construct a data standardization processing engine, thereby avoiding the tedious process of traversing a huge set of rules, greatly improving the efficiency of massive data processing, especially in high-concurrency scenarios, effectively alleviating performance bottlenecks, and ensuring the real-time nature of data processing and the high responsiveness of the system.

[0081] Furthermore, the data processing rules include data cleaning rules, data extraction rules and data normalization processing rules. The data to be processed is standardized using a data standardization processing engine to obtain standardized data, which specifically includes:

[0082] Use data cleaning rules to clean the data to be processed;

[0083] Use data extraction rules to extract data entities and entity relationships from the cleaned data to be processed;

[0084] Use data normalization processing rules to normalize the extracted data entities and entity relationships to obtain standardized data.

[0085] In this embodiment, the system integrates data cleaning, extraction and normalization processing rules into the data standardization processing engine to achieve efficient standardization of complex data. Taking financial data as an example, a standardization processing engine that matches the data structure of the financial data is constructed. When performing standardization processing, the standardization processing engine first removes invalid or erroneous data to ensure data quality; then, it extracts key financial entities (such as transaction parties, amounts, etc.) and their relationships to build a logical framework between data; finally, through normalization processing, the data format and dimension are unified, so that financial data can be seamlessly connected between different systems. This process not only improves data processing efficiency, but also enhances the comparability and availability of data.

[0086] Through the above steps, this application has achieved comprehensive standardization of data processing by integrating cleaning, extraction and normalization rules, effectively improving data quality, ensuring data accuracy, and overall improving data processing efficiency and data value. Among them, extracting key entities and relationships enhances data relevance; normalization processing unifies data formats for cross-platform applications.

[0087] Furthermore, the data table template includes a data output table template and a data storage table template, and the steps of importing standardized data into the data table template to obtain a standardized data table, and storing and outputting the standardized data table specifically include:

[0088] Import the standardized data table into the data output table template to obtain a standardized data output table;

[0089] Outputting the standardized data output table to the user terminal;

[0090] Import the standardized data table into the data storage table template to obtain a standardized data storage table;

[0091] Store the standardized data storage table in the preset standardized data repository.

[0092] In this embodiment, the system realizes the flexible configuration and efficient management of standardized data tables by designing data output and storage table templates. For example, for financial data, the standardized data output table facilitates users to intuitively obtain cleaned, extracted and normalized financial information, such as market trends, risk assessments, etc., to help make quick decisions; at the same time, the standardized data storage table is safely stored in the preset data repository, which not only ensures the security of the data, but also promotes the effective accumulation and reuse of data assets. This process not only optimizes the financial data processing process, but also improves the availability and value of data.

[0093] In actual cases, the design of data output tables and data storage tables will vary according to specific business requirements and data characteristics. Data storage tables are mainly used to persist data in the database, including the original records, detailed information and various attributes of the data, and are the basis for data analysis and processing. Data output tables are data tables that are extracted, organized and presented to users from data storage tables according to specific requirements. They contain data summaries, analysis results or displays in specific formats to facilitate user understanding and use.

[0094] Through the above steps, this application realizes the efficient conversion, storage and output of standardized data by applying the data output and storage table template. The data output table template ensures that the data is displayed in a user-friendly manner and improves data availability; the data storage table template ensures the consistency and security of the data, which is convenient for subsequent analysis and reuse.

[0095] Furthermore, the steps of constructing a data standardization processing engine based on the data processing rules specifically include:

[0096] The data cleaning rules, data extraction rules and data normalization processing rules are respectively modularly encapsulated to obtain a data cleaning module, a data extraction module and a data normalization processing module;

[0097] Build a processing engine environment, and install the data cleaning module, data extraction module and data normalization processing module into the processing engine environment to obtain a data standardization processing engine.

[0098] In this embodiment, the system effectively builds a data standardization processing engine by modularly encapsulating data processing rules. Decomposing the complex cleaning, extraction, and normalization processes into independent modules not only improves the maintainability and reusability of the code, but also makes the data processing process clearer and more controllable. By integrating these modules into a unified processing engine environment, the automation and standardization of the data processing process is achieved, significantly improving the efficiency and accuracy of data processing.

[0099] Rule modular encapsulation refers to decomposing complex data processing rules (including data cleaning rules, data extraction rules and data normalization processing rules) into several independent and reusable modules. Through modular design, each processing rule is encapsulated into a unit with clear interfaces and functions, thereby simplifying the complexity of the system and improving the maintainability and scalability of the code.

[0100] Through the above steps, this application has built a flexible and efficient data standardization processing engine by modularly encapsulating data processing rules. The modular design simplifies the complexity of the system, improves code reusability and maintainability, and the construction of the processing engine realizes an automated and standardized data processing process, significantly improving data processing efficiency and accuracy.

[0101] Furthermore, before the step of obtaining the data to be processed and identifying the data structure of the data to be processed, the method further includes:

[0102] Acquire historical data from a multi-source database to obtain first historical data, wherein the first historical data includes data obtained from different preset data sources;

[0103] Acquire first historical standardized data corresponding to the first historical data, and identify data information of the first historical standardized data, the data information including standardized field name, field comment, field type, field length, field precision, field code table, and field security level;

[0104] A data table template for the first historical data is defined according to the data information of the first historical standardized data.

[0105] In this embodiment, before data standardization processing, the system integrates historical data from multi-source databases and obtains the corresponding standardized data. It deeply analyzes the detailed information of the standardized data, such as field name, comment, type, length, precision, code table and security level, so as to define the data table template corresponding to the standardized data according to these data information. Based on these detailed data information, the standardized data table template of historical data is defined. The standardized data table template lays a solid foundation for subsequent data cleaning, extraction and normalization processing, effectively improves the standardization and efficiency of data processing, and promotes the effective management and utilization of data assets.

[0106] Through the above steps, this application accurately defines the data table template by integrating multi-source historical data and obtaining its standardized information. This process enhances the consistency and comparability of data, provides a standardized basis for subsequent data processing, and improves the standardization and efficiency of data management.

[0107] Furthermore, after the step of defining a data table template of the first historical data according to the data information of the first historical standardized data, the method further includes:

[0108] Acquire a standardization processing program corresponding to the first historical standardized data to obtain a first standardization processing program, wherein the first standardization processing program includes a data cleaning program, a data extraction program, and a data normalization processing program;

[0109] Get historical data cleaning rules from the data cleaning program;

[0110] Obtain historical data extraction rules from the data extraction program;

[0111] Obtain historical data normalization processing rules from the data normalization processing program;

[0112] The historical data cleaning rules, historical data extraction rules and historical data normalization processing rules are integrated to obtain the data processing rules.

[0113] In this embodiment, after defining the historical data table template, the system further obtains the standardized processing procedures corresponding to the standardized data, and extracts specific data processing rules from them to form a set of standardized data processing rules corresponding to the historical data, so as to facilitate the subsequent construction of a data standardization processing engine that matches the data structure of the historical data. This step not only ensures the integrity and accuracy of the data processing rules, but also realizes the effective inheritance and reuse of historical data processing experience. By integrating the cleaning, extraction and normalization processing rules, a comprehensive set of data processing specifications is constructed, which effectively improves the standardization and automation level of data processing, and promotes the improvement of data quality and the mining of data value.

[0114] Through the above steps, this application extracts and integrates comprehensive data processing rules by obtaining the processing procedures of historical standardized data, improves the standardization of data processing, and ensures the consistency and accuracy of data quality. At the same time, the automated processing process improves work efficiency and reduces the risk of human error.

[0115] Furthermore, after the step of defining a data table template of the first historical data according to the data information of the first historical standardized data, the method further includes:

[0116] Monitor the data docking interfaces of multi-source databases;

[0117] When the data docking interface is connected to a new data source, historical data of the new data source is obtained to obtain second historical data;

[0118] Acquire a standardized processing program corresponding to the second historical data to obtain a second standardized processing program;

[0119] executing a second standardization processing procedure to process the second historical data to obtain second historical standardized data;

[0120] Data information of the second historical standardized data is identified, and a data table template of the second historical data is defined according to the data information of the second historical standardized data.

[0121] In this embodiment, after defining the first historical data table template, the data docking interface is monitored, and when a new data source is connected, a data table template that matches the data structure of the new data is defined, further expanding its flexibility and scalability. Through continuous monitoring of the data docking interface of the multi-source database, it is possible to respond immediately to the access of new data sources. When a new data source is added, its historical data is automatically obtained, and the corresponding standardized processing procedures are applied to ensure that a data table template matching the new data is generated in a timely manner. This process not only ensures the standardization and consistency of data, but also realizes the dynamic update of the data table template to adapt to the ever-changing data environment, effectively improving the adaptability and response speed of the data processing system.

[0122] Through the above steps, this application expands the coverage of data standardization processing by monitoring data interfaces and dynamically processing new data sources. It can instantly adapt to changes in data sources, ensure rapid standardization of new data and build corresponding data table templates. This mechanism enhances the flexibility and responsiveness of data processing.

[0123] In the above embodiment, the present application discloses a data standardization method, which belongs to the field of big data technology. The present application obtains the data to be processed and identifies the data structure of the data to be processed; determines the adapted data table template and data processing rules according to the data structure of the data to be processed; builds a data standardization processing engine based on the data processing rules; uses the data standardization processing engine to perform standardization processing on the data to be processed to obtain standardized data; imports the standardized data into the data table template to obtain a standardized data table, and stores and outputs the standardized data table. The present application matches the corresponding processing rules according to the data structure of the data to be processed to dynamically build a data standardization processing engine, avoiding the tedious process of traversing a huge set of rules, and significantly improving the efficiency of massive data processing, especially in high concurrency scenarios, effectively alleviating performance bottlenecks, and ensuring the real-time nature of data processing and the high responsiveness of the system.

[0124] In this embodiment, the electronic device (eg, Figure 1 The server shown in the figure) can receive instructions or obtain data through a wired connection or a wireless connection. It should be noted that the above wireless connection method may include but is not limited to 3G / 4G connection, WiFi connection, Bluetooth connection, WiMAX connection, Zigbee connection, UWB (ultra wideband) connection, and other wireless connection methods currently known or to be developed in the future.

[0125] It should be emphasized that in order to further ensure the privacy and security of the above-mentioned data to be processed, the above-mentioned data to be processed can also be stored in a node of a blockchain.

[0126] The blockchain referred to in this application is a new application model of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanism, encryption algorithm, etc. Blockchain is essentially a decentralized database, a string of data blocks generated by cryptographic methods. Each data block contains a batch of network transaction information, which is used to verify the validity of its information (anti-counterfeiting) and generate the next block. Blockchain can include the underlying blockchain platform, platform product service layer, and application service layer.

[0127] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing related hardware through computer-readable instructions, and the computer-readable instructions can be stored in a computer-readable storage medium. When the computer-readable instructions are executed, they can include the processes of the embodiments of the above-mentioned methods. Among them, the aforementioned storage medium can be a non-volatile storage medium such as a magnetic disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).

[0128] It should be understood that, although the steps in the flowchart of the accompanying drawings are displayed in sequence as indicated by the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least a part of the steps in the flowchart of the accompanying drawings may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be executed in turn or alternately with other steps or at least a part of the sub-steps or stages of other steps.

[0129] Further references Figure 3 , as a response to the above Figure 2In order to realize the method shown in the figure, the present application provides an embodiment of a data standardization device, and the device embodiment is Figure 2 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0130] like Figure 3 As shown, the data standardization device 300 described in this embodiment includes:

[0131] The structure identification module 301 is used to obtain the data to be processed and identify the data structure of the data to be processed;

[0132] The template rule module 302 is used to determine the adapted data table template and data processing rules according to the data structure of the data to be processed;

[0133] The processing engine module 303 is used to construct a data standardization processing engine based on data processing rules;

[0134] A standardization processing module 304 is used to use a data standardization processing engine to perform standardization processing on the data to be processed to obtain standardized data;

[0135] The storage and output module 305 is used to import the standardized data into the data table template to obtain a standardized data table, and store and output the standardized data table.

[0136] Furthermore, the data processing rules include data cleaning rules, data extraction rules and data normalization processing rules. The normalization processing module 304 is specifically used to:

[0137] Use data cleaning rules to clean the data to be processed; use data extraction rules to extract data entities and entity relationships from the cleaned data to be processed; use data normalization processing rules to normalize the extracted data entities and entity relationships to obtain standardized data.

[0138] Furthermore, the data table template includes a data output table template and a data storage table template, and the storage output module 305 is specifically used for:

[0139] Import the standardized data table into the data output table template to obtain a standardized data output table; output the standardized data output table to the user terminal; import the standardized data table into the data storage table template to obtain a standardized data storage table; store the standardized data storage table in a preset standardized data storage library.

[0140] Furthermore, the processing engine module 303 is specifically configured to:

[0141] The data cleaning rules, data extraction rules and data normalization processing rules are modularly encapsulated respectively to obtain the data cleaning module, data extraction module and data normalization processing module; a processing engine environment is built, and the data cleaning module, data extraction module and data normalization processing module are installed in the processing engine environment to obtain a data standardization processing engine.

[0142] Furthermore, the data standardization device 300 further includes a first template definition module, and the first template definition module is used to:

[0143] Acquire historical data from a multi-source database to obtain first historical data, wherein the first historical data includes data obtained from preset different data sources; acquire first historical standardized data corresponding to the first historical data, and identify data information of the first historical standardized data, wherein the data information includes standardized field names, field comments, field types, field lengths, field precisions, field code tables, and field security levels; define a data table template for the first historical data according to the data information of the first historical standardized data.

[0144] Furthermore, the data standardization device 300 further includes a rule acquisition module, which is used to:

[0145] Obtain a standardization processing program corresponding to the first historical standardized data to obtain a first standardization processing program, wherein the first standardization processing program includes a data cleaning program, a data extraction program and a data normalization processing program; obtain historical data cleaning rules from the data cleaning program; obtain historical data extraction rules from the data extraction program; obtain historical data normalization processing rules from the data normalization processing program; integrate the historical data cleaning rules, historical data extraction rules and historical data normalization processing rules to obtain data processing rules.

[0146] Furthermore, the data standardization device 300 further includes a second template definition module, and the second template definition module is used to:

[0147] The data docking interface of the multi-source database is monitored; when the data docking interface is connected to a new data source, the historical data of the new data source is obtained to obtain the second historical data; the standardized processing program corresponding to the second historical data is obtained to obtain the second standardized processing program; the second standardized processing program is executed to process the second historical data to obtain the second historical standardized data; the data information of the second historical standardized data is identified, and a data table template of the second historical data is defined according to the data information of the second historical standardized data.

[0148] In the above embodiment, the present application discloses a data standardization device, which belongs to the field of big data technology. The present application obtains the data to be processed and identifies the data structure of the data to be processed; determines the adapted data table template and data processing rules according to the data structure of the data to be processed; builds a data standardization processing engine based on the data processing rules; uses the data standardization processing engine to perform standardization processing on the data to be processed to obtain standardized data; imports the standardized data into the data table template to obtain a standardized data table, and stores and outputs the standardized data table. The present application matches the corresponding processing rules according to the data structure of the data to be processed to dynamically build a data standardization processing engine, avoiding the tedious process of traversing a huge set of rules, and significantly improving the efficiency of massive data processing, especially in high concurrency scenarios, effectively alleviating performance bottlenecks, and ensuring the real-time nature of data processing and the high responsiveness of the system.

[0149] To solve the above technical problems, the present application also provides a computer device. Figure 4 , Figure 4 This is a basic structural block diagram of the computer device in this embodiment.

[0150] The computer device 4 includes a memory 41, a processor 42, and a network interface 43 that are interconnected through a system bus. It should be noted that the figure only shows a computer device 4 with components 41-43, but it should be understood that it is not required to implement all the components shown, and more or fewer components can be implemented instead. Among them, those skilled in the art can understand that the computer device here is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes but is not limited to microprocessors, application specific integrated circuits (Application Specific Integrated Circuit, ASIC), programmable gate arrays (Field-Programmable Gate Array, FPGA), digital processors (Digital Signal Processor, DSP), embedded devices, etc.

[0151] The computer device may be a computing device such as a desktop computer, a notebook, a PDA, a cloud server, etc. The computer device may interact with a user through a keyboard, a mouse, a remote controller, a touch pad, or a voice control device.

[0152] The memory 41 includes at least one type of readable storage medium, and the readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (for example, SD or DX memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, disk, optical disk, etc. In some embodiments, the memory 41 can be an internal storage unit of the computer device 4, such as a hard disk or memory of the computer device 4. In other embodiments, the memory 41 can also be an external storage device of the computer device 4, such as a plug-in hard disk equipped on the computer device 4, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (FlashCard), etc. Of course, the memory 41 can also include both the internal storage unit of the computer device 4 and its external storage device. In this embodiment, the memory 41 is generally used to store the operating system and various application software installed on the computer device 4, such as computer-readable instructions of the data standardization method, etc. In addition, the memory 41 can also be used to temporarily store various types of data that have been output or are to be output.

[0153] The processor 42 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chips in some embodiments. The processor 42 is generally used to control the overall operation of the computer device 4. In this embodiment, the processor 42 is used to run computer-readable instructions stored in the memory 41 or process data, such as computer-readable instructions for running the data normalization method.

[0154] The network interface 43 may include a wireless network interface or a wired network interface. The network interface 43 is generally used to establish a communication connection between the computer device 4 and other electronic devices.

[0155] In the above embodiment, the present application discloses a computer device, which belongs to the field of big data technology. The present application obtains the data to be processed and identifies the data structure of the data to be processed; determines the adapted data table template and data processing rules according to the data structure of the data to be processed; builds a data standardization processing engine based on the data processing rules; uses the data standardization processing engine to perform standardization processing on the data to be processed to obtain standardized data; imports the standardized data into the data table template to obtain a standardized data table, and stores and outputs the standardized data table. The present application matches the corresponding processing rules according to the data structure of the data to be processed to dynamically build a data standardization processing engine, avoiding the tedious process of traversing a huge set of rules, and significantly improving the efficiency of massive data processing, especially in high concurrency scenarios, effectively alleviating performance bottlenecks, and ensuring the real-time nature of data processing and the high responsiveness of the system.

[0156] The present application also provides another embodiment, namely, providing a computer-readable storage medium, wherein the computer-readable storage medium stores computer-readable instructions, and the computer-readable instructions can be executed by at least one processor to enable the at least one processor to perform the steps of the data standardization method as described above.

[0157] In the above embodiment, the present application discloses a computer-readable storage medium, which belongs to the field of big data technology. The present application obtains the data to be processed and identifies the data structure of the data to be processed; determines the adapted data table template and data processing rules according to the data structure of the data to be processed; builds a data standardization processing engine based on the data processing rules; uses the data standardization processing engine to perform standardization processing on the data to be processed to obtain standardized data; imports the standardized data into the data table template to obtain a standardized data table, and stores and outputs the standardized data table. The present application matches the corresponding processing rules according to the data structure of the data to be processed to dynamically build a data standardization processing engine, avoiding the tedious process of traversing a huge set of rules, and significantly improving the efficiency of massive data processing, especially in high concurrency scenarios, effectively alleviating performance bottlenecks, and ensuring the real-time nature of data processing and the high responsiveness of the system.

[0158] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in each embodiment of the present application.

[0159] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0160] Obviously, the embodiments described above are only some embodiments of the present application, rather than all embodiments. The preferred embodiments of the present application are given in the accompanying drawings, but they do not limit the patent scope of the present application. The present application can be implemented in many different forms. On the contrary, the purpose of providing these embodiments is to make the understanding of the disclosure of the present application more thorough and comprehensive. Although the present application is described in detail with reference to the aforementioned embodiments, for those skilled in the art, it is still possible to modify the technical solutions recorded in the aforementioned specific implementation methods, or to perform equivalent replacement of some of the technical features therein. Any equivalent structure made using the contents of the specification and drawings of this application, directly or indirectly used in other related technical fields, is similarly within the scope of patent protection of this application.

Claims

1. A data standardization method, characterized in that: include: Acquire the data to be processed and identify the data structure of the data to be processed; Determine an adapted data table template and data processing rules according to the data structure of the data to be processed; Building a data standardization processing engine based on the data processing rules; Using the data standardization processing engine to perform standardization processing on the data to be processed to obtain standardized data; The standardized data is imported into the data table template to obtain a standardized data table, and the standardized data table is stored and output.

2. The data standardization method according to claim 1, characterized in that: The data processing rules include data cleaning rules, data extraction rules and data normalization processing rules. The step of using the data standardization processing engine to perform standardization processing on the data to be processed to obtain standardized data specifically includes: Using the data cleaning rules to clean the data to be processed; Extracting data entities and entity relationships from the cleaned data to be processed using the data extraction rules; The data normalization processing rules are used to perform data normalization processing on the extracted data entities and entity relationships to obtain the standardized data.

3. The data standardization method according to claim 1, characterized in that: The data table template includes a data output table template and a data storage table template. The steps of importing the standardized data into the data table template to obtain a standardized data table, and storing and outputting the standardized data table specifically include: Importing the standardized data table into the data output table template to obtain a standardized data output table; Outputting the standardized data output table to a user terminal; Importing the standardized data table into the data storage table template to obtain a standardized data storage table; The standardized data storage table is stored in a preset standardized data storage library.

4. The data standardization method according to claim 2, characterized in that: The step of constructing a data standardization processing engine based on the data processing rules specifically includes: The data cleaning rule, the data extraction rule and the data normalization processing rule are respectively modularly packaged to obtain a data cleaning module, a data extraction module and a data normalization processing module; A processing engine environment is constructed, and the data cleaning module, the data extraction module and the data normalization processing module are installed into the processing engine environment to obtain the data standardization processing engine.

5. The data standardization method according to claim 4, characterized in that: Before the step of obtaining the data to be processed and identifying the data structure of the data to be processed, the method further includes: Acquire historical data from a multi-source database to obtain first historical data, wherein the first historical data includes data obtained from different preset data sources; Acquire first historical standardized data corresponding to the first historical data, and identify data information of the first historical standardized data, the data information including standardized field name, field comment, field type, field length, field precision, field code table, and field security level; A data table template for the first historical data is defined according to the data information of the first historical standardized data.

6. The data standardization method according to claim 5, characterized in that: After the step of defining the data table template of the first historical data according to the data information of the first historical standardized data, the method further includes: Acquire a standardization processing program corresponding to the first historical standardized data to obtain a first standardization processing program, wherein the first standardization processing program includes a data cleaning program, a data extraction program, and a data normalization processing program; Obtaining historical data cleaning rules from the data cleaning program; Obtaining historical data extraction rules from the data extraction program; Obtaining historical data normalization processing rules from the data normalization processing program; The historical data cleaning rule, the historical data extraction rule and the historical data normalization processing rule are integrated to obtain the data processing rule.

7. The data standardization method according to claim 5, characterized in that: After the step of defining the data table template of the first historical data according to the data information of the first historical standardized data, the method further includes: Monitoring the data connection interface of the multi-source database; When the data docking interface is connected to a new data source, historical data of the new data source is acquired to obtain second historical data; Acquire a standardized processing program corresponding to the second historical data to obtain a second standardized processing program; executing the second standardization processing program to process the second historical data to obtain second historical standardized data; The data information of the second historical standardized data is identified, and a data table template of the second historical data is defined according to the data information of the second historical standardized data.

8. A data standardization device, characterized in that: include: A structure recognition module is used to obtain the data to be processed and identify the data structure of the data to be processed; A template rule module, used to determine an adapted data table template and data processing rules according to the data structure of the data to be processed; A processing engine module, used to construct a data standardization processing engine based on the data processing rules; A standardization processing module, used to use the data standardization processing engine to perform standardization processing on the data to be processed to obtain standardized data; The storage and output module is used to import the standardized data into the data table template to obtain a standardized data table, and store and output the standardized data table.

9. A computer device, characterized in that: The method comprises a memory and a processor, wherein the memory stores computer-readable instructions, and the processor implements the steps of the data standardization method according to any one of claims 1 to 7 when executing the computer-readable instructions.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-readable instructions, which, when executed by a processor, implement the steps of the data standardization method according to any one of claims 1 to 7.

Images