Data optimization method and device and electronic equipment

Abstract

The invention discloses a data optimization method and device and electronic equipment. The method comprises the steps that user data and enterprise data are obtained, the user data are original data in a system, and the enterprise data are data meeting enterprise standards; the user data and the enterprise data are compared with standard data, comparison results are obtained, and the standard data are data benchmarking with industry standards; the user data and the enterprise data are optimized according to the comparison result, and the data formats of the optimized user data and enterprise data are the same as the standard data. The technical problems that a traditional enterprise management system is scattered in data, non-uniform in treatment standard and high in data use difficulty are solved.

Description

Technical Field

[0001] The present application relates to the field of data processing, and in particular to a method, device and electronic equipment for data optimization. Background Art

[0002] Enterprise management systems usually include the core functions of enterprise management and have the function of providing real-time, relevant, accurate and complete data. They are an important part of modern enterprise operations. With the increasing amount and type of enterprise data, data governance has been widely used in enterprises as a systematic approach. Data governance involves the management of data assets, and regulates the application and technical management of data in enterprises by formulating and implementing policies and processes.

[0003] Traditional enterprise management systems have problems such as data dispersion, inconsistent governance standards, and difficulty in using data, which hinder the effective integration and utilization of data.

[0004] To address the above-mentioned problems, no effective solution has been proposed yet. Summary of the invention

[0005] The embodiments of the present application provide a method, device and electronic device for data optimization to at least solve the technical problems of traditional enterprise management systems such as data dispersion, inconsistent governance standards and difficulty in data use.

[0006] According to one aspect of an embodiment of the present application, a method for data optimization is provided, including: obtaining user data and enterprise data, wherein the user data is original data in the system, and the enterprise data is data that complies with enterprise standards; comparing the user data and enterprise data with standard data respectively to obtain comparison results, wherein the standard data is data benchmarked against industry standards; optimizing the user data and enterprise data based on the comparison results, wherein the data formats of the optimized user data and enterprise data are the same as those of the standard data.

[0007] Optionally, standard data is determined by: obtaining original data standards and obtaining business requirements of the system; adjusting the original data standards according to the business requirements to obtain updated data standards, wherein the updated data standards include standards corresponding to at least one of the following data: metadata elements, naming conventions, data types, and lengths; and determining standard data based on the updated data standards.

[0008] Optionally, after obtaining user data and enterprise data, the method also includes: identifying duplicate data in the user data and / or enterprise data; deduplicating the duplicate data in the user data and / or enterprise data through a Bloom filter; identifying abnormal data in the user data and / or enterprise data, and correcting the abnormal data.

[0009] Optionally, the user data and the enterprise data are respectively compared with the standard data to obtain a comparison result, including: obtaining field information in the first data, wherein the first data is any one of the user data and the enterprise data, and the field information includes: field name, data type and length; comparing the field information in the first data with the standard field information in the standard data to obtain a comparison result, wherein the standard field information includes the standard field name, standard data type and standard length.

[0010] Optionally, user data and enterprise data are optimized based on the comparison results, including: when the comparison result indicates that the field name of the first data is inconsistent with the standard field name, modifying the field name of the first data to be consistent with the standard field name; when the comparison result indicates that the data type of the first data is inconsistent with the standard data type, converting the data type of the first data to the standard data type; when the comparison result indicates that the length of the first data is less than the standard length, determining the missing column data in the first data; adding the missing column data to the first data; when the comparison result indicates that the length of the first data is greater than the standard length, determining the redundant column data in the first data; and deleting the redundant column data from the first data.

[0011] Optionally, the method further includes: acquiring subsystems corresponding to the standard data and the business system; dividing the fields in the standard data according to the subsystems to which they belong to obtain multiple categories of standard data; and storing the multiple categories of standard data in the corresponding subsystems.

[0012] Optionally, the method also includes: obtaining optimized user data and optimized enterprise data; testing the optimized user data and optimized enterprise data to obtain a test result; if the test result is passed, determining the access rights of the optimized user data and optimized enterprise data according to the security level.

[0013] According to another aspect of an embodiment of the present application, a data optimization device is also provided, including: an acquisition module, used to acquire user data and enterprise data, wherein the user data is the original data in the system, and the enterprise data is data that complies with enterprise standards; a comparison module, used to compare the user data and enterprise data with the standard data respectively to obtain a comparison result, wherein the standard data is data benchmarked against industry standards; an optimization module, used to optimize the user data and enterprise data based on the comparison result, wherein the data format of the optimized user data and enterprise data is the same as that of the standard data.

[0014] According to another aspect of the embodiments of the present application, there is also provided an electronic device, including: a memory for storing program instructions; a processor, connected to the memory, for executing program instructions to implement the following functions: obtaining user data and enterprise data, wherein the user data is the original data in the system, and the enterprise data is data that complies with enterprise standards; comparing the user data and enterprise data with standard data respectively to obtain comparison results, wherein the standard data is data benchmarked against industry standards; optimizing the user data and enterprise data based on the comparison results, wherein the data format of the optimized user data and enterprise data is the same as the standard data.

[0015] According to another aspect of the embodiments of the present application, a non-volatile storage medium is provided, which includes a stored computer program, wherein the device where the non-volatile storage medium is located executes the above-mentioned data optimization method by running the computer program.

[0016] According to another aspect of the embodiments of the present application, a computer product is provided, including computer instructions, which implement the above-mentioned data optimization method when executed by a processor.

[0017] In an embodiment of the present application, user data and enterprise data are obtained, wherein the user data is the original data in the system and the enterprise data is the data that meets the enterprise standards; the user data and enterprise data are respectively compared with the standard data to obtain comparison results, wherein the standard data is the data benchmarked with the industry standard; the user data and enterprise data are optimized based on the comparison results, wherein the data format of the optimized user data and enterprise data is the same as the standard data, thereby achieving the purpose of data standardization and consistency, thereby achieving the technical effect of improving data quality, simplifying data use and enhancing data governance efficiency, and further solving the technical problems of traditional enterprise management systems such as data dispersion, inconsistent governance standards and difficulty in data use. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0019] Figure 1 is a hardware structure block diagram of a computer terminal for implementing a method for data optimization according to an embodiment of the present application;

[0020] Figure 2 is a flow chart of a data optimization method according to an embodiment of the present application;

[0021] Figure 3 It is a structural diagram of a data optimization device according to an embodiment of the present application. DETAILED DESCRIPTION

[0022] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.

[0023] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0024] The information collected in the embodiments of the present application is information and data authorized by the user or fully authorized by all parties, and the collection, storage, use, processing, transmission, provision, disclosure and application of the relevant data comply with the relevant laws, regulations and standards of the relevant regions, take necessary confidentiality measures, do not violate public order and good morals, and provide corresponding operation entrances for users to choose to authorize or reject automated decision-making results; if the user chooses to reject, the expert decision-making process will be entered.

[0025] First, some nouns or terms that appear in the process of explaining the embodiments of the present application are subject to the following explanations:

[0026] Data governance: refers to the integration, cleansing, classification and management of enterprise data through technical means and management processes to ensure the quality, integrity, consistency and security of data, and ultimately provide efficient data support for the company's business decisions and applications.

[0027] ETL (Extract, Transform, Load): A data integration process that includes data extraction, transformation, and loading. It is often used to convert data from multiple data sources into a unified format and import it into a data warehouse to ensure data cleanliness and consistency.

[0028] Data model: An abstract model that describes the organizational structure and relationships of data and is used to define the attributes, types, and relationships of data. In data governance, data models are often used to achieve data standardization and consistency.

[0029] The data optimization method embodiment provided in the embodiment of the present application can be executed in a mobile terminal, a computer terminal or a similar computing device. Figure 1 FIG. 1 shows a hardware structure block diagram of a computer terminal for implementing a method for data optimization. Figure 1 As shown, the computer terminal 10 may include one or more (102a, 102b, ..., 102n are used to illustrate) processors (the processor may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA), a memory 104 for storing data, and a transmission module 106 for communication functions connected via a wired and / or wireless network. In addition, it may also include: a display, a keyboard, a cursor control device, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of the I / O interface), a network interface, and a BUS bus. Those skilled in the art can understand that Figure 1 The structure shown is only for illustration and does not limit the structure of the above electronic device. Figure 1 More or fewer components as shown, or with Figure 1 Different configurations are shown.

[0030] It should be noted that the one or more processors and / or other data processing circuits described above may generally be referred to herein as "data processing circuits". The data processing circuits may be embodied in whole or in part as software, hardware, firmware, or any other combination thereof. In addition, the data processing circuit may be a single independent processing module, or may be incorporated in whole or in part into any of the other components in the computer terminal 10. As described in the embodiments of the present application, the data processing circuit acts as a processor control (e.g., selection of a variable resistor terminal path connected to an interface).

[0031] The memory 104 can be used to store software programs and modules of application software, such as program instructions / data storage devices corresponding to the data optimization method in the embodiment of the present application. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory 104, that is, realizing the above-mentioned data optimization method. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 104 may further include a memory remotely arranged relative to the processor, and these remote memories may be connected to the computer terminal 10 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0032] The transmission module 106 is used to receive or send data via a network. The specific example of the above network may include a wireless network provided by a communication provider of the computer terminal 10. In one example, the transmission module 106 includes a network adapter (Network Interface Controller, NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission module 106 can be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0033] The display may be, for example, a touch screen liquid crystal display (LCD) that enables a user to interact with a user interface of the computer terminal 10 .

[0034] It should be noted that, in some optional embodiments, the above Figure 1 The computer terminal shown may include hardware elements (including circuits), software elements (including computer code stored on a computer-readable medium), or a combination of hardware elements and software elements. It should be noted that Figure 1 This is merely one example of a particular embodiment and is intended to illustrate the types of components that may be present in the computer terminal described above.

[0035] In the above-mentioned operating environment, an embodiment of the present application provides an embodiment of a method for data optimization. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although the logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0036] Figure 2 is a flow chart of a data optimization method according to an embodiment of the present application. Figure 2As shown, the method comprises the following steps:

[0037] Step S202, obtaining user data and enterprise data, wherein the user data is the original data in the system, and the enterprise data is the data that complies with the enterprise standard.

[0038] In the above step S202, user data generally refers to raw data collected directly from the system or data source, which directly reflects the user's behavior, attributes or interactions. In some embodiments of the present application, user data may also be referred to as a user data model, which can be represented by a user data model A1. In an enterprise environment, user data may include but is not limited to the following types: registration information: such as user name, email address, phone number, etc.; transaction records: purchase history, payment details, transaction amount, etc.; user behavior data: website browsing behavior, application usage habits, search history, etc.; feedback and evaluation: customer satisfaction surveys, product evaluations, complaint records, etc. User data is characterized by diversity and complexity, and may contain a large amount of noise, repeated information or data in different formats, which needs to be optimized and standardized through data cleaning and quality control processes.

[0039] Enterprise data is a data set that complies with the internal standards and specifications of the enterprise. These data match the enterprise's specific business processes, information systems, and data strategies. In some embodiments of the present application, enterprise data may also be referred to as an enterprise data model, which can be represented by an enterprise data model A2. Enterprise data includes: standardized information: for example, user information stored in a format and data type predefined by the enterprise; integrated data: enterprise data may come from multiple data sources, and after ETL processing, it is integrated according to the enterprise data model; key business data: such as inventory information, financial data, sales performance, etc., which are directly related to the core business processes of the enterprise. The characteristics of enterprise data are standardization and consistency. They meet the internal data quality standards of the enterprise and facilitate data analysis and business decision-making.

[0040] Step S204, comparing the user data and the enterprise data with the standard data respectively to obtain a comparison result, wherein the standard data is data that is benchmarked against industry standards.

[0041] In the above step S204, when user data and enterprise data are collected, there may be inconsistencies and quality differences due to their independent sources and formats. Therefore, during the data integration and cleaning process, these data need to be compared with a predefined standard data to evaluate and adjust the structure and content of the data to a unified standard to ensure that they can be recognized, understood and used by the system.

[0042] Standard data is a data model and format developed with reference to common standards and best practices in the industry. It provides a set of fixed rules and parameters to measure and regulate the quality and consistency of user data and enterprise data. In some embodiments of the present application, standard data may also be referred to as a standard data model, which can be represented by a standard data model A0. The industry standards here include but are not limited to data integrity, data type, data format, field naming convention, data length, etc. For example, in the financial industry, standard data may specify specific requirements for data fields such as account information, transaction records, and credit scores.

[0043] Step S206, optimizing the user data and the enterprise data according to the comparison result, wherein the data formats of the optimized user data and the enterprise data are the same as the standard data.

[0044] In the above step S206, during the comparison process, the data governance platform compares the definition and content of each field in the user data model A1 and the enterprise data model A2 with the standard data model A0. Any mismatches, such as inconsistent data types, different field names, non-compliant data lengths, or data values ​​that do not meet the standards, will be recorded. Based on the identified differences, the data governance platform or data engineer will develop an optimization plan to adjust the structure and content of user data and enterprise data to match the standard data model. The optimization plan may include standardization of field names, conversion of data types, adjustment of data lengths, correction of data values, or filling of missing values.

[0045] Through the above steps S202 to S206, the data governance platform can ensure that the user data and enterprise data finally stored and used are completely consistent with the industry standard data model A0, thereby improving the standardization and overall quality of the data, reducing the problems of data silos and data inconsistency, and making data more efficient in cross-system and cross-departmental integration and sharing. Standardized data formats also help simplify data processing processes, improve the accuracy and speed of business decisions, and reduce potential risks caused by data quality issues.

[0046] In the above data optimization method, standard data is determined in the following manner: obtaining original data standards and obtaining business requirements of the system; adjusting the original data standards according to business requirements to obtain updated data standards, wherein the updated data standards include standards corresponding to at least one of the following data: metadata elements, naming conventions, data types, and lengths; determining standard data based on the updated data standards.

[0047] In some embodiments of the present application, the original data standard is usually derived from industry standards, data specifications, or data models previously adopted by the enterprise. These standards may include the definition of data fields, metadata elements, data types, and length requirements. For example, customer information in the financial industry may follow specific coding rules, such as the definition of fields such as customer ID and transaction date. Business requirements are the purpose and method of using data by an enterprise, which may involve specific fields, formats, or data relationships of the data. For example, an e-commerce platform may need to access the user's location information in real time to optimize logistics distribution, which requires ensuring the accuracy and real-time nature of the location information field. Based on the business needs of the enterprise, the data governance platform needs to adjust or revise the original data standards. For example, if the location information field defined in the original data standard uses the latitude and longitude format, and the business needs require the use of cities and postal codes, then the data standards need to be adjusted accordingly to ensure that the data can meet the actual business needs. Adjustments may include the following aspects: Metadata elements: Review and modify the description information of the field, such as field name, data type, unit, etc., to match business needs. Naming specifications: Standardize field names so that data can be uniformly identified and processed in the system. Data type: Adjust the data type of the field according to business needs, such as converting a text field to a date and time type. Length requirement: Set the length of the field to optimize storage space while ensuring data integrity. The updated data standard is an adjusted data model and format specification that reflects the business needs of the enterprise and industry best practices. This updated standard will serve as the basis for subsequent data cleaning, conversion, and storage. Standard data is a data model built based on the updated data standard, which provides a unified reference framework for the data governance process. All user data and enterprise data processed through the data governance platform should be consistent with the standard data, which means that they need to follow the metadata elements, naming specifications, data types, and length requirements in the updated data standard. The data governance platform will use this standard data model as a benchmark for comparison to standardize all collected data.

[0048] Through the above process, the data governance platform can meet the specific business needs of the enterprise while ensuring data quality, allowing data to flow smoothly in the system and be efficiently used, avoiding problems caused by data silos and data inconsistencies, and improving the business value and utilization efficiency of data.

[0049] In step S202 of the above data optimization method, after obtaining user data and enterprise data, the method also includes: identifying duplicate data in user data and / or enterprise data; deduplicating duplicate data in user data and / or enterprise data through a Bloom filter; identifying abnormal data in user data and / or enterprise data, and correcting the abnormal data.

[0050] In some embodiments of the present application, duplicate data refers to records that appear multiple times in a data set, and these records may be exactly the same or have the same values ​​in key fields. In user data and enterprise data, the presence of duplicate data can lead to inaccurate data, waste of storage space, and misleading subsequent analysis. For example, a user may register multiple times through different channels at different time points, resulting in multiple identical user records appearing in the database.

[0051] Bloom Filter is a highly space-efficient probabilistic data structure used to determine whether an element is in a set. In data deduplication, the use of Bloom filters can quickly identify duplicate data, reducing data processing time and storage requirements. It works as follows: Insertion process: For each element, use multiple different hash functions to calculate its position in the bit array, and set the bits at these positions to 1. Search process: For an element, use these hash functions to calculate the value of its position. If all bits are 1, it is possible that the element exists in the set. Since Bloom filters may have false positives (that is, an element may be mistakenly considered to exist in the set), but will not be mistakenly deleted, it is an efficient data deduplication tool when processing large amounts of data, especially when the data set is very large and cannot be fully loaded into memory.

[0052] Abnormal data refers to data points that significantly deviate from normal values ​​or expected patterns. They may be caused by data entry errors, system failures, fraud, etc. The existence of abnormal data may distort the statistical characteristics of the data and affect the accuracy of data analysis. For example, a transaction with an abnormally high amount appears in a user's transaction record, which may be caused by an input error. Once abnormal data is identified, the data governance platform will need to correct or clean up the data to ensure the integrity and accuracy of the data set. Correction methods may include the following: Data cleaning: Delete or correct obviously erroneous data points, for example, replace outliers with medians or means. Rule checking: Correct data according to predefined business rules or data integrity constraints. For example, if the value of a field exceeds a reasonable range, it can be reset to a default value or null value. Manual review: For some abnormal data that is difficult to handle through automated methods, manual review and correction may be required to ensure the accuracy of the data.

[0053] The above data preprocessing process, including identifying and processing duplicate data, as well as detecting and correcting abnormal data, is a key step for the data governance platform to ensure the quality of the data set. Through Bloom filter deduplication and abnormal data correction, the accuracy and consistency of the data can be greatly improved, the errors in data processing can be reduced, and a solid foundation can be laid for subsequent data analysis, integration and application.

[0054] In step S204 of the above data optimization method, the user data and the enterprise data are respectively compared with the standard data to obtain a comparison result, including: obtaining field information in the first data, wherein the first data is any one of the user data and the enterprise data, and the field information includes: field name, data type and length; comparing the field information in the first data with the standard field information in the standard data to obtain a comparison result, wherein the standard field information includes a standard field name, a standard data type and a standard length.

[0055] In some embodiments of the present application, the first data can be either user data or enterprise data, depending on the current stage of comparison. The data governance platform will first extract the information of all fields in the first data, including field name, data type, length, etc. The data governance platform compares the field information in the first data with the standard field information in the standard data model item by item. The standard field information contains standard definitions of field names, data types, and lengths, which are usually based on industry standards or internal enterprise specifications. The comparison process aims to check whether the fields in the first data conform to the definition of the standard data model, specifically involving the following aspects:

[0056] 1. Field name comparison: Check whether the field name in the first data is consistent with the standard field name. If the field name does not match, the field needs to be renamed to ensure data consistency;

[0057] 2. Data type comparison: Compare whether the data type of the field in the first data is the same as the standard field type. If the data type is different, the data governance platform will perform type conversion, such as converting a text type date field to a date and time type;

[0058] 3. Length comparison: Check whether the length of the field in the first data meets the standard length requirements. If the field length is too long or too short, the data governance platform will adjust the length, such as by truncating or padding to ensure compliance with the standard.

[0059] The comparison results will point out all fields that do not meet the standards, including inconsistent field names, mismatched data types, or lengths that do not meet the standards. These results will be used to guide subsequent data cleaning and conversion operations to ensure that the final data set is fully compatible with the standard data model and achieve the goal of data governance.

[0060] In step S206 of the above-mentioned data optimization method, user data and enterprise data are optimized according to the comparison result, including: when the comparison result indicates that the field name of the first data is inconsistent with the standard field name, modifying the field name of the first data to be consistent with the standard field name; when the comparison result indicates that the data type of the first data is inconsistent with the standard data type, converting the data type of the first data to the standard data type; when the comparison result indicates that the length of the first data is less than the standard length, determining the missing column data in the first data; adding the missing column data to the first data; when the comparison result indicates that the length of the first data is greater than the standard length, determining the redundant column data in the first data; and deleting the redundant column data from the first data.

[0061] In some embodiments of the present application, when the comparison result shows that the field name in the first data (i.e., user data or enterprise data) is inconsistent with the standard field name, the data governance platform will perform a standardization operation on the field name. This means that the field name in the first data is modified to be exactly the same as the field name defined in the standard data model to ensure that all data fields can be correctly identified and used in subsequent data processing and analysis processes. For example, if the user's birthday field in the standard data model is named "BirthDate", and the corresponding field in the first data is named "UserBirthday", then the platform will rename "UserBirthday" to "BirthDate". If the comparison result finds that the data type in the first data does not match the standard data type, the data governance platform will perform data type conversion. This means that all unmatched data fields in the first data are converted to the data type defined in the standard data model, such as converting a date field of text type to a date and time type. Data type conversion ensures the consistency of the data format and provides an accurate data basis for subsequent data analysis and application. When the comparison result indicates that the length of the first data (i.e., the number of columns) is less than the standard length, the system will identify and determine the missing column data in the first data. This usually means that certain fields included in the standard data model are missing in the first data. The data governance platform will ensure the integrity of the data by adding these missing fields to the first data. The added missing fields may contain default values, null values, or values ​​inferred based on business logic, depending on the nature of the field and the data governance policy. On the contrary, if the comparison result indicates that the length of the first data exceeds the standard length, that is, there is redundant column data, the data governance platform will identify and determine these redundant fields. Subsequently, the platform will delete these redundant fields from the first data to reduce data redundancy, optimize storage space, and ensure that the data structure is consistent with the standard data model. For data that requires special processing, such as complex data conversion, validation, and cleaning, Java programs are called to perform specific data processing or integration processes to ensure that the data meets standardization requirements.

[0062] The entire optimization process ensures the consistency, integrity and standardization of data, reduces the complexity of data processing, improves data quality, and facilitates subsequent data integration, analysis and cross-system sharing. By executing the above steps, the data governance platform can effectively manage data sets, ensure that all data conforms to the established data model standards, and provide more reliable and efficient data support for corporate decision-making and business operations.

[0063] In the above data optimization method, the method also includes: obtaining subsystems corresponding to standard data and business systems; dividing the fields in the standard data according to the subsystems to which they belong to obtain multiple categories of standard data; and storing the multiple categories of standard data in the corresponding subsystems.

[0064] In some embodiments of the present application, the data governance platform will obtain the defined standard data (model), which contains all the standard fields and data definitions required by the enterprise. At the same time, the platform will also identify the subsystems corresponding to each business system within the enterprise, such as the customer relationship management (CRM) system, the financial system, the supply chain management system (SCM), etc. Each subsystem may focus on different data levels or business functions. The data governance platform will divide the fields in the standard data model according to the business needs of each subsystem. This means that the platform will identify which fields are required for a specific subsystem, and then combine these fields together to form a standard data set related to the subsystem. For example, the CRM system may require basic customer information (such as name, address, contact information) and purchase history, while the financial system may pay more attention to financial-related fields such as transaction amount, payment method and date. Through the above steps, the data governance platform will generate multiple standard data sets optimized for different subsystems. Each data set contains fields related to the business needs of a specific subsystem, which helps to reduce data redundancy and improve the pertinence and efficiency of data. Finally, the data governance platform will store each optimized standard data set in the data warehouse or database of the corresponding subsystem. This storage method ensures the reasonable distribution and efficient use of data within the enterprise. For example, the optimized data set of the CRM system will be stored in the CRM-specific database, and the financial data set will be stored in the database of the financial system. In this way, each subsystem can directly access and process data that is directly related to its own business and meets the standards, greatly improving the efficiency and accuracy of data processing.

[0065] By dividing the standard data corresponding to the subsystems of the business system, data access rights can be better controlled, allowing only users of the relevant subsystems to access specific data, thereby enhancing data security. In addition, the subsystem can directly access and process data related to its own business without having to process irrelevant fields, thereby improving data processing efficiency. By storing only the fields required by the subsystem, data storage redundancy is reduced, storage space is saved, and the complexity of data processing is reduced.

[0066] In the above-mentioned data optimization method, the method also includes: obtaining optimized user data and optimized enterprise data; testing the optimized user data and optimized enterprise data to obtain a test result; when the test result is passed, determining the access rights of the optimized user data and optimized enterprise data according to the security level.

[0067] In some embodiments of the present application, after standardization processing, quality inspection and format adjustment by the data governance platform, the system will obtain optimized user data and enterprise data sets. These data sets have been cleaned, deduplicated, formatted, and compared and matched with the standard data model to ensure the consistency, completeness and accuracy of the data. The optimized data is more in line with the business needs and industry standards within the enterprise. After the data is optimized, the data governance platform will conduct further quality inspections on these data to ensure the correctness of all processing steps. The inspection may include the following aspects: Data integrity inspection: Check whether necessary fields or records are missing in the data set to ensure the integrity of the data. Data accuracy inspection: Verify whether the data accurately reflects the business situation, such as the correct date format and reasonable numerical range. Data consistency inspection: Confirm that the data is consistent with the standard data model in terms of field name, data type and length. Data repetitiveness inspection: Use tools such as Bloom filters to check again whether there are duplicate records in the data set. After the inspection process is completed, the platform will generate a detailed inspection result report, indicating any problems or non-standard areas that may exist in the data set. If the data passes all inspections, the final result is inspection pass.

[0068] After the data is verified, the data governance platform will determine the access rights of optimized user data and enterprise data based on the security level of the data. Data security levels are usually divided based on the sensitivity and importance of the data. For example, customer personal information, financial data, and sensitive business information may be classified as high-level security data, while public information or general statistical data may belong to a lower security level. The allocation of access rights, for example, involves the following aspects: User role: Different data access rights are assigned according to the user's role in the enterprise (such as administrator, data analyst, ordinary employee). Data classification: For different types of sensitive data, set different access restrictions to ensure that only authorized users can access it. Data encryption: Encrypt high-level sensitive data so that even if the data is accessed without authorization, its content cannot be read. Audit and monitoring: Establish a log and monitoring mechanism for data access to record every data access and operation for auditing and tracking.

[0069] Through the above steps, the data governance platform not only ensures the quality and consistency of data, but also ensures the security and compliance of data. Reasonable access permission settings can prevent data leakage or abuse, protect the core data assets of the enterprise, and promote the effective use of data to support the decision-making and business operations of the enterprise.

[0070] In the embodiment of the present application, the data governance platform is composed of a data processing platform core, which is responsible for interacting with the user data verification and management module, the enterprise data management and control module, and the data acquisition and integration module in the enterprise management system. This platform mainly relies on an efficient ETL (Extract, Transform, Load) algorithm to integrate data. The ETL algorithm is designed to clean, filter, reorganize and optimize data from different sources to ensure data quality and consistency.

[0071] The ETL algorithm of the data processing platform is divided into two parts: input and output: Input: including data quality management, data interaction and filtering, and reorganization of key data. This part is responsible for cleaning and optimizing the original data to ensure high-quality data input. Output: including the establishment of data architecture, data reorganization governance and data management. These functions achieve visualization and synchronization by establishing data tags and data architecture to ensure data unification and real-time update.

[0072] The core of the data governance platform is to implement data governance by building and managing different data models, including user data models, enterprise data models, and standard data models, to ensure data quality and consistency. The specific steps are as follows:

[0073] (1) Establishment of Model A1 and Model A2: Construct the user data model A1 (the original data model that has not been cleaned and benchmarked) and the enterprise data model A2 (the data model that meets the enterprise standards) respectively.

[0074] (2) Generation of control group A0: Standard control group A0 is a data model that is benchmarked against industry standards and is used to ensure data quality. Model A1 and Model A2 will be compared with A0 to check metadata elements, data types, names, lengths, and other information.

[0075] (3) Model reduction and optimization: Based on the comparison results, unnecessary data in models A1 and A2 are trimmed, and the main data is retained to improve the overall data quality. This process effectively improves data consistency and reduces storage redundancy.

[0076] In the data cleaning process, Bloom filters are used to improve the efficiency and accuracy of data filtering. Bloom filters use hash functions to determine whether data exists in a set, and have the characteristics of efficient filtering and space saving. The specific steps are as follows:

[0077] (1) Insertion process: For each data element to be inserted, its position in the bit array is calculated using k hash functions, and the bit at the corresponding position is set to 1;

[0078] (2) Search process: When searching, use the same k hash functions to check whether the values ​​of the corresponding positions in the bit array are all 1 to determine whether the data exists. If all bits are 1, the element may exist (with a certain probability of false positive); if any bit is 0, the element must not exist. If there is a false positive, the false positive rate is optimized by the bit array size m, the number of elements n, and the number of hash functions k;

[0079] The false positive rate of a Bloom filter can be estimated by the following formula:

[0080] P(false positive)=(1-e -kn / m ) k Where m is the size of the bit array, n is the number of inserted elements, and k is the number of hash functions. To minimize the false positive rate, the value of k should be m / nln2.

[0081] (3) Supporting deletion operations: In order to support data deletion, a counter array is used instead of a simple bit array to support the addition or subtraction of bits, thus solving the problem that traditional Bloom filters do not support deletion operations.

[0082] After Bloom filter screening and data cleaning, the key data will be further sorted and reorganized to ensure data consistency and integrity. The data reorganization process at this stage includes the following:

[0083] (1) Column mapping update: including data type conversion, column name adjustment, and column addition or deletion to adapt to the standard data model;

[0084] (2) Special data processing: Calling Java programs to perform complex cleansing and verification operations on specific data to ensure that the data complies with corporate standards;

[0085] (3) Data analysis and monitoring engine: Use data quality analysis and verification engine to monitor data to maintain data quality standards.

[0086] After data cleaning is completed, the system will automatically establish and maintain the master data architecture to ensure the standardization and consistency of data governance. The steps to maintain the master data architecture include:

[0087] Construction of standard data model: In the master data architecture, a standard model is constructed to enable unified management of accessed data.

[0088] Establishment of data warehouse: The cleaned data is stored in multiple libraries, including raw library, feature library, subject library and knowledge base, to support data retrieval and management in different application scenarios.

[0089] In order to improve data sharing and collaboration capabilities, the integrated data is stored in a standardized manner in the data warehouse, and data access services are provided to enterprise application systems through external service interfaces. Specifically, it includes:

[0090] Data access and data services: Rapid data access is achieved through data exploration, definition, reading and statistics; data processing is completed using tools such as data identification, Bloom filters, cleaning and extraction.

[0091] Database retrieval and model analysis: Provide real-time or quasi-real-time data information to external applications through query retrieval and data operations.

[0092] After data integration and governance are completed, the data governance platform will use the enterprise management module to classify and store the data in a hierarchical manner to meet management needs at different levels.

[0093] The following explains the above process in combination with specific application scenarios:

[0094] 1. Application of user data verification and management:

[0095] In an e-commerce platform, user data is verified and managed through a data governance platform. The specific implementation process is as follows:

[0096] 1) User data model creation: The system first compares the user data (such as registration information, consumption records, etc.) of the e-commerce platform with the standard model through the ETL algorithm to establish the uncleaned user data model A1.

[0097] 2) Model comparison and optimization: Compare model A1 with the standard data model A0 (for example, the industry-wide e-commerce user data model), delete redundant information through the data quality management module, and retain only necessary data fields, such as user ID, consumption amount, access frequency, etc.

[0098] 3) Bloom filter cleaning: Bloom filters filter out duplicate and invalid data, such as abnormal registrations, abnormal orders, etc., to ensure the authenticity and validity of the remaining data.

[0099] 4) Data synchronization and display: The processed data is displayed in the data analysis module of the platform through labels and architecture, and synchronized with the user management module. The above steps can improve the accuracy of user data, making subsequent data analysis and user profiling more efficient and reliable.

[0100] 2. Application of enterprise data management and control:

[0101] In a manufacturing company, the process of enterprise data management and control is implemented through a data governance platform to achieve integrated data management and control of production and sales.

[0102] 1) Enterprise data model creation: The system collects multiple data sources such as production, inventory, and sales, and builds the uncleaned enterprise data model A2 through data integration ETL algorithms.

[0103] 2) Standard model control group: Compare A2 with the industry standard model A0, delete duplicate information and unnecessary historical data, and retain only core data fields such as production batch, inventory, shipment quantity, etc.

[0104] 3) Data cleaning and reorganization: Use Bloom filters to remove redundant data and reorganize and restructure key data to form high-quality standardized data.

[0105] 4) Master data architecture synchronization: Update the processed enterprise data to the master data architecture to achieve real-time production management, inventory warning and logistics information sharing, and improve the coordination of production and sales.

[0106] 3. Data collection and integration application of multiple data sources:

[0107] In a financial institution, data from different business systems (such as loan systems, customer management systems, and transaction systems) need to be integrated for use in data analysis and decision-making.

[0108] 1) Multi-source data collection: The system collects data from multiple business systems through the data processing platform, including basic customer information, transaction records, and credit ratings.

[0109] 2) Data filtering and quality management: Use the data interaction and filtering module to remove noise data, and use the data quality management module to standardize the field names, data types, etc. of different systems to form a unified data model.

[0110] 3) Data cleaning and Bloom filter application: Use Bloom filters to filter duplicate customer records and invalid transaction records to ensure the uniqueness of the data.

[0111] 4) Data architecture establishment and display: The cleaned data is classified and displayed through the master data architecture and data tags, supporting the intuitive display of core data such as customer credit rating and loan risk in the management system, realizing cross-system linkage and real-time updating of data.

[0112] 4. Operator's user data synchronization and verification:

[0113] In the communications industry, user data involves multiple sources and multiple formats, and a unified data governance platform is needed to achieve data synchronization and verification.

[0114] 1) Data collection and modeling: Collect various types of user data (such as user location, call records, network traffic) and build an A1 model, which is initially integrated into an original data model containing multi-source information.

[0115] 2) Model comparison and optimization: Compare the A1 model with the industry standard model A0, standardize the field names, data types, and metadata attributes, ensure that the model complies with industry standards, and remove redundant information.

[0116] 3) Bloom filter cleaning: Through the Bloom filter method, false positives of non-user real data (such as test data, redundant information) are screened out to optimize data accuracy.

[0117] 4) Data synchronization and enterprise sharing: Upload processed data to the master data architecture and synchronize it to multiple systems, including user billing systems, traffic analysis systems, and customer service systems, to achieve instant data updates and support real-time verification of customer information and business activation verification.

[0118] Figure 3 is a structural diagram of a data optimization device according to an embodiment of the present application, such as Figure 3 As shown, the device comprises:

[0119] An acquisition module 30 is used to acquire user data and enterprise data, wherein the user data is the original data in the system and the enterprise data is the data that complies with enterprise standards;

[0120] A comparison module 32 is used to compare the user data and the enterprise data with the standard data respectively to obtain a comparison result, wherein the standard data is data that is compared with the industry standard;

[0121] The optimization module 34 is used to optimize the user data and the enterprise data according to the comparison result, wherein the data format of the optimized user data and the enterprise data is the same as the standard data.

[0122] In the above-mentioned data optimization device, standard data is determined in the following manner: obtaining original data standards and obtaining business requirements of the system; adjusting the original data standards according to business requirements to obtain updated data standards, wherein the updated data standards include standards corresponding to at least one of the following data: metadata elements, naming conventions, data types, and lengths; determining standard data based on the updated data standards.

[0123] In the acquisition module in the above-mentioned data optimization device, the acquisition module is also used to identify duplicate data in user data and / or enterprise data; deduplicate the duplicate data in user data and / or enterprise data through a Bloom filter; identify abnormal data in user data and / or enterprise data, and correct the abnormal data.

[0124] In the comparison module in the above-mentioned data optimization device, the comparison module is also used to obtain field information in the first data, wherein the first data is any one of the user data and the enterprise data, and the field information includes: field name, data type and length; the field information in the first data is compared with the standard field information in the standard data to obtain a comparison result, wherein the standard field information includes the standard field name, standard data type and standard length.

[0125] In the optimization module in the above-mentioned data optimization device, the optimization module is also used to modify the field name of the first data to be consistent with the standard field name when the comparison result indicates that the field name of the first data is inconsistent with the standard field name; when the comparison result indicates that the data type of the first data is inconsistent with the standard data type, convert the data type of the first data to the standard data type; when the comparison result indicates that the length of the first data is less than the standard length, determine the missing column data in the first data; add the missing column data to the first data; when the comparison result indicates that the length of the first data is greater than the standard length, determine the redundant column data in the first data; and delete the redundant column data from the first data.

[0126] The above-mentioned data optimization device also includes a processing module 36, which is also used to obtain the subsystems corresponding to the standard data and the business system; divide the fields in the standard data according to the subsystems to which they belong to obtain multiple types of standard data; and store the multiple types of standard data in the corresponding subsystems.

[0127] In the processing module in the above-mentioned data optimization device, the processing module is also used to obtain optimized user data and optimized enterprise data; to inspect the optimized user data and optimized enterprise data to obtain inspection results; when the inspection result is that the inspection is passed, the access rights of the optimized user data and optimized enterprise data are determined according to the security level.

[0128] It should be noted that Figure 3 The data optimization device shown is used to perform Figure 2 The data optimization method shown in the figure, therefore the relevant explanations in the above data optimization method are also applicable to the data optimization device, and will not be repeated here.

[0129] An embodiment of the present application also provides an electronic device, which includes a memory and a processor, wherein the memory is used to store program instructions; the processor is connected to the memory, and is used to execute program instructions to implement the following functions: obtaining user data and enterprise data, wherein the user data is the original data in the system, and the enterprise data is data that complies with enterprise standards; comparing the user data and enterprise data with standard data respectively to obtain comparison results, wherein the standard data is data benchmarked against industry standards; optimizing the user data and enterprise data based on the comparison results, wherein the data format of the optimized user data and enterprise data is the same as that of the standard data.

[0130] It should be noted that the above electronic equipment is used to execute Figure 2 The data optimization method shown in the figure, therefore the relevant explanations and instructions in the above data optimization method are also applicable to the electronic device and will not be repeated here.

[0131] An embodiment of the present application also provides a non-volatile storage medium, which includes a stored computer program, wherein the device where the non-volatile storage medium is located executes the following data optimization method by running the computer program: obtaining user data and enterprise data, wherein the user data is the original data in the system, and the enterprise data is data that complies with enterprise standards; comparing the user data and enterprise data with standard data respectively to obtain comparison results, wherein the standard data is data benchmarked against industry standards; optimizing the user data and enterprise data based on the comparison results, wherein the data format of the optimized user data and enterprise data is the same as that of the standard data.

[0132] It should be noted that the above non-volatile storage medium is used to execute Figure 2 The data optimization method shown in the figure, therefore the relevant explanations and instructions in the above data optimization method are also applicable to the non-volatile storage medium, and will not be repeated here.

[0133] An embodiment of the present application also provides a computer program product, including computer instructions, which, when executed by a processor, implement the steps of the data optimization method in each embodiment of the present application.

[0134] The embodiments of the present application also provide a computer program, which, when executed by a processor, implements the steps of the data optimization method in each embodiment of the present application.

[0135] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0136] In the above embodiments of the present application, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.

[0137] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of the units can be a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0138] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0139] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0140] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, in essence, or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, disk or optical disk and other media that can store program codes.

[0141] The above is only a preferred implementation of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A method for data optimization, characterized in that: include: Acquire user data and enterprise data, wherein the user data is original data in the system, and the enterprise data is data that complies with enterprise standards; Comparing the user data and the enterprise data with standard data respectively to obtain comparison results, wherein the standard data is data benchmarked against industry standards; The user data and the enterprise data are optimized according to the comparison result, wherein the data formats of the optimized user data and enterprise data are the same as the standard data.

2. The method according to claim 1, characterized in that The standard data is determined in the following way: Obtain original data standards and business requirements of the system; Adjusting the original data standard according to the business requirements to obtain an updated data standard, wherein the updated data standard includes at least one of the following standards corresponding to the data: metadata elements, naming conventions, data types, and lengths; The standard data is determined according to the update data standard.

3. The method according to claim 1, characterized in that After acquiring the user data and the enterprise data, the method further includes: identifying duplicate data in the user data and / or the enterprise data; Deduplication processing is performed on duplicate data in the user data and / or the enterprise data by using a Bloom filter; Abnormal data in the user data and / or the enterprise data is identified, and the abnormal data is corrected.

4. The method according to claim 1, characterized in that: The user data and the enterprise data are respectively compared with the standard data to obtain comparison results, including: Acquire field information in first data, wherein the first data is any one of the user data and the enterprise data, and the field information includes: field name, data type and length; The field information in the first data is compared with the standard field information in the standard data to obtain the comparison result, wherein the standard field information includes a standard field name, a standard data type and a standard length.

5. The method according to claim 4, characterized in that Optimizing the user data and the enterprise data according to the comparison result includes: If the comparison result indicates that the field name of the first data is inconsistent with the standard field name, modify the field name of the first data to be consistent with the standard field name; If the comparison result indicates that the data type of the first data is inconsistent with the standard data type, converting the data type of the first data to the standard data type; If the comparison result indicates that the length of the first data is less than the standard length, determining missing column data in the first data; Add the missing column data to the first data; If the comparison result indicates that the length of the first data is greater than the standard length, determining redundant column data in the first data; The redundant column data is deleted from the first data.

6. The method according to claim 1, characterized in that The method further comprises: Acquire the subsystem corresponding to the standard data and the business system; Dividing the fields in the standard data according to the subsystems to which they belong, to obtain multiple types of standard data; The multiple types of standard data are stored in corresponding subsystems.

7. The method according to claim 1, characterized in that The method further comprises: Obtain optimized user data and optimized enterprise data; Testing the optimized user data and the optimized enterprise data to obtain a test result; When the inspection result is that the inspection is passed, the access rights of the optimized user data and the optimized enterprise data are determined according to the security level.

8. A data optimization device, characterized in that: include: An acquisition module, used to acquire user data and enterprise data, wherein the user data is the original data in the system, and the enterprise data is the data that complies with enterprise standards; A comparison module, used to compare the user data and the enterprise data with standard data respectively to obtain a comparison result, wherein the standard data is data benchmarked against industry standards; An optimization module is used to optimize the user data and the enterprise data according to the comparison result, wherein the data format of the optimized user data and enterprise data is the same as the standard data.

9. An electronic device, characterized in that: include: A memory for storing program instructions; A processor is connected to the memory and is used to execute program instructions to implement the following functions: obtain user data and enterprise data, wherein the user data is original data in the system and the enterprise data is data that complies with enterprise standards; compare the user data and the enterprise data with standard data respectively to obtain comparison results, wherein the standard data is data that is benchmarked against industry standards; optimize the user data and the enterprise data based on the comparison results, wherein the data format of the optimized user data and enterprise data is the same as that of the standard data.

10. A non-volatile storage medium, characterized in that: The non-volatile storage medium includes a stored computer program, wherein the device where the non-volatile storage medium is located executes the data optimization method described in any one of claims 1 to 7 by running the computer program.

11. A computer program product comprising computer instructions, characterized in that: When the computer instructions are executed by a processor, the data optimization method described in any one of claims 1 to 7 is implemented.

Images