Data processing method, device and system, and storage medium

Abstract

The present application discloses a data processing method, device and system, and a storage medium. The method of the present application comprises: constructing a quality problem dictionary on the basis of an experience database, the quality problem dictionary comprising data quality problems and corresponding problem identifiers; acquiring data to be processed, and identifying a target data problem present in the data to be processed; querying the quality problem dictionary for the target data problem to obtain a corresponding target problem identifier; adding the target question identifier into target data by means of a predefined field; reading a plurality of target fields in the target data, determining the plurality of target fields according to a preset fusion rule, and on the basis of a determination result, determining whether there is conflict data; if there is conflict data, identifying a service application scenario of the current data, and selecting an authority source determination strategy on the basis of an identification result; and processing the credible source authority level of each field in the conflict data according to the authority source determination strategy, and updating the database.

Description

Data processing method, device, system and storage medium Technical Field

[0001] The present application relates to the field of data processing, and in particular to a data processing method, device, system and storage medium. Background Art

[0002] In modern businesses and organizations, data is generated and recorded across a variety of data sources and systems. Different departments and individuals often contribute to these data sources, leading to potential inconsistencies and inaccuracies. This can pose significant challenges to data analysis, decision-making, and overall business efficiency. Several approaches have been explored in existing technologies to address data management challenges, including manual data validation, data deduplication algorithms, and basic data integration techniques.

[0003] However, these traditional approaches are limited because they often lack a comprehensive approach to address all aspects of data management, including data cleansing, entity merging, and data aggregation. Manual validation processes are time-consuming and prone to human error, while existing data deduplication algorithms may not adequately handle complex data relationships. Furthermore, current data integration technologies may not fully leverage authoritative sources or adequately address missing data.

[0004] Therefore, an innovative data management and integration system is needed that can efficiently cleanse, merge and aggregate data from various sources while ensuring high accuracy, consistency and availability of the data.

[0005] Summary of the Invention

[0006] In order to solve the above technical problems, the present application provides a data processing method, which includes:

[0007] Building a quality problem dictionary based on the experience database, wherein the quality problem dictionary contains data quality problems and corresponding problem identifiers;

[0008] Acquire data to be processed, and identify target data problems existing in the data to be processed;

[0009] Searching the target data problem in the quality problem dictionary to obtain a corresponding target problem identifier;

[0010] Adding the target problem identifier to the target data through a predefined field;

[0011] Read multiple target fields in the target data, including name, ID number, mobile phone number, date of birth, organization identification code, and company name, and generate four judgment factors based on the aforementioned multiple target fields. The four judgment factors are:

[0012] The first judgment factor is the combination of name and ID number, the second judgment factor is the combination of name and mobile phone number, the third judgment factor is the combination of name, date of birth and organization identification code, and the fourth judgment factor is the combination of name and company name;

[0013] The judgment strategy is: compare the data to be compared using the above four judgment factors. If one of the above four judgment factors matches in the data to be compared, the data to be compared is determined to be conflicting data.

[0014] The method for determining whether the fourth judgment factor matches is as follows:

[0015] When the names are confirmed to be consistent, the semantic similarity between the company names in the data to be compared is calculated;

[0016] If the semantic similarity meets a preset threshold, determining that the fourth judgment factor matches;

[0017] When there is conflicting data, identify the business application scenario of the current data and select the authoritative source judgment strategy based on the identification results;

[0018] Processing the trusted source authority level of each field in the conflicting data according to the authoritative source judgment strategy, and updating the database;

[0019] If there is no conflicting data, a unique identification code is generated for the current data and stored in the database.

[0020] Optionally, identify the business application scenario of the current data and select an authoritative source judgment strategy based on the identification results, including:

[0021] If the current business application scenario is for processing data in an existing historical database, the first authoritative source judgment strategy is selected. The first authoritative source judgment strategy includes:

[0022] A pre-built data authority source level table is queried, and according to the query result, the data with the highest authority level is selected from the conflicting data as the business primary key, and the historical database is updated.

[0023] Optionally, identify the business application scenario of the current data and select an authoritative source judgment strategy based on the identification results, including:

[0024] If the current business application scenario is for warehousing real-time data, the second authoritative judgment strategy is selected. The second authoritative judgment strategy includes:

[0025] Query the pre-built field authority source level table and compare the authority source levels corresponding to each field in the conflicting data. Based on the comparison results, select the field with the highest authority source level for each field, generate result data, and store the result data in the database. The field authority source level records the level of the authority source of each field.

[0026] Optionally, after the result data is stored in the database, modification rules are set for the stored data based on the pre-built field authority source level table. The modification rules include:

[0027] Identify the target field currently being modified, and query the corresponding field authority source registration table based on the modified field;

[0028] Compare the authority source level of the current modifying user with the authority source level of the target field based on the query results;

[0029] If the authority source level of the modifying user is greater than or equal to the current authority source level of the target field, the modification is allowed; otherwise, the modification is rejected.

[0030] Optionally, if modification is allowed, after the modification, the current authoritative source level of the target field is updated to the authoritative source level of the modifying user.

[0031] Optionally, identifying target data problems in the data to be processed includes:

[0032] The data to be processed are uniformly pre-cleaned according to predetermined pre-cleaning rules. During the pre-cleaning process, target data that cannot be uniformly pre-cleaned and corresponding target data problems are marked.

[0033] A second aspect of the present application provides a data processing device, the device comprising:

[0034] A quality problem dictionary construction module is used to construct a quality problem dictionary based on an experience database, wherein the quality problem dictionary contains data quality problems and corresponding problem identifiers;

[0035] A target data problem identification module is used to obtain the data to be processed and identify target data problems existing in the data to be processed;

[0036] A target problem identification query module is used to query the target data problem in the quality problem dictionary to obtain a corresponding target problem identification;

[0037] A target problem identifier adding module, configured to add the target problem identifier to the target data through a predefined field;

[0038] The target field judgment and conflict detection module is used to read multiple target fields in the target data, including name, ID number, mobile phone number, date of birth, organization identification code and company name, and generate four judgment factors based on the aforementioned multiple target fields. The four judgment factors are:

[0039] The first judgment factor is the combination of name and ID number, the second judgment factor is the combination of name and mobile phone number, the third judgment factor is the combination of name, date of birth and organization identification code, and the fourth judgment factor is the combination of name and company name;

[0040] The judgment strategy is: compare the data to be compared using the above four judgment factors. If one of the above four judgment factors matches in the data to be compared, the data to be compared is determined to be conflicting data.

[0041] The method for determining whether the fourth judgment factor matches is as follows:

[0042] When the names are confirmed to be consistent, the semantic similarity between the company names in the data to be compared is calculated;

[0043] If the semantic similarity meets a preset threshold, determining that the fourth judgment factor matches;

[0044] The authoritative source judgment strategy selection module is used to identify the business application scenario of the current data when there is conflicting data, and select the authoritative source judgment strategy based on the identification results;

[0045] a conflict data processing module, configured to process the trusted source authority level of each field in the conflict data according to the authoritative source judgment strategy, and update the database;

[0046] The unique identification code generation module is used to generate a unique identification code for the current data and store it in the database when there is no conflicting data.

[0047] A third aspect of the present application provides a data processing system, the device comprising:

[0048] processor, memory, input and output units, and buses;

[0049] The processor is connected to the memory, the input and output unit, and the bus;

[0050] The memory stores a program, and the processor calls the program to execute the first aspect and any optional method in the first aspect.

[0051] In a fourth aspect, the present application provides a computer-readable storage medium, on which a program is stored. When the program is executed on a computer, the program executes the first aspect and any optional method in the first aspect.

[0052] It can be seen from the above technical solutions that this application has the following advantages:

[0053] The method provided in this application identifies and marks problems in the data by constructing a quality problem dictionary and identification, which helps to accurately identify and handle data quality problems and improve the accuracy and credibility of the data.

[0054] By reading multiple fields in the target data and making judgments based on preset fusion rules, duplicate data and entities can be identified and merged into a unified entity, thereby eliminating data conflicts and duplications and improving data consistency and integrity.

[0055] By identifying the business application scenarios of the current data and selecting an appropriate authoritative source judgment strategy based on the identification results, the trusted source authority level of each field in the conflicting data can be processed to ensure the accuracy and credibility of the data.

[0056] The solution mentions selecting authoritative source judgment strategies based on business application scenarios, which makes data processing more flexible and adaptable to different business needs, ensuring the effectiveness of data processing methods in different scenarios.

[0057] By adding target problem identifiers to target data through predefined fields, problems in the data can be quickly identified and addressed, improving data processing efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] In order to more clearly illustrate the technical solutions in this application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of this application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0059] FIG1 is a flow chart of an embodiment of a data processing method provided in this application;

[0060] FIG2 is a schematic diagram of the combination of judgment factors in this application;

[0061] FIG3 is a flow chart of a method for updating the authority source level based on modification rules in this application;

[0062] FIG4 is a schematic structural diagram of a data processing device provided in this application;

[0063] FIG5 is a schematic diagram of the structure of the data processing system provided in this application. DETAILED DESCRIPTION

[0064] It should be noted that the method provided in this application can be applied to a terminal, a system, or a server. For example, the terminal can be a smartphone, a computer, a tablet computer, a smart TV, a smartwatch, a portable computer terminal, or a fixed terminal such as a desktop computer. For ease of explanation, this application uses a terminal as the execution subject for example.

[0065] This applicant declares that the personal information collected through this program is collected legally through public channels and with the consent of the individual, and that the collected personal information will not be illegally disclosed. All personal information involved in this application has been authorized by the relevant parties.

[0066] The specific embodiments of the method provided in this application are described in detail below:

[0067] Referring to FIG1 , the present application first provides an embodiment of a data processing method, which includes:

[0068] S101. Construct a quality problem dictionary based on an experience database, wherein the quality problem dictionary includes data quality problems and corresponding problem identifiers;

[0069] In this step, we first build a quality problem dictionary based on the experience database. This dictionary contains various possible data quality problems and corresponding problem identifiers. The problem identifiers are used to subsequently identify and mark quality problems in the target data. First, we need to establish an experience database, which is used to record various data quality problems that have occurred in the past data processing process, as well as the corresponding problem identifiers. This experience database can be accumulated by a professional data management team, data quality experts, or past data processing experience. In the experience database, common data quality problems for different data sources and different types of data are collected and organized, such as missing data, erroneous data, duplicate data, format errors, etc. Each data quality problem should have a corresponding problem identifier.

[0070] For example, we first construct a quality problem dictionary to sort out the data quality issues of information items such as company names, personal names, and mobile phone numbers and assign them problem identifiers. The following is an example of a quality problem dictionary:

[0071] S102: Acquire data to be processed, and identify target data problems in the data to be processed;

[0072] In this step, the data to be processed is acquired and identified to determine any potential data quality issues within it. This step is used to identify potential data quality issues within the data. The algorithms and rules within the data processing method are then used to identify potential data quality issues within the data.

[0073] First, the data to be processed is obtained from various data sources and systems. This data can be structured data tables, database records, or unstructured data such as text, images, and audio. After obtaining the data to be processed, predefined algorithms and rules are used to identify potential data issues. Target data issues may include, but are not limited to, the following:

[0074] Missing data: Certain fields or attributes lack required data values.

[0075] Bad data: Data values ​​do not conform to the specified format, range, or logical conditions.

[0076] Duplicate data: There are multiple identical or nearly identical data records.

[0077] Format Error: The data is not in the expected format.

[0078] Logical error: The logical relationship between data is incorrect or inconsistent.

[0079] Once the target data problem is identified, the system can add a problem indicator to the corresponding location or field in the data based on the identification results. The problem indicator can be a pre-defined code, symbol or specific value used to mark the part of the data with the problem.

[0080] This step can also perform data quality assessments, such as calculating the proportion of missing data, testing the logical consistency of the data, etc. These assessments can provide an overall picture of data quality and possible problems.

[0081] One implementation of this step is to perform unified pre-cleaning processing on the data to be processed according to predetermined pre-cleaning rules, and during the pre-cleaning processing, mark target data that cannot be uniformly pre-cleaned and corresponding target data problems.

[0082] First, a set of predetermined pre-cleaning rules is developed. These rules may include data formatting, removing null values ​​or outliers, fixing common errors, etc. The predetermined rules can be based on empirical databases or professional domain knowledge, or they can be obtained through data analysis and data mining methods.

[0083] The data to be processed is uniformly pre-cleaned using pre-defined pre-cleaning rules. During this step, the system automatically processes the data according to the pre-defined rules, converting it to a unified format and standard, correcting common errors, and removing or filling missing values. During the pre-cleaning process, the system can flag target data that cannot undergo uniform pre-cleaning using pre-defined problem indicators. These target data issues can include data that does not conform to the pre-defined rules and other data quality issues that cannot be automatically addressed.

[0084] S103, searching the quality problem dictionary for the target data problem to obtain a corresponding target problem identifier;

[0085] In the above steps, data quality issues that may exist in the target data have been identified. In this step, the system will traverse the quality problem dictionary based on these target data issues, and check one by one whether the issues in the dictionary match the target data issues. When a data quality issue that matches the target data issue is found in the dictionary, the system will obtain the problem identifier corresponding to the issue. The problem identifier is then added to the corresponding field or position in the target data to mark the quality issues that exist in the target data. If the target data issue does not find a match in the quality problem dictionary, the system can handle these unmatched issues separately, such as generating a new problem identifier or recording it as a specific unknown problem. In order to improve query efficiency and reduce data processing time, appropriate data structures and algorithms can be used to implement queries on the quality problem dictionary, such as using hash tables, binary tree search methods, dictionary trees, etc.

[0086] S104, adding the target problem identifier to the target data through a predefined field;

[0087] In step S104, the target problem identifier obtained in the previous step S103 is added to the target data through a predefined field, thereby marking the existing problem and providing a basis for subsequent processing. The specific implementation method can be as follows:

[0088] One or more fields can be predefined to store target problem identifiers. These fields can be newly created or existing. In step S103, the quality problem dictionary is queried to identify the problems present in the target data and obtain the corresponding target problem identifiers. These target problem identifiers are then added one by one to the corresponding locations or fields in the target data using the predefined fields.

[0089] The target problem identifier can be added in different ways, such as recording the problem identifier code or symbol in a new field, or marking the problem identifier as part of the target field. The marking method should be selected based on actual needs and data processing scenarios.

[0090] After adding the target problem identifier to the target data, the data needs to be organized and stored. This may involve operations such as data cleaning, format conversion, and data archiving to ensure data consistency and integrity.

[0091] Through the above implementation, step S104 can add target problem identifiers to the target data, thereby marking any problems within the data. This allows subsequent data processing to be classified, filtered, and integrated based on the problem identifiers, improving the accuracy and efficiency of data processing. Furthermore, the addition of target problem identifiers facilitates data quality assessment and troubleshooting.

[0092] S105, reading multiple target fields in the target data, judging the multiple target fields according to a preset fusion rule, and determining whether there is conflicting data based on the judgment result;

[0093] In step S105, the goal is to read multiple target fields in the target data and judge these fields according to the preset fusion rules to determine whether there is conflicting data, that is, inconsistent data of the same entity. First, multiple target fields that need to be fused are read from the target data. These fields may contain data from different sources, different times or different formats. According to the preset fusion rules, the multiple target fields read are compared and judged one by one. If it is found in the fusion judgment that the entity information represented by multiple fields is inconsistent, that is, there is conflicting data, the system will mark or record these conflicting data. Specific identifiers, status values ​​or other methods may be used for marking.

[0094] In this application, the multiple target fields identified can be name, ID number, mobile phone number, date of birth, organization identification code and company name. When executing the conflict data judgment strategy, these fields can be combined into judgment conditions to check the data for duplicates. The specific field combination rules are not limited in this application. Below, this application provides a preferred embodiment, which includes a preferred judgment strategy:

[0095] Referring to FIG2 , four judgment factors are generated based on the aforementioned multiple target fields. The four judgment factors are:

[0096] The first judgment factor is the combination of name and ID number, the second judgment factor is the combination of name and mobile phone number, the third judgment factor is the combination of name, date of birth and organization identification code, and the fourth judgment factor is the combination of name and company name;

[0097] The judgment strategy is: compare the data to be compared using the above four judgment factors. If one of the above four judgment factors matches in the data to be compared, the data to be compared is determined to be conflicting data.

[0098] In this embodiment, four judgment factors are generated based on the aforementioned multiple target fields, each judgment factor being composed of a different combination of target fields. The data to be compared is then compared using these four judgment factors. If any judgment factor in the data to be compared matches any judgment factor in the target data, the data to be compared is determined to be conflicting data.

[0099] Generate judgment factors: According to the preferred judgment strategy, extract the name, ID number, mobile phone number, date of birth, organization identification code and company name fields from the target data, and generate four judgment factors according to the combination of the first judgment factor, the second judgment factor, the third judgment factor and the fourth judgment factor.

[0100] The first judgment factor: consists of name and ID number.

[0101] The second judgment factor: consists of name and mobile phone number.

[0102] The third judgment factor: consists of name, date of birth and organization identification code.

[0103] The fourth judgment factor: consists of name and company name.

[0104] When performing data comparison, for the data to be compared, four judgment factors are generated according to the combination of the first judgment factor, the second judgment factor, the third judgment factor and the fourth judgment factor.

[0105] Conflict Data Determination: Compare the four determination factors of the data to be compared with the four determination factors of the target data. If any determination factor in the data to be compared matches any determination factor of the target data, the data to be compared is determined to be conflicting data.

[0106] This optimal judgment strategy can effectively identify conflicting data, that is, data that contains the same entity but inconsistent information. This data processing method helps improve data quality and credibility, which is of great significance in subsequent data management and application processes.

[0107] Furthermore, this embodiment also provides a more specific implementation method, that is, for the fourth judgment factor, in data processing, the similarity judgment of enterprise names is performed because in actual situations, enterprise names may have certain variations, typos, abbreviations, etc., resulting in enterprise names expressed in different ways actually referring to the same enterprise. However, this diversity and difference in enterprise names may lead to misjudgments or omissions in the data fusion or matching process, affecting the accuracy and consistency of the data. Therefore, for the fourth judgment factor, after determining that the names are consistent, it is necessary to calculate the semantic similarity between the enterprise names. If the calculated semantic similarity meets the preset threshold, it is determined that the fourth judgment factor matches, thereby confirming that the data to be compared is conflicting data.

[0108] According to the above-mentioned preferred judgment strategy, the names are matched. After determining that the names are consistent, the semantic similarity of the company name in the data to be compared and the company name in the target data is calculated. The semantic similarity calculation can use natural language processing technology, such as text matching algorithm, word vector model, etc., to measure the semantic similarity between the two company names. A preset semantic similarity threshold can be set to determine whether the company name of the data to be compared is similar to the company name of the target data to a certain extent. The calculated semantic similarity is compared with the preset threshold. If the semantic similarity meets the preset threshold, that is, the semantic similarity between the company names is higher than the set threshold, then it is determined that the fourth judgment factor matches, and the data to be compared is regarded as conflicting data. Otherwise, the fourth judgment factor does not match, and the processing ends.

[0109] S106. When there is conflicting data, identify the business application scenario of the current data and select an authoritative source judgment strategy based on the identification result;

[0110] In step S106, when conflicting data is found, it is necessary to identify the business application scenario of the current data and select an appropriate authoritative source judgment strategy based on this scenario to handle the conflicting data, thereby determining the trusted source authority level of the field. This process can ensure the accuracy and credibility of the data in different business application scenarios.

[0111] In this step, it is necessary to determine the business application scenario of the current data based on the business needs and application scenarios of data management. This scenario may involve different data processing goals, data usage methods, and requirements for data accuracy. Based on the identified business application scenarios, a set of authoritative source judgment strategies are pre-defined. These strategies may match the needs of different business application scenarios to ensure the authority and credibility of the data in the scenario. Based on the selected authoritative source judgment strategy, the authority of each field in the conflicting data is judged. Different authority evaluation algorithms, data source considerations, and other methods may be used to judge the trusted source authority level of the field. Based on the authority judgment results of the conflicting data fields, the authority level information of the corresponding fields in the database is updated. In this way, in the subsequent data processing and application process, corresponding decisions and uses can be made based on the authority level of the field.

[0112] For step S106, this application provides two possible data processing scenarios. One is for data processing in an existing historical database. In this scenario, the data processing goal is to insert new data into the historical database. In this case, it is necessary to adjust the authoritative source level of the data. The authoritative source determination strategy in this scenario is:

[0113] If the current business application scenario is for processing data in an existing historical database, the first authoritative source judgment strategy is selected. The first authoritative source judgment strategy includes:

[0114] A pre-built data authority source level table is queried, and according to the query result, the data with the highest authority level is selected from the conflicting data as the business primary key, and the historical database is updated.

[0115] This application provides a first authoritative source judgment strategy, which is briefly described below:

[0116] A pre-built data authority source rating table records the authority ratings of different data sources. This table contains the identifiers of different data sources and their corresponding authority ratings, indicating the trustworthiness of the data sources. For any conflicting data discovered, the authority ratings of the conflicting data sources can be obtained by querying the data authority source rating table.

[0117] Based on the query results from the data authority source level table, the data source with the highest authority level is selected as the source of the business primary key. In other words, in the event of conflicting data, the information provided by the data source with the highest authority level takes precedence and is used as the business primary key. The information determined using the business primary key in the conflicting data is then updated to the historical database. This way, the corresponding data in the historical database is updated with the information from the highest authority level, ensuring data authority and accuracy.

[0118] By selecting the first authoritative source judgment strategy and determining the business primary key based on the data authority source hierarchy table, we can ensure the authority and credibility of data processing in the historical database. This strategy fully utilizes the data authority hierarchy information and prioritizes the most trusted data sources for data updates, avoiding data confusion and errors while improving the quality and availability of historical database data.

[0119] For another data processing scenario, the goal is to process real-time data. In this scenario, the authoritative source judgment strategy is required to respond to real-time data processing. Therefore, this application also provides a strategy, which is defined as the second authoritative source judgment strategy. The second authoritative source judgment strategy is described below:

[0120] If the current business application scenario is for warehousing real-time data, the second authoritative judgment strategy is selected. The second authoritative judgment strategy includes:

[0121] Query the pre-built field authority source level table and compare the authority source levels corresponding to each field in the conflicting data. Based on the comparison results, select the field with the highest authority source level for each field, generate result data, and store the result data in the database. The field authority source level records the level of the authority source of each field.

[0122] First, a field authority source level table is pre-built to record the authority source level information of each field. The table contains the identifier of each field and the corresponding authority level, which is used to indicate the credibility of each field. For the conflicting data found, the field authority source level table is queried to obtain the authority level information of each field involved in the conflicting data. According to the query results, each field in the conflicting data is compared, and the field with the highest authority source level is selected as the result data. That is, in the conflicting data, the information of the field with the highest authority level is used as the basis to generate the result data. The selected result data is processed and stored in the database. In this way, the data stored in the database will be based on the field information with the highest authority level, ensuring the authority and credibility of the data entered into the database.

[0123] The second-authoritative source judgment strategy is suitable for real-time data processing scenarios, enabling timely processing and storage of conflicting data. Since real-time data can change frequently within a short period of time, the second-authoritative source judgment strategy allows for rapid processing of incoming data, timely database updates, and the preservation of real-time and flexibility.

[0124] This strategy dynamically selects authoritative sources based on the authority level of each field in the actual conflicting data, adapting to different data scenarios and business needs. This adaptability can cope with changes in data sources and the diversity of business scenarios, improving the flexibility and adaptability of data processing.

[0125] By selecting the fields with the highest authority source level as the result data for storage, the database storage structure can be optimized. Only the most reliable data information is retained, which reduces the storage of redundant data and optimizes the space utilization efficiency of the database.

[0126] S107: Process the trusted source authority level of each field in the conflicting data according to the authoritative source judgment strategy, and update the database.

[0127] In step S107, according to the selected authoritative source judgment strategy, the trusted source authority level of each field in the conflicting data is processed and the processed data is updated to the database. This processing flow helps to ensure that the data in the database is authoritative and reliable.

[0128] In this embodiment, the trusted source authority level for each field in the conflicting data is processed based on the selected authoritative source determination strategy. The trusted source authority level information for each field in the processed conflicting data is updated to the database. In this way, the data stored in the database will contain the most up-to-date trusted source authority level information, ensuring the authority and reliability of the data in the database.

[0129] When updating a database, conflicting data needs to be merged, combining inconsistent data for the same entity into a single record. During the merging process, the field information with the highest authority level is prioritized to ensure the authority and accuracy of the final data.

[0130] S108. If there is no conflicting data, a unique identification code is generated for the current data and stored in the database.

[0131] In step S107, if no conflicting data is found in step S105, it means that the current data does not need to be further merged and processed, and the data is already unique. In this step, a unique identification code can be generated for the current data and the data is stored in the database.

[0132] A unique identification code can be generated for the current data using a certain algorithm or rule. The identification code can be a number, a string, or other unique identifier to distinguish different data entities.

[0133] The above provides an embodiment of the data processing method of this application, which aims to address data quality issues and information disorganization that may arise when managing data from multiple data sources. The method includes three stages: cleaning, fusion, and aggregation. Each stage is equipped with corresponding algorithms and steps to ensure data accuracy and consistency.

[0134] The advantages of this embodiment are:

[0135] Effectively handle data quality issues: By pre-building a quality issue dictionary and fusion rules, data quality issues can be efficiently identified and handled, improving data accuracy and credibility.

[0136] Adaptive processing of different data scenarios: Select different authoritative source judgment strategies based on business application scenarios, so that data processing can flexibly adapt to different data sources and business needs.

[0137] Data aggregation and fusion improve data integrity: Through aggregation rules and fusion rules, data integrity and consistency are ensured, and data quality is improved.

[0138] Database optimization and unique identification code generation: Optimize the database storage structure, improve the space utilization efficiency of the database, and generate a unique identification code for each piece of data to ensure the uniqueness of the data.

[0139] Regarding the second authoritative source determination strategy provided in the above embodiment, this application also provides a method for dynamically updating the authoritative source level. This method can automatically update the authoritative source level of each field to a higher level, thereby ensuring that the data modification permissions will not be confused. The method is described below and includes the following steps:

[0140] In the method provided in this embodiment, after the result data is stored in the database, modification rules are set for the stored data based on the pre-built field authority source level table. The modification rules include:

[0141] S001. Identify the target field currently being modified and query the corresponding field authority source registration table based on the modified field;

[0142] S002. Compare the authority source level of the current modifying user with the current authority source level of the target field based on the query result;

[0143] S003. If the authority source level of the modifying user is greater than or equal to the current authority source level of the target field, the modification is allowed;

[0144] S004. If the authority source level of the modifying user is lower than the current authority source level of the target field, the modification is rejected.

[0145] S005. After the modification, the current authoritative source level of the target field is updated to the authoritative source level of the modifying user.

[0146] In the above embodiment, the dynamic update method is to ensure that the modification permissions of the data are not confused, prevent low-authority users from accidentally modifying high-authority fields, and thus maintain the authority and integrity of the data. The method is described in detail as follows:

[0147] Set modification rules based on the field authority source rating table: After data is stored, modification rules are set for the stored data based on the pre-built field authority source rating table. The field authority source rating table records the authority source rating information of each field, indicating the trustworthiness of each field.

[0148] When a user modifies data in the database, the system identifies the target field being modified. Based on the target field being modified, the system queries the corresponding field authority source level in the field authority source level table. The system obtains the authority source level of the current modifying user and compares it with the current authority source level of the target field. If the current modifying user's authority source level is greater than or equal to the current authority source level of the target field, the modification is allowed. Otherwise, the modification is rejected.

[0149] In the aforementioned dynamic update method, if it is confirmed that modification is allowed after comparing the permission levels, the modification operation on the target field is executed.

[0150] After the modification, the current authority source level of the target field is updated to the authority source level of the modifying user. In this way, the authority source level of the target field will be updated along with the permission level of the modifying user, maintaining the consistency of data management, thus ensuring that the data will not be modified by users with lower authority source levels.

[0151] By updating the target field's current authority source level to the modifying user's authority source level, data permissions are dynamically adjusted. Only users with sufficiently high permissions can modify data, and the modified field's authority source level is automatically updated, ensuring that the authority and integrity of the data are maintained. This dynamic update method helps ensure the security and reliability of data management and provides more flexible data permission control.

[0152] The above embodiments describe the data processing method in this application. The following describes the device, system, and storage medium involved in this application:

[0153] Referring to FIG4 , the present application provides a data processing device, including:

[0154] A quality problem dictionary building module 401 is used to build a quality problem dictionary based on an experience database, wherein the quality problem dictionary includes data quality problems and corresponding problem identifiers;

[0155] The target data problem identification module 402 is used to obtain the data to be processed and identify the target data problems existing in the data to be processed;

[0156] A target problem identifier query module 403 is configured to query the target data problem in the quality problem dictionary to obtain a corresponding target problem identifier;

[0157] A target problem identifier adding module 404 is configured to add the target problem identifier to the target data through a predefined field;

[0158] The target field judgment and conflict detection module 405 is used to read multiple target fields in the target data, judge the multiple target fields according to the preset fusion rules, and determine whether there is conflicting data based on the judgment result;

[0159] The authoritative source judgment strategy selection module 406 is used to identify the business application scenario of the current data when there is conflicting data, and select the authoritative source judgment strategy based on the identification result;

[0160] The conflict data processing module 407 is used to process the trusted source authority level of each field in the conflict data according to the authoritative source judgment strategy and update the database.

[0161] Optionally, the target field determination and conflict detection module 405 is specifically configured to:

[0162] Four judgment factors are generated based on the aforementioned multiple target fields. The four judgment factors are:

[0163] The first judgment factor is the combination of name and ID number, the second judgment factor is the combination of name and mobile phone number, the third judgment factor is the combination of name, date of birth and organization identification code, and the fourth judgment factor is the combination of name and company name;

[0164] The judgment strategy is: compare the data to be compared using the above four judgment factors. If one of the above four judgment factors matches in the data to be compared, the data to be compared is determined to be conflicting data.

[0165] Optionally, the target field determination and conflict detection module 405 is specifically configured to:

[0166] When the names are confirmed to be consistent, the semantic similarity between the company names in the data to be compared is calculated;

[0167] If the semantic similarity meets a preset threshold, the fourth judgment factor is determined to be matched.

[0168] Optionally, the authoritative source judgment strategy selection module 406 is specifically configured to:

[0169] If the current business application scenario is for processing data in an existing historical database, the first authoritative source judgment strategy is selected. The first authoritative source judgment strategy includes:

[0170] A pre-built data authority source level table is queried, and according to the query result, the data with the highest authority level is selected from the conflicting data as the business primary key, and the historical database is updated.

[0171] Optionally, the authoritative source judgment strategy selection module 406 is specifically configured to:

[0172] If the current business application scenario is for warehousing real-time data, the second authoritative judgment strategy is selected. The second authoritative judgment strategy includes:

[0173] Query the pre-built field authority source level table and compare the authority source levels corresponding to each field in the conflicting data. Based on the comparison results, select the field with the highest authority source level for each field, generate result data, and store the result data in the database. The field authority source level records the level of the authority source of each field.

[0174] Optionally, a rule modification setting module 408 is further included, configured to:

[0175] After the result data is stored in the database, modification rules are set for the stored data based on the pre-built field authority source level table. The modification rules include:

[0176] Identify the target field currently being modified, and query the corresponding field authority source registration table based on the modified field;

[0177] Compare the authority source level of the current modifying user with the authority source level of the target field based on the query results;

[0178] If the authority source level of the modifying user is greater than or equal to the current authority source level of the target field, the modification is allowed; otherwise, the modification is rejected.

[0179] Also used for:

[0180] If modification is allowed, then after the modification, the current authoritative source level of the target field is updated to the authoritative source level of the modifying user.

[0181] Optionally, the system further includes a unique identification code generating module 409, configured to:

[0182] If there is no conflicting data, a unique identification code is generated for the current data and stored in the database.

[0183] Also included is a unified pre-cleaning module 410 for:

[0184] The data to be processed are uniformly pre-cleaned according to predetermined pre-cleaning rules. During the pre-cleaning process, target data that cannot be uniformly pre-cleaned and corresponding target data problems are marked.

[0185] Referring to FIG5 , the present application further provides a data processing system, including:

[0186] Processor 501, memory 502, input and output unit 503, bus 504;

[0187] The processor 501 is connected to the memory 502, the input and output unit 503 and the bus 504;

[0188] The memory 502 stores a program, and the processor 501 calls the program to execute any of the above data processing methods.

[0189] The present application also relates to a computer-readable storage medium having a program stored thereon, wherein when the program is run on a computer, the computer is caused to execute any of the above methods.

[0190] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0191] In the several embodiments provided in this application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of the units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0192] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0193] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0194] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

Claims

1. A data processing method, characterized in that: The method comprises: Based on the experience database, a quality problem dictionary is constructed, wherein the quality problem dictionary contains data quality problems and corresponding problem identifiers; Acquire data to be processed, and identify target data problems existing in the data to be processed; Searching the target data problem in the quality problem dictionary to obtain a corresponding target problem identifier; Adding the target problem identifier to the target data through a predefined field; Read multiple target fields in the target data, the multiple target fields include: name, ID number, mobile phone number, date of birth, organization identification code and company name, and generate four judgment factors based on the aforementioned multiple target fields. The four judgment factors are: The first judgment factor: the combination of name and ID number, the second judgment factor: the combination of name and mobile phone number, the third judgment factor: the combination of name, date of birth and organization identification code, and the fourth judgment factor: the combination of name and company name; The judgment strategy is: compare the data to be compared by using the above four judgment factors. If one of the above four judgment factors matches in the data to be compared, the data to be compared is determined to be conflicting data. The method for determining whether the fourth determination factor matches is as follows: When the names are determined to be consistent, the semantic similarity between the enterprise names in the data to be compared is calculated; If the semantic similarity meets a preset threshold, determining that the fourth judgment factor matches; When there is conflicting data, identify the business application scenario of the current data and select the authoritative source judgment strategy based on the identification results; Processing the trusted source authority level of each field in the conflicting data according to the authoritative source judgment strategy, and updating the database; If there is no conflicting data, a unique identification code is generated for the current data and stored in the database.

2. The data processing method according to claim 1, characterized in that: Identify the business application scenarios of the current data and select the authoritative source judgment strategy based on the identification results, including: If the current business application scenario is to process data in an existing historical database, the first authoritative source judgment strategy is selected. The first authoritative source judgment strategy includes: A pre-constructed data authority source level table is queried, and according to the query result, the data with the highest authority level is used as the business primary key from the conflicting data, and the historical database is updated.

3. The data processing method according to claim 1, characterized in that: Identify the business application scenarios of the current data and select the authoritative source judgment strategy based on the identification results, including: If the current business application scenario is for warehousing real-time data, the second authoritative judgment strategy is selected, and the second authoritative judgment strategy includes: Query the pre-constructed field authority source level table, and compare the authority source levels corresponding to each field in the conflicting data. According to the comparison result, select the field with the highest authority source level for each field, generate result data, and store the result data. The field authority source level records the level of the authority source of each field.

4. The data processing method according to claim 3, characterized in that: After the result data is stored in the database, modification rules are set for the stored data based on the pre-built field authority source level table. The modification rules include: Identify the target field that is currently being modified, and query the corresponding field authority source registration table based on the modified field; Compare the authority source level of the current modifying user with the authority source level of the target field based on the query results; If the authoritative source level of the modifying user is greater than or equal to the current authoritative source level of the target field, the modification is allowed, otherwise the modification is rejected.

5. The data processing method according to claim 4, characterized in that: If modification is allowed, then after the modification, the current authoritative source level of the target field is updated to the authoritative source level of the modifying user.

6. The data processing method according to claim 1, characterized in that: The identifying target data problems existing in the data to be processed comprises: The data to be processed are uniformly pre-cleaned according to predetermined pre-cleaning rules. During the pre-cleaning process, target data that cannot be uniformly pre-cleaned and corresponding target data problems are marked.

7. A data processing device, characterized in that: The device comprises: A quality problem dictionary construction module is used to construct a quality problem dictionary based on an experience database, wherein the quality problem dictionary contains data quality problems and corresponding problem identifiers; A target data problem identification module is used to obtain the data to be processed and identify the target data problems existing in the data to be processed; A target problem identification query module, used to query the target data problem in the quality problem dictionary to obtain a corresponding target problem identification; A target problem identifier adding module, used for adding the target problem identifier to the target data through a predefined field; The target field judgment and conflict detection module is used to read multiple target fields in the target data, the multiple target fields include: name, ID number, mobile phone number, date of birth, organization identification code and company name, and generate four judgment factors according to the aforementioned multiple target fields. The four judgment factors are: The first judgment factor: the combination of name and ID number, the second judgment factor: the combination of name and mobile phone number, the third judgment factor: the combination of name, date of birth and organization identification code, and the fourth judgment factor: the combination of name and company name; The judgment strategy is: compare the data to be compared by using the above four judgment factors. If one of the above four judgment factors matches in the data to be compared, the data to be compared is determined to be conflicting data. The method for determining whether the fourth determination factor matches is as follows: When the names are determined to be consistent, the semantic similarity between the enterprise names in the data to be compared is calculated; If the semantic similarity meets a preset threshold, determining that the fourth judgment factor matches; The authoritative source judgment strategy selection module is used to identify the business application scenario of the current data when there is conflicting data, and select the authoritative source judgment strategy based on the identification result; A conflict data processing module, used to process the trusted source authority level of each field in the conflict data according to the authoritative source judgment strategy, and update the database; The unique identification code generation module is used to generate a unique identification code for the current data and store it in the database when there is no conflicting data.

8. A data processing system, characterized in that: The device comprises: Processor, memory, input-output unit, and bus; The processor is connected to the memory, the input and output unit, and the bus; The memory stores a program, and the processor calls the program to execute the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a program stored thereon, wherein the program, when executed on a computer, performs the method according to any one of claims 1 to 6.

Images