A multi-source data processing method and related device

Abstract

The present invention proposes a multi-source data processing method and related devices, the method comprising: determining the target of data collection, the data to be collected and the data source; performing a first classification on potential data sources, scoring each data source in the first classification, and obtaining a first score; determining a first candidate data source and a first data source according to the first score and the update frequency; extracting data from the data source and transmitting it to a data processing center; preprocessing the data from different data sources; and converting the data from different data sources into a unified format, integrating them, and storing the integrated data in a target database; by comprehensively considering the quality of the data source, the update frequency, the real-time requirements and the business needs, an efficient, accurate and adaptable multi-source data processing solution is provided to meet the needs of modern data processing environments for multi-source data integration and consolidation.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a multi-source data processing method and related devices. Background Art

[0002] With the rapid development of information technology and the advent of the big data era, multi-source data processing has become one of the important challenges faced by enterprises and organizations. Multi-source data processing involves collecting, integrating, cleaning and storing data from multiple different sources for subsequent data analysis and mining. In the modern data processing environment, the integration and consolidation of multi-source data is essential to support complex business decisions, analysis or operations. However, due to the diversity of data sources, including different databases, API interfaces, file systems, etc., the differences in data format, data quality and data update frequency have brought great challenges to multi-source data processing.

[0003] Traditional multi-source data processing methods usually lack a comprehensive evaluation mechanism for data sources, resulting in the selected data source may not meet specific business needs or data quality requirements. In addition, the data quality, update frequency and real-time requirements of the data source are also important factors to consider when selecting a data source, which are often overlooked in traditional processing methods.

[0004] At the same time, with the continuous changes in business needs and the update of data sources, traditional processing methods are often unable to adapt to these changes in a timely manner, resulting in the accuracy and timeliness of data processing being affected. In addition, for the processing of missing data, traditional methods often lack an effective supplementary mechanism and cannot ensure the integrity and accuracy of the data. Summary of the invention

[0005] The purpose of this application is to provide a multi-source data processing method, which aims to score and classify potential data sources by determining the target of data collection, the collected data and the source, so as to select the most suitable data source as the first data source. At the same time, the method also introduces a mechanism for regularly updating the score and re-determining the data source to ensure that the selection of the data source can be adjusted in time with the changes in business needs and data sources. In addition, for the processing of missing data, the method adopts a supplementary mechanism based on the second score to ensure the completeness and accuracy of the data.

[0006] The purpose of this application is achieved by the following technical solutions:

[0007] The present application proposes a multi-source data processing method, the method comprising:

[0008] S1. Determine the data collection objectives, data to be collected, and data sources;

[0009] S2. Perform a first classification on potential data sources, score each data source in the first classification, and obtain a first score; determine a first candidate data source and a first data source according to the first score and the update frequency;

[0010] S3, extracting data from the data source and transmitting it to the data processing center;

[0011] S4. Preprocess the data from different data sources; convert the data from different data sources into a unified format, integrate them, and store the integrated data in the target database.

[0012] Further, the S2 includes:

[0013] Create a data mapping table for each data source and obtain the fields and their descriptions in each data source;

[0014] Perform cluster analysis on the data mapping table of each data source to obtain a first classification;

[0015] According to the data mapping table and the requirement list, score each data source in the first category to obtain a first score;

[0016] Determining a first data source according to the first score, the update frequency of each data source, and data real-time requirements;

[0017] The first score is updated regularly, and the first data source is re-determined according to the update result.

[0018] Furthermore, scoring each data source in the first category according to the data mapping table and the requirement list to obtain a first score includes:

[0019] The first score is obtained by the following formula:

[0020]

[0021]

[0022]

[0023] in, is the first score of the i-th data source in the first category, Score the coverage of the i-th data source in the first category, Score the similarity of the i-th data source in the first category; Score the importance of the jth field of the i-th data source in the first category, is the missing rate of the jth field of the ith data source in the first category, is the total number of fields included in the requirement list; m is the total number of requirement fields included in the i-th data source in the first category; Score the highest similarity between the jth field of the i-th data source and multiple requirement fields in the first category.

[0024] Further, determining the first data source according to the first score, the update frequency of each data source, and the data real-time requirement includes:

[0025] Compare the update frequency of each data source with the real-time requirements, and select the data source whose update frequency meets the feasibility requirements as the first candidate data source;

[0026] Among the first candidate data sources, a data source with a first highest score is selected as the first data source in the first category.

[0027] Furthermore, the regularly updating the first score includes:

[0028] The update cycle is:

[0029]

[0030]

[0031]

[0032] is the feature similarity between the current demand and the previous demand; is the preset threshold of similarity between two adjacent requirements; max() is the maximum value, abs() is the absolute value, T is the current update cycle, is the preset update cycle, is the first score obtained for the k-1th time for the i-th data source; is the first score obtained for the k-2th time for the i-th data source; Based on multiple data sources The maximum value of is the maximum threshold between two consecutive score changes.

[0033] Preferably, S3 includes:

[0034] extracting data from a first data source as primary data;

[0035] Compare the primary data with the target data to obtain missing data;

[0036] Selecting a data source other than the first data source from the first candidate data source and containing the missing data, and performing a second scoring on the missing data;

[0037] Select the data source with the second maximum score of the missing data to supplement the missing data.

[0038] Preferably, the method for obtaining the second score includes:

[0039]

[0040] in, The second score for the e-th missing data based on the i-th data source, Score the highest similarity between the e-th missing data field and multiple required fields in the i-th data source; is the missing rate of the e-th missing data in the i-th data source, and w1 and w2 are weight coefficients.

[0041] Preferably, the S3 further includes:

[0042] According to the time series, the main data are uploaded to the data processing center in the order of the first score;

[0043] Depending on the time series, the missing data are uploaded sequentially or simultaneously in the order of the second score.

[0044] Preferably, the S4 includes:

[0045] Preprocessing the data, including deleting duplicate data and filling missing values;

[0046] After the preprocessed data is converted and reduced, it is integrated into the target database.

[0047] The present application provides a multi-source data processing device, the device comprising a memory and at least one processor, the memory storing a computer program, and the at least one processor being configured to implement the steps of any method described in the present application when executing the computer program.

[0048] The beneficial effects of the present invention include: By first classifying and scoring potential data sources, a data source with high quality and wide coverage can be selected as the main data source, ensuring the accuracy and reliability of the data. At the same time, the second scoring mechanism for missing data can select the most suitable data source for data supplementation, further improving the integrity of the data; at the same time, data duplication is reduced, and the first data source can be dynamically determined according to the update frequency and real-time requirements of the data source, and the order of data extraction and upload is adjusted accordingly; the data processing strategy based on time series and scoring order ensures that important and high-quality data are processed first, significantly improving the efficiency and response speed of data processing. Data from different data sources are preprocessed, converted and reduced to ensure the uniformity and consistency of data formats. Subsequently, the processed data is integrated into the target database, realizing the centralized storage and management of data, and providing convenience for subsequent data analysis and application; the method of dynamically adjusting the update cycle enables the system to flexibly adjust the frequency of score updates according to changes in data source quality and business needs. When the quality of the data source or business needs change significantly, the score can be updated more frequently to reflect these changes; when the change is small, the update cycle can be appropriately extended to reduce unnecessary computing overhead.

[0049] In summary, this multi-source data processing method significantly improves the quality, accuracy and processing efficiency of data through intelligent data source selection, efficient data processing strategies and flexible data management methods, enhances the adaptability and flexibility of the system, and provides strong data support for various business applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 It is a schematic diagram of a multi-source data processing method provided in an embodiment of the present application. DETAILED DESCRIPTION

[0051] Below, the present application is further described in conjunction with the accompanying drawings and specific implementation methods. It should be noted that, under the premise of no conflict, the various embodiments or technical features described below can be arbitrarily combined to form a new embodiment.

[0052] See also Figure 1 Some embodiments of the present application provide a multi-source data processing method, the method comprising:

[0053] S1. Determine the goals of data collection, the data to be collected, and the data sources. The goal of data collection is usually to ensure the completeness, accuracy, and timeliness of the data to meet the specific needs of the enterprise or organization. The following are some possible data collection goals:

[0054] Integrate data: Integrate data from different departments, systems or businesses into a unified data storage for centralized management and analysis. By collecting and analyzing data, potential risks and problems can be discovered in a timely manner, and appropriate measures can be taken to control and prevent them.

[0055] For example, an e-commerce company wants to collect data on sales, inventory, user behavior, etc., in order to conduct sales trend analysis, inventory optimization, and user portrait construction, so as to support its business development.

[0056] The data that needs to be collected depends on the goals of the data collection and the specific business needs; for example:

[0057] Customer information: including customer name, address, contact information, purchase history, etc.;

[0058] Sales data: including product name, sales quantity, sales amount, sales channel, etc.;

[0059] Inventory data: including product inventory, storage time, storage time, inventory cost, etc.;

[0060] Market research data: including market size, competitor analysis, consumer preferences, etc.;

[0061] Network data: including website visits, user behavior, social media data, etc.

[0062] Data source refers to the origin or provider of data. The following are some common data sources:

[0063] Internal data source: refers to data generated within an enterprise or organization, such as databases, files, logs, etc.

[0064] External data source: refers to data from outside the enterprise or organization, such as government public data, industry reports, third-party data providers, etc.

[0065] Network data sources: refers to data collected through the Internet, such as social media, search engines, websites, etc.;

[0066] After determining the data collection objectives, the data to be collected, and the data sources, you can start to develop a data collection implementation plan; the implementation plan should detail the data collection methods, processes, technology selection, and other details to ensure that data collection can proceed smoothly and achieve the expected goals;

[0067] S2. Perform a first classification on potential data sources, score each data source in the first classification, and obtain a first score; determine a first candidate data source and a first data source according to the first score and the update frequency;

[0068] S3, extracting data from the data source and transmitting it to the data processing center;

[0069] S4. Preprocess the data from different data sources; convert the data from different data sources into a unified format, integrate them, and store the integrated data in the target database.

[0070] The working principle of the above technical solution is: determine the target of data collection, the data to be collected and the source of data; first clarify the target of data processing, that is, what kind of business effect needs to be achieved or what analysis needs to be met; then, determine the specific data type and scope that needs to be collected according to the target. Finally, identify and determine the source of these data, including different databases, API interfaces, file systems, etc.; preliminarily classify potential data sources so as to uniformly manage similar or related data sources; then, score the data sources in each category according to multiple indicators of the data source (such as data integrity, accuracy, coverage, etc.) to obtain the first score; during the scoring process, the quality, real-time, reliability and other factors of the data source will be comprehensively considered; after the scoring is completed, the first candidate data source and the first data source will be determined in combination with the update frequency of the data source and the business requirements for data real-time; the first candidate data source is when the first data source has problems or cannot meet all requirements, the alternative data source is selected; and the first data source is the data source that best suits the business needs, has the highest quality and is updated most timely. Use appropriate tools or technologies (such as ETL tools, API interfaces, etc.) to extract data from the selected data source and transfer it to the data processing center; the data processing center is a centralized data processing platform that can uniformly process and manage data from different data sources; pre-process data from different data sources, including data cleaning (such as removing duplicate data, processing missing values, correcting erroneous data, etc.), data conversion (such as converting data into a unified format, unit, etc.) and data specification (such as reducing the amount of data through data aggregation, dimensionality reduction, etc.) and other operations; after the pre-processing is completed, the data from different data sources are integrated together to form a complete data set; during the integration process, the accuracy and consistency of the data will be ensured, while maintaining the integrity and interpretability of the data; finally, the integrated data is stored in the target database for subsequent data analysis and mining; the target database can be a relational database, a NoSQL database or other types of data storage systems, depending on business needs and data characteristics. The entire working principle reflects the efficiency, accuracy and adaptability of the multi-source data processing method proposed in this application, and can meet the needs of modern data processing environments for multi-source data integration and integration.

[0071] The effect of the above technical solution is: through clear data collection goals and data source selection, unnecessary data collection and processing are avoided, thereby improving the overall data processing efficiency. The first classification and scoring mechanism helps to quickly screen out high-quality, highly relevant data sources, reducing the complexity and time cost of data processing; through the scoring mechanism, it can ensure that the selected data source has high data quality, such as accuracy, completeness and consistency, thereby improving the accuracy of subsequent analysis and mining. The preprocessing step can effectively clean and transform data, further improving the quality and availability of data. When determining the data source, the update frequency of the data source and the business's requirements for real-time data are taken into account, thereby ensuring the timeliness and freshness of the data; The mechanism of regularly updating the scoring and re-determining the data source enables the system to adjust in time with changes in data sources and data requirements to maintain its continued effectiveness.

[0072] In some embodiments, the S2 includes:

[0073] Create a data mapping table for each data source and obtain the fields and their descriptions in each data source;

[0074] Perform cluster analysis on the data mapping table of each data source to obtain a first classification;

[0075] According to the data mapping table and the requirement list, score each data source in the first category to obtain a first score;

[0076] Determining a first data source according to the first score, the update frequency of each data source, and data real-time requirements;

[0077] The first score is updated regularly, and the first data source is re-determined according to the update result.

[0078] In some embodiments, the first score is obtained by the following formula:

[0079]

[0080]

[0081]

[0082] in, is the first score of the i-th data source in the first category, Score the coverage of the i-th data source in the first category, Score the similarity of the i-th data source in the first category; The importance score of the jth field of the i-th data source in the first category is greater than 0, and can be between 1-5, or between 0 and 1. There is no specific limitation here. is the missing rate of the jth field of the ith data source in the first category, is the total number of fields included in the requirement list; m is the total number of requirement fields included in the i-th data source in the first category; Score the highest similarity between the jth field of the i-th data source and multiple requirement fields in the first category, The value range is (0,1].

[0083] The working principle of the above technical solution is as follows: create a data mapping table, which is a key tool that lists the fields contained in each data source and describes these fields, which helps to understand the structure and content of each data source and provides a basis for subsequent data processing and analysis; perform cluster analysis on the data mapping table. Cluster analysis is an unsupervised learning method used to group similar data points or objects into the same group (i.e., cluster). Grouping similar data sources into the same category forms the first classification; it helps to identify and manage data sources with similar characteristics.

[0084] According to the data mapping table and the requirements list, score each data source in the first category. The scoring formula involves two main parts: coverage score and similarity score; the coverage score measures the degree to which the data source meets the requirements list. It takes into account the field missing rate of the data source and the importance score of the field. The lower the field missing rate and the higher the importance of the field, the higher the coverage score; the similarity score measures the similarity between the fields in the data source and the fields in the requirements list. Here, the highest similarity score is used, that is, the maximum match between each data source and the requirements list; the final first score is the product of the coverage score and the similarity score, which comprehensively considers the performance of the data source in terms of coverage and similarity.

[0085] The first data source is determined based on the first score, the update frequency of each data source, and the data real-time requirements. The first data source is the data source with the highest score, the update frequency that meets the requirements, and good real-time performance; if there are multiple data sources that meet these conditions, one of them can be selected as the main data source and the others as alternative data sources; the quality and characteristics of the data source may change over time. Therefore, the first score is updated regularly, and the first data source is re-determined based on the update results, ensuring that the data processing process can be adjusted in time with the changes in the data source to maintain its continued effectiveness.

[0086] Through the above steps, the multi-source data processing method proposed in this application can efficiently and accurately select and manage data from different data sources, providing high-quality data support for subsequent data processing and analysis.

[0087] The effect of the above technical solution is: by creating a data mapping table and performing cluster analysis, the structure and content of each data source can be quickly understood, and it can be classified into groups with similar characteristics, reducing the workload of manual analysis of data sources and improving the efficiency of data source selection; using the scoring mechanism, the quality and applicability of the data source can be comprehensively considered based on the coverage score and similarity score, so as to select the data source that best suits the business needs as the first data source. The coverage score considers the field missing rate and field importance score of the data source, ensuring the integrity and accuracy of the data source on key fields, which helps to reduce errors and deviations in subsequent data processing and improve data quality. The similarity score measures the similarity between the data source field and the field in the requirement list, further ensuring the consistency and availability of the data. When determining the first data source, not only the score of the data source is considered, but also the update frequency and real-time requirements of the data source are combined to ensure that the selected data source can provide the latest and most accurate data in a timely manner to meet the business's real-time requirements. The regular update of the scoring mechanism allows the system to re-evaluate the quality and applicability of the data source according to changes in the data source and changes in business needs, ensuring the flexibility and adaptability of data source selection, so that the system can continue to provide high-quality data support.

[0088] In some embodiments, determining the first data source according to the first score, the update frequency of each data source, and the data real-time requirement includes:

[0089] Compare the update frequency of each data source with the real-time requirements, and select the data source whose update frequency meets the feasibility requirements as the first candidate data source;

[0090] Among the first candidate data sources, a data source with a first highest score is selected as the first data source in the first category.

[0091] The working principle and effect of the above technical solution are as follows: when selecting a data source, the first thing to consider is the update frequency of the data source and the real-time requirements of the business. The real-time requirements refer to the business's requirements for data freshness and timeliness. The system will compare the update frequency and real-time requirements of each data source to ensure that the data source can provide sufficiently fresh data to support business needs. By screening out data sources whose update frequency meets the real-time requirements, a preliminary list of candidate data sources, namely the first candidate data source, is constructed. After the first candidate data source is determined, it is further selected based on the first score of the data source. The first score is calculated by comprehensively considering the coverage score and similarity score of the data source, which can fully reflect the performance of the data source in terms of quality and applicability. Among the first candidate data sources, the data source with the highest first score is selected as the first data source in the first category. This is because the data source with a high first score performs well in terms of coverage and similarity, and can better meet business needs and provide high-quality data support. It should be emphasized that this process is dynamic and adjustable. As the data source and data requirements change, the system can regularly update the score and re-determine the first data source based on the new score results, ensuring that the system can continue to provide the most suitable data source to meet the changing needs of the business.

[0092] To summarize, the working principle of determining the first data source is to compare the update frequency of the data source with the real-time requirements, screen out the data sources that meet the real-time requirements as alternatives, and further select the data source with the highest score as the first data source based on the score. This process ensures the freshness, high quality and applicability of the data source, and provides a solid foundation for subsequent data processing and analysis.

[0093] In some embodiments, the periodically updating the first score includes:

[0094] The update cycle is:

[0095]

[0096]

[0097]

[0098] is the feature similarity between the current demand and the previous demand, and its value range is (0,1]; is the preset threshold of similarity between two adjacent requirements; max() is the maximum value, abs() is the absolute value, T is the current update cycle, is the preset update cycle, is the first score obtained for the k-1th time for the i-th data source; is the first score obtained for the k-2th time for the i-th data source; Based on multiple data sources The maximum value of is the maximum threshold between two consecutive score changes.

[0099] The working principle and effect of the above technical solution are:

[0100] Indicates the similarity between the characteristics of the current demand and the previous demand. This similarity reflects the degree of change in business demand. smaller), you may need to update the score more frequently to reflect the suitability of the data source for your current needs.

[0101] is the preset similarity threshold of two adjacent requirements. Greater than , it means that the change in business demand is relatively small, then only the change in the score is considered.

[0102] and They represent the first score of the i-th data source at the k-1th and k-2th scoring times respectively; the change between these two scores reflects the change in the quality of the data source.

[0103] Calculated from all data sources The larger the value, the greater the change in data source quality, and the more frequent the score update may be needed.

[0104] Dynamically adjust the update cycle:

[0105] In the formula This is partly to comprehensively consider the impact of demand similarity and score changes on the update cycle; this maximum value determines the extent to which the update cycle needs to be shortened.

[0106] Dangdang Greater than When , it means that the demand change is relatively small, so the update cycle is adjusted mainly based on the score change. At this time, the calculation formula of T is simplified to ;

[0107] when When , it means that the demand changes greatly, so the update cycle is adjusted by considering both the demand similarity and the score change. At this time, the calculation formula of T is

[0108] The method of dynamically adjusting the update cycle enables the system to flexibly adjust the frequency of score updates based on changes in data source quality and business needs. When data source quality or business needs change significantly, the system can update the score more frequently to reflect these changes; when the changes are small, the update cycle can be appropriately extended to reduce unnecessary computing overhead.

[0109] In summary, by comprehensively considering demand similarity and score changes, the system can dynamically adjust the update cycle of the first score, thereby ensuring the accuracy and timeliness of data source selection while maintaining the adaptability and flexibility of the system.

[0110] In some embodiments, the S3 includes:

[0111] extracting data from a first data source as primary data;

[0112] Compare the primary data with the target data to obtain missing data;

[0113] Selecting a data source other than the first data source from the first candidate data source and containing the missing data, and performing a second scoring on the missing data;

[0114] Select the data source with the second maximum score of the missing data to supplement the missing data.

[0115] In some embodiments, the method for obtaining the second score includes:

[0116]

[0117] in, The second score for the e-th missing data based on the i-th data source, The highest similarity score between the e-th missing data field and multiple required fields in the i-th data source is given in the range of (0,1]; is the missing rate of the e-th missing data in the i-th data source, w1 and w2 are weight coefficients, 0 <w1<1, 0<w2<1, w1+ w2=1。

[0118] The working principle and effect of the above technical solution are as follows: First, extract data from the determined first data source as the main data; these data are usually selected based on previous scores, update frequency and real-time requirements, and are considered to be optimal in quality and applicability. Compare the extracted main data with the target data, where "target data" usually refers to the data state or standard that the business or system needs to achieve; through comparison, the system can identify the missing parts of the main data, that is, the missing data; after determining the missing data, find the data source containing these missing data in the first candidate data source, and for each candidate data source, perform a second score for each missing data; the second score is based on two main factors: field similarity and missing rate. Field similarity measures the similarity between the missing data field and the required field in the candidate data source, while the missing rate reflects the completeness of the field in the data source. Use weight coefficients w1 and w2 to weight these two factors and obtain the second score of each missing data based on each candidate data source. The weight coefficient can be adjusted according to business needs or system policies to emphasize different factors; after obtaining the second score of each missing data from all candidate data sources, the data source with the maximum second score is selected to supplement the missing data; this is based on the assumption that the data source with a high second score has higher quality and reliability when providing missing data. In this way, a complete data set can be effectively obtained from multiple data sources to meet business needs or system requirements.

[0119] By intelligently selecting and using multiple data sources, the integrity and accuracy of the data is ensured, and the efficiency and reliability of data processing are improved. At the same time, this flexible and configurable approach also enables it to adapt to different business scenarios and data requirements.

[0120] In some embodiments, the S3 further includes:

[0121] According to the time series, the main data are uploaded to the data processing center in the order of the first score;

[0122] Depending on the time series, the missing data are uploaded sequentially or simultaneously in the order of the second score.

[0123] The working principle and effect of the above technical solution are as follows: After the main data are determined, these data are uploaded to the data processing center in the order of the first score according to the time series; the "time series" here refers to the time order of data generation, and the "first score" is the score calculated based on factors such as the quality, coverage and similarity of the data source. A data source with a high score usually means higher quality and more reliable data. Therefore, uploading the main data in the order of the first score can ensure that the data with higher quality is processed first, improving the efficiency and accuracy of data processing. After the missing data are identified and the supplementary data source is selected, the missing data will also be uploaded in sequence or simultaneously in the order of the second score according to the time series; the "second score" here is the score calculated for each missing data based on the alternative data source, mainly considering factors such as field similarity and missing rate. Similar to uploading the main data, uploading the missing data in the order of the second score is also to ensure that the data with higher quality is processed first. However, in some cases, in order to speed up data processing, you can also choose to upload multiple missing data with higher second scores at the same time. By uploading data in the order of time series and score, the system can ensure the orderliness and efficiency of data processing. Orderliness means that data is processed in a certain order, which helps reduce confusion and errors; efficiency means that the system can give priority to processing higher quality data, thereby improving the efficiency and accuracy of data processing.

[0124] This data upload method based on time series and scoring order also has certain adaptability and flexibility. For example, the scoring weight or upload strategy can be adjusted according to actual needs to adapt to different business scenarios and data requirements. In addition, the upload speed or order can be dynamically adjusted according to the load of the data processing center to ensure the stability and reliability of data processing.

[0125] In some embodiments, the S4 includes:

[0126] Preprocessing the data, including deleting duplicate data and filling missing values;

[0127] After the preprocessed data is converted and reduced, it is integrated into the target database.

[0128] The working principle and effect of the above technical solution are: use algorithms or tools to identify and delete duplicates in a data set. This is usually achieved by comparing records in a data set, based on specific fields (such as primary keys) or the content of the entire record. Missing values ​​refer to situations where certain fields or records in a data set lack information. In order to maintain the integrity and accuracy of the data, the system needs to fill these missing values; there are many ways to fill them, such as using statistical values ​​such as mean, median, mode, etc. to replace missing values, or using machine learning algorithms (such as decision trees, K nearest neighbors, etc.) to predict missing values.

[0129] Data transformation refers to the conversion of data from one format or structure to another format or structure to meet the requirements of data analysis or data warehouse, including data type conversion (such as converting text to numbers), data unit conversion (such as converting miles to kilometers), data standardization (such as scaling data to the same range), etc. Data reduction is the process of simplifying data analysis tasks by reducing the size of the data set; including methods such as removing unnecessary attributes (dimensionality reduction), data aggregation (such as calculating averages, sums, etc.), and data sampling (selecting subsets from large data sets). The reduced data retains the key features of the original data, but reduces the complexity of processing and analysis. After preprocessing and conversion reduction, the system will integrate the processed data into the target database. The target database is a system for storing and managing data to support various business applications and analysis tasks; the integration process may include steps such as loading data into the database, creating indexes, setting permissions and security. The integrated data can be further analyzed, queried, and report generated in the database.

[0130] The working principle of the entire S4 step is to ensure the accuracy and completeness of the data through preprocessing, simplify the data analysis task through data conversion and specification, and finally integrate the processed data into the target database for subsequent data analysis and application. This process helps to improve the efficiency and quality of data processing and support more accurate business decisions and insights.

[0131] An embodiment of the present application further provides a multi-source data processing device, the device comprising a memory and at least one processor, the memory storing a computer program, and the at least one processor being configured to implement the steps of any method described in the present application when executing the computer program.

[0132] Although the embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and cannot be understood as limiting the present invention. A person skilled in the art can change, modify, replace and modify the above embodiments within the scope of the invention without departing from the principles and purpose of the present invention, and all these changes should fall within the scope of protection of the claims of the present invention. This application is explained from the perspectives of purpose of use, efficiency, progress and novelty, and has met the functional enhancement and use requirements emphasized by the patent law. The above description and drawings of this application are only preferred embodiments of this application, and are not intended to limit this application. Therefore, all structures, devices, features, etc. that are similar or identical to this application, that is, all equivalent replacements or modifications made according to the scope of the patent application of this application, should fall within the scope of protection of the patent application of this application.

Claims

1. A multi-source data processing method, characterized in that: The method comprises: S1. Determine the data collection objectives, data to be collected, and data sources; S2. Perform a first classification on potential data sources, score each data source in the first classification, and obtain a first score; determine a first candidate data source and a first data source according to the first score and the update frequency; S3, extracting data from the data source and transmitting it to the data processing center; S4, preprocessing the data from different data sources; converting the data from different data sources into a unified format, integrating them, and storing the integrated data in the target database; Wherein, the S2 includes: Create a data mapping table for each data source and obtain the fields and their descriptions in each data source; Perform cluster analysis on the data mapping table of each data source to obtain a first classification; According to the data mapping table and the requirement list, score each data source in the first category to obtain a first score; Determining a first data source according to the first score, the update frequency of each data source, and data real-time requirements; regularly updating the first score, and re-determining the first data source according to the update result; The step of scoring each data source in the first category according to the data mapping table and the requirement list to obtain a first score includes: The first score is obtained by the following formula: P i =F i ×S i Among them, P i is the first score of the i-th data source in the first category, F i Score the coverage of the i-th data source in the first category, S i Score the similarity of the i-th data source in the first category; Z ij Score the importance of the jth field of the i-th data source in the first category, C ij is the missing rate of the jth field of the ith data source in the first category, m a is the total number of fields included in the requirement list; m is the total number of requirement fields included in the i-th data source in the first category; S ij Score the highest similarity between the jth field of the i-th data source and multiple requirement fields in the first category; The regularly updating the first score includes: The update cycle is: S a is the feature similarity between the current demand and the previous demand; S y is the preset threshold of similarity between two adjacent requirements; max() is the maximum value, abs() is the absolute value, T is the current update cycle, T y is the preset update cycle, P i(k-1) is the first score obtained for the k-1th time by the ith data source; P i(k-2) is the first score obtained for the k-2th time for the ith data source; max(α i ) is the α obtained from multiple data sources i ΔY is the maximum threshold of the change between two adjacent scores.

2. The multi-source data processing method according to claim 1, characterized in that: The determining the first data source according to the first score, the update frequency of each data source, and the data real-time requirement includes: Compare the update frequency of each data source with the real-time requirements, and select the data source whose update frequency meets the feasibility requirements as the first candidate data source; Among the first candidate data sources, a data source with a first highest score is selected as the first data source in the first category.

3. The multi-source data processing method according to claim 1, characterized in that: The S3 includes: extracting data from a first data source as primary data; Compare the primary data with the target data to obtain missing data; Selecting a data source other than the first data source from the first candidate data source and containing the missing data, and performing a second scoring on the missing data; Select the data source with the second maximum score of the missing data to supplement the missing data.

4. The multi-source data processing method according to claim 3, characterized in that: The method for obtaining the second score includes: P ie =w1×S ie +w2×C ie Among them, P ie is the second score of the e-th missing data based on the i-th data source, S ie Score the highest similarity between the e-th missing data field and multiple required fields in the i-th data source; C ie is the missing rate of the e-th missing data in the i-th data source, and w1 and w2 are weight coefficients.

5. The multi-source data processing method according to claim 3, characterized in that: The S3 further includes: According to the time series, the main data are uploaded to the data processing center in the order of the first score; Depending on the time series, the missing data are uploaded sequentially or simultaneously in the order of the second score.

6. The multi-source data processing method according to claim 1, characterized in that: The S4 includes: Preprocessing the data, including deleting duplicate data and filling missing values; After the preprocessed data is converted and reduced, it is integrated into the target database.

7. A multi-source data processing device, characterized in that: The device comprises a memory and at least one processor, wherein the memory stores a computer program, and the at least one processor is configured to implement the steps of the method according to any one of claims 1 to 6 when executing the computer program.

Images