A multi-source data processing method and system for supplier evaluation

Abstract

The present invention proposes a multi-source data processing method and system for supplier evaluation. The method belongs to the field of data fusion processing technology. The method comprises: collecting multi-source data from different sources and pre-processing the collected multi-source data; determining the main data source and related data sources according to the supplier evaluation requirements, extracting valid data from the main data source based on preset data screening rules, and associating the extracted valid data with the data in the related data sources. By collecting and pre-processing data from different sources, the quality of the data is ensured.

Description

Technical Field

[0001] The invention proposes a multi-source data processing method and system for supplier evaluation, belonging to the technical field of data fusion processing. Background Art

[0002] With the acceleration of globalization and digitalization, enterprise supply chain management is becoming increasingly complex. As a key link in supply chain management, the accuracy and comprehensiveness of supplier evaluation directly affect the operational efficiency and market competitiveness of enterprises. Traditional supplier evaluation methods mainly rely on a single data source, such as internal transaction data of the enterprise or self-reported information from suppliers. This method has problems such as single data, one-sided information, and susceptibility to subjective factors, making it difficult to fully and objectively reflect the true performance of suppliers.

[0003] As an important feature of modern data processing technology, multi-source data provides new possibilities for supplier evaluation due to its large data volume, wide sources, and diverse structures. Multi-source data not only includes structured data such as internal transaction data and financial statements, but also covers unstructured or semi-structured data such as social media comments, industry reports, and market research. These data sources together constitute a comprehensive information base for supplier evaluation, which can more truly and comprehensively reflect key factors such as supplier capabilities, quality, and reputation.

[0004] However, the processing of multi-source data also faces many challenges. First, the data formats, structures and quality standards of different sources are different, and pre-processing work such as data cleaning, conversion and standardization is required to ensure the consistency and comparability of the data. Secondly, there are complex relationships and potential conflicts between multi-source data, which need to be integrated and consolidated through appropriate data fusion algorithms to form a comprehensive and accurate comprehensive data set. Finally, the processing and analysis of multi-source data requires the use of advanced machine learning, data mining and other intelligent technologies to extract key evaluation indicators and features to provide strong support for supplier evaluation. Summary of the invention

[0005] The present invention provides a multi-source data processing method and system for supplier evaluation to solve the problems mentioned in the above background technology:

[0006] The present invention proposes a multi-source data processing method for supplier evaluation, the method comprising:

[0007] S1. Collect multi-source data from different sources and pre-process the collected multi-source data;

[0008] S2. Determine the main data source and related data sources according to the supplier evaluation requirements, extract valid data from the main data source based on the preset data screening rules, and associate the extracted valid data with the data in the related data sources;

[0009] S3. Integrate valid data and valid related data to form a comprehensive data set, and build an indicator system for supplier evaluation based on the integrated multi-source data;

[0010] S4. For each evaluation dimension, a data processing model is constructed to conduct in-depth analysis of multi-source data, extract key evaluation indicators and features, and use a comprehensive evaluation algorithm to conduct a comprehensive evaluation of suppliers based on the evaluation indicators and features of each dimension;

[0011] S5. Based on the results of the comprehensive evaluation algorithm, output the supplier's comprehensive evaluation report and feed back the evaluation results to the supplier and relevant personnel.

[0012] Furthermore, the S1 includes:

[0013] S11. Collect multi-source data from different sources through collection scripts or API interfaces;

[0014] S12, sending the collected multi-source data to a cloud space, where the cloud space stores the received multi-source data through a hierarchical storage structure;

[0015] S13. Preprocess the stored multi-source data, encrypt the sensitive data, and desensitize the non-sensitive but privacy-related data.

[0016] Furthermore, the S12 includes:

[0017] The uploaded multi-source data is compressed by a compression algorithm to obtain a compressed first data packet, and the compressed data packet is redundantly backed up to obtain a second data packet, and the first data packet and the second data packet are respectively sent to the cloud space through a multi-channel transmission protocol;

[0018] After receiving the first data packet and the second data packet, the cloud space stores the second data packet in the second storage space; and stores the first data packet in the first storage space;

[0019] Decompressing the first data packet in the first storage space, and capturing the data source in the first storage space that needs to be processed immediately by using a stream processing framework through a real-time data stream processing layer;

[0020] And eliminate erroneous or invalid data, analyze the processed data in real time, extract key indicators or perform preliminary data aggregation;

[0021] Build a layered historical data storage architecture in the first storage space, using a combination of object storage and block storage to automatically migrate data based on data access frequency and importance;

[0022] Introduce a data archiving strategy to migrate data that has been inactive for a long time but still needs to be retained to a cold storage area, and use search engine technology to index historical data.

[0023] Furthermore, the S2 includes:

[0024] S21. Through a multi-dimensional data source quality assessment framework, quantitatively score each data source based on preset thresholds and weights to identify the primary data source;

[0025] S22. For the primary data source, automatically adjust the extraction time interval based on the frequency of data update, and use regular expressions and machine learning models to identify and extract key information fields;

[0026] S23. During the extraction process, the logical relationship and consistency between historical data and verification data are compared based on the data quality secondary verification mechanism;

[0027] S24. When associating related data sources, dynamically adjust association rules based on data characteristics and business needs through common identification and combined with multi-dimensional association strategies;

[0028] S25. After association, a consistency check is performed; and a feedback mechanism is established to feed back the consistency check results to the data collection and processing links. At the same time, the association rules and verification algorithms are continuously optimized based on the feedback results to form a closed-loop iterative optimization process.

[0029] Furthermore, the S22 includes:

[0030] Monitor the update log or timestamp of the main data source in real time or regularly, record the time interval of each data update, and analyze the historical update frequency data to obtain analysis results;

[0031] According to the analysis results, the initial data extraction time interval is set. Through the dynamic adjustment mechanism, when the data update frequency changes, the extraction time interval is automatically adjusted to match the new update speed;

[0032] For structured data, regular expressions are used to locate and extract key information fields;

[0033] For unstructured or semi-structured data, identify and extract key information through existing named entity recognition models;

[0034] Combine regular expressions and named entity recognition models to build a hybrid extraction strategy;

[0035] Monitor the extraction process in real time, record the key indicators of each extraction task, and through the anomaly detection mechanism, trigger an alarm and take corresponding emergency measures when an anomaly occurs during the extraction process;

[0036] Through the extraction result verification mechanism, the extracted data is preliminarily verified, and the effectiveness of the extraction strategy is evaluated based on the data quality analysis of the extraction results; according to the analysis results, the hybrid strategy is optimized and adjusted.

[0037] Furthermore, the S3 includes:

[0038] S31. Identify and process heterogeneity issues among multi-source data; and convert data from different sources into a unified format;

[0039] S32. Based on the multi-dimensional indicators of data, the data weight distribution mechanism is designed through the hierarchical analysis method and the fuzzy comprehensive evaluation method combined with the expert database; and the weight distribution mechanism is dynamically adjusted through the dynamic adjustment strategy of the weight;

[0040] S33, fusing the data through a fusion algorithm, and performing outlier detection and data cleaning mechanisms during the fusion process;

[0041] S34. After the integration is completed, the integrated data is verified and evaluated by comparing historical data, industry benchmarks and business logic;

[0042] S35. Divide the index system into multiple dimensions based on the needs and goals of supplier evaluation; further refine the evaluation indicators under each dimension to form a multi-level index system structure; establish an index dynamic adjustment mechanism to dynamically adjust the index system based on influencing factors.

[0043] Furthermore, the S33 includes:

[0044] Select the fusion algorithm based on data characteristics (such as data volume, data distribution, degree of heterogeneity, etc.) and business needs; use statistical methods to re-detect outliers in the data, remove or correct them; and clean the multi-source data after re-detection;

[0045] Formulate fusion strategies based on the selected fusion algorithm and data characteristics, perform data fusion operations according to the fusion strategies, and monitor the fusion process in real time;

[0046] By calculating the statistical indicators and visual analysis of the fusion results, the quality of the fusion results is preliminarily evaluated;

[0047] Based on the preliminary evaluation results, feedback and adjustments are made to the fusion strategy, and the fusion algorithm and strategy are continuously optimized through multiple iterations.

[0048] Furthermore, the S4 includes:

[0049] S41. Build advanced models by combining multiple ensemble learning methods with multiple machine learning algorithms, and perform parameter tuning and model fusion; use deep learning models to process complex data, extract deep features and perform predictive analysis;

[0050] S42. Evaluate the model performance through cross-validation and ROC curve analysis, and select the model for deployment based on the evaluation results;

[0051] S43. According to the evaluation requirements, an evaluation method is selected, and a composite evaluation algorithm is formed through weighted fusion and nonlinear combination. The weight of each evaluation method is automatically adjusted according to the changes in the evaluation objectives and the real-time update of the data through the dynamic weight adjustment mechanism;

[0052] S44. Build a risk assessment model through machine learning to quantitatively assess the risks faced by suppliers; by presetting early warning thresholds, when the risk assessment results exceed the preset thresholds, an early warning is automatically triggered and responded to through the early warning response mechanism.

[0053] Furthermore, the S5 includes:

[0054] S51. Build a trend prediction system based on time series analysis or machine learning models to model suppliers’ historical performance data and predict future development trends;

[0055] S52. Compare and analyze supplier evaluation results with industry benchmarks, and generate personalized improvement suggestions based on the supplier's specific performance;

[0056] S53. Establish an improvement effect tracking mechanism to regularly collect supplier performance data after adopting improvement suggestions, evaluate the improvement effect, and adjust the suggestions based on feedback;

[0057] S54. Establish a multi-channel user feedback collection mechanism, and organize and analyze the collected user feedback through an iterative optimization process, and formulate corresponding optimization plans; continuously iterate and optimize system functions and performance based on optimization strategies.

[0058] The present invention proposes a system for implementing the multi-source data processing method for supplier evaluation, the system comprising:

[0059] Data collection module: collects multi-source data from different sources and pre-processes the collected multi-source data;

[0060] Data association module: Determine the main data source and related data sources according to the supplier evaluation requirements, extract valid data from the main data source based on the preset data screening rules, and associate the extracted valid data with the data in the related data sources;

[0061] Data fusion module: fuses valid data and valid related data to form a comprehensive data set, and builds an indicator system for supplier evaluation based on the fused multi-source data;

[0062] Comprehensive evaluation module: For each evaluation dimension, a data processing model is constructed to conduct in-depth analysis of multi-source data, extract key evaluation indicators and features, and use a comprehensive evaluation algorithm to conduct a comprehensive evaluation of suppliers based on the evaluation indicators and features of each dimension;

[0063] Result feedback module: Based on the results of the comprehensive evaluation algorithm, output the supplier's comprehensive evaluation report and feedback the evaluation results to the supplier and relevant personnel.

[0064] Beneficial effects of the present invention: By collecting and preprocessing data from different sources, the quality of data is ensured. By adopting cloud storage technology and data processing mechanism, the security and reliability of data can be enhanced. Sensitive information can be protected by encrypting and desensitizing data. By utilizing the data source quality assessment framework, the most reliable and representative data sources can be identified. The timeliness of data is ensured by dynamically adjusting the data extraction time and rules. The fusion and standardization of data solve the problem of data heterogeneity, so that data can be effectively compared and analyzed on the same platform. By combining multiple machine learning algorithms and integrated learning methods, the accuracy and depth of data analysis are improved. The use of composite evaluation algorithms and dynamic weight adjustment mechanisms can more comprehensively reflect the performance of suppliers. The use of risk assessment models can help discover potential problems in advance and reduce the risk of supply chain disruptions. The trend prediction system can provide insights into future performance changes of suppliers and help enterprises make more informed decisions. By comparing and analyzing with industry benchmarks, the strengths and weaknesses of suppliers can be identified. The improvement effect tracking mechanism ensures the effective implementation of improvement suggestions and promotes the process of continuous optimization. Providing detailed evaluation reports to suppliers helps to enhance mutual understanding and trust. The user feedback collection mechanism ensures continuous improvement of the system to meet changing business needs. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] Figure 1 It is a step diagram of the method of the present invention;

[0066] Figure 2 This is a system module diagram of the present invention. DETAILED DESCRIPTION

[0067] In order to more clearly understand the above-mentioned purpose, features and advantages of the present invention, the present invention is described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be noted that the embodiments of the present application and the features in the embodiments can be combined with each other without conflict.

[0068] In the following description, many specific details are set forth to facilitate a full understanding of the present invention. The embodiments described are only a part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0069] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art of the present invention. The terms used in the specification of the present invention herein are only for the purpose of describing specific embodiments and are not intended to limit the present invention.

[0070] One embodiment of the present invention, as Figure 1 As shown, a multi-source data processing method for supplier evaluation includes:

[0071] S1. Collect multi-source data from different sources and pre-process the collected multi-source data;

[0072] S2. Determine the main data source and related data sources according to the supplier evaluation requirements. The main data source is the core data in the evaluation process, such as financial statements; the related data source is auxiliary data, such as social media evaluations; based on the preset data screening rules, extract valid data from the main data source, and associate the extracted valid data with the data in the related data source; in the association process, use the common identifiers in the data (such as supplier ID, product number, etc.) for matching;

[0073] S3. Integrate valid data and valid related data to form a comprehensive data set, and build an indicator system for supplier evaluation based on the integrated multi-source data;

[0074] S4. For each evaluation dimension, build a data processing model, conduct in-depth analysis of multi-source data, extract key evaluation indicators and features, and use comprehensive evaluation algorithms (such as hierarchical analysis method, fuzzy comprehensive evaluation method, etc.) to conduct comprehensive evaluation of suppliers based on the evaluation indicators and features of each dimension;

[0075] S5. Based on the results of the comprehensive evaluation algorithm, a comprehensive evaluation report of the supplier is output, which includes the supplier's score, ranking, strengths and weaknesses; and the evaluation results are fed back to the supplier and relevant personnel to help the supplier understand its own problems and weaknesses and formulate corresponding improvement measures. At the same time, the data processing model and evaluation algorithm are continuously optimized and improved based on the evaluation results and feedback.

[0076] The working principle of the above technical solution is: collect multi-source data related to supplier evaluation from different channels (such as internal enterprise systems, public databases, social media, third-party evaluation platforms, etc.). The multi-source data includes various types such as financial statements, transaction records, customer evaluations, and social media feedback; pre-process the collected multi-source data; and determine the main data source (such as financial statements) and related data sources (such as social media evaluations) according to the evaluation requirements. Among them, the main data source is the core of the evaluation, and the related data source is used to provide auxiliary information; based on the preset data screening rules, extract valid data from the main data source. The data screening rules involve the time range, integrity, and rationality of the data, such as screening the latest data obtained within one week; use the common identifiers in the data (such as supplier ID, product number, etc.) to match and associate the extracted valid data with the data in the related data source. ; Merge the valid data and the valid related data to form a comprehensive data set. Based on the fused multi-source data, construct an indicator system for supplier evaluation. The index system covers multiple aspects of suppliers (such as product quality, delivery capability, service attitude, price competitiveness, etc.), and determines specific evaluation indicators for each aspect, such as product quality must meet national standards or service attitude must be five stars; for each evaluation dimension, a data processing model is constructed to conduct in-depth analysis of multi-source data and extract key evaluation indicators and features. Based on the evaluation indicators and features of each dimension, a comprehensive evaluation algorithm (such as hierarchical analysis method, fuzzy comprehensive evaluation method, etc.) is used to comprehensively evaluate suppliers, comprehensively consider multiple factors, and give a comprehensive score and ranking; according to the results of the comprehensive evaluation algorithm, a comprehensive evaluation report of the supplier is output. The comprehensive evaluation report includes information on the supplier's score, ranking, advantages and disadvantages; the evaluation results are fed back to suppliers and relevant personnel to help them understand their own problems and deficiencies and formulate corresponding improvement measures. At the same time, the data processing model and evaluation algorithm are continuously optimized and improved based on the evaluation results and feedback.

[0077] The effects of the above technical solution are as follows: by collecting and integrating multi-source data from different sources (such as financial statements, social media reviews, customer feedback, etc.), it can more comprehensively reflect the true situation of suppliers and avoid the one-sidedness or limitations brought by a single data source; by building an evaluation index system based on the integrated multi-source data, it can cover multiple aspects of suppliers (such as product quality, price, delivery capability, service attitude, innovation capability, etc.), thereby achieving multi-dimensional and comprehensive evaluation; by preprocessing and screening the collected multi-source data, it can effectively remove noise data and outliers, and improve the accuracy and reliability of the data; in the process of data association, using common identifiers (such as supplier ID, product number, etc.) for precise matching can ensure the consistency and accuracy of the data and reduce the evaluation errors caused by inconsistent data; the output comprehensive evaluation report includes the supplier's score, Rankings, strengths and weaknesses and other information can provide decision makers with a clear and intuitive reference, helping them make more scientific and reasonable decisions; feeding back the evaluation results to suppliers and relevant personnel can help suppliers understand their own problems and shortcomings, so as to formulate targeted improvement measures and improve overall performance; based on the evaluation results and feedback, continuously optimizing and improving data processing models and evaluation algorithms can improve the accuracy and effectiveness of the evaluation and ensure the continuous optimization and improvement of the evaluation system; the process of multi-source data processing and comprehensive evaluation increases the transparency of supplier evaluation, helps to establish a more fair and just evaluation system, and promotes healthy competition among suppliers; through the feedback mechanism, suppliers can promptly understand their own shortcomings and take measures to improve them, which not only helps to enhance the competitiveness of the suppliers themselves, but also helps enterprises to establish long-term and stable cooperative relationships with them and achieve win-win results.

[0078] In one embodiment of the present invention, the S1 includes:

[0079] S11. Collect multi-source data from different sources through collection scripts or API interfaces; the multi-source data includes financial statements provided by suppliers, historical transaction records, logistics tracking data, social media reviews, and market research reports;

[0080] S12, sending the collected multi-source data to a cloud space, where the cloud space stores the received multi-source data through a hierarchical storage structure;

[0081] S13. Preprocess the stored multi-source data, encrypt the sensitive data, and desensitize the non-sensitive but privacy-related data.

[0082] The working principle of the above technical solution is: through the collection script, relevant data is obtained from various data sources (such as databases, websites, file systems, etc.). For data sources that support API interfaces, data is obtained directly by calling API. The collected multi-source data includes financial statements provided by suppliers (reflecting financial status and operating results), historical transaction records (recording transaction details and payment status), logistics tracking data (providing cargo transportation status and location information), social media reviews (reflecting the public's views on suppliers' products or services) and market research reports (including industry trends, competitor analysis and other information); the collected multi-source data is sent to the cloud space for storage. The cloud space uses a hierarchical storage structure to manage data. According to factors such as data access frequency, importance or storage cost, the data is divided into different levels (such as hot data layer, warm data layer, cold data layer), and stored on storage media with different performance, cost and availability. For example, data with high access frequency can be divided into the hot data layer and stored on storage media with better performance. Preprocess the multi-source data stored in the cloud space, including data cleaning (removing duplicate, erroneous or irrelevant data), data conversion (converting data into a unified format or structure) and data integration (integrating data from different sources into a unified data set). During the data preprocessing process, sensitive data is encrypted. The sensitive data includes financial information, personal identity information, etc. Through encryption, the security of sensitive data during storage and transmission can be ensured; for non-sensitive but privacy-related data (such as user names, phone numbers, etc.), desensitization is performed. Among them, desensitization is to reduce the privacy risks in the data by replacing, deleting or deforming, while retaining the analysis and evaluation value of the data.

[0083] The effects of the above technical solutions are as follows: by collecting multi-source data through acquisition scripts or API interfaces, it can flexibly adapt to the characteristics and access methods of different data sources to ensure the comprehensiveness and timeliness of data; through the automated data collection process, human intervention is reduced, the efficiency and accuracy of data collection are improved, and the risk of human error is reduced; the collected multi-source data includes financial statements, historical transaction records, logistics tracking data, social media evaluations, and market research reports, etc., and the collected diverse data sources provide a rich information basis for subsequent supplier evaluation; the cloud space provides almost unlimited scalability, which can easily cope with the rapid growth of data volume and ensure the flexibility and efficiency of data storage; the cloud space usually adopts redundant storage and backup mechanisms to ensure Ensure high data availability, and even in the event of hardware failure or natural disasters, the integrity and accessibility of data can be guaranteed; through a hierarchical storage structure, storage resources are reasonably allocated according to the access frequency and importance of data, which reduces storage costs and improves storage efficiency; through preprocessing steps such as data cleaning, conversion and integration, the quality and availability of data are improved, providing an accurate and consistent data set for subsequent data analysis and evaluation; by encrypting sensitive data, the security of data during storage and transmission can be ensured, and the risk of data leakage and illegal access can be prevented; non-sensitive but privacy-related data is desensitized to reduce privacy risks in the data, while retaining the analysis and evaluation value of the data, achieving a balance between privacy protection and data utilization.

[0084] In one embodiment of the present invention, the S12 includes:

[0085] The uploaded multi-source data is compressed by a compression algorithm to obtain a compressed first data packet, and the compressed data packet is redundantly backed up to obtain a second data packet, and the first data packet and the second data packet are respectively sent to the cloud space through a multi-channel transmission protocol;

[0086] After receiving the first data packet and the second data packet, the cloud space stores the second data packet in the second storage space; and stores the first data packet in the first storage space; wherein the second data packet in the second storage space is for backing up the data;

[0087] Decompress the first data packet in the first storage space, and capture the data source that needs to be processed immediately in the first storage space by using a stream processing framework (such as Apache Kafka and Apache Flink) through a real-time data stream processing layer;

[0088] And eliminate erroneous or invalid data, analyze the processed data in real time, extract key indicators (KPIs) or perform preliminary data aggregation;

[0089] Build a layered historical data storage architecture in the first storage space, using a combination of object storage (such as Amazon S3, Alibaba Cloud OSS) and block storage (such as EBS) to automatically migrate data based on data access frequency and importance;

[0090] Introduce a data archiving strategy to migrate data that has been inactive for a long time but still needs to be retained (such as historical transaction records and old versions of market research reports) to a cold storage area to reduce storage costs; use search engine technology (such as Elasticsearch) to index historical data.

[0091] The working principle of the above technical solution is as follows: first, the uploaded multi-source data is compressed using a compression algorithm, and then the compressed data is encapsulated into a first data packet; and the compressed first data packet is redundantly backed up to generate a second data packet. Through redundant backup, even if a data packet is damaged or lost during transmission or storage, it can be restored through another data packet; after the first data packet and the second data packet are sent to the cloud space through a multi-channel transmission protocol, the cloud space stores the second data packet (redundant backup) in the second storage space as a backup after receiving the data packet, and stores the first data packet (original compressed data) in the first storage space for main data processing and analysis; the first data packet in the first storage space is decompressed, and the data source that needs to be processed immediately is captured through a real-time data stream processing layer (such as Apache Kafka, Apache Flink and other stream processing frameworks). The real-time data stream framework is used to capture, process and analyze data from the data stream in real time; in the process of stream processing, erroneous or invalid data is eliminated, and the processed data is analyzed in real time to extract key indicators (KPIs) or perform preliminary data aggregation. The processing results can be used immediately for monitoring, alarming or preliminary business decision support; a layered historical data storage architecture is built in the first storage space, combining the advantages of object storage (such as Amazon S3, Alibaba Cloud OSS) and block storage (such as EBS), and data is automatically migrated according to the access frequency and importance of the data. High-frequency access data is stored in high-performance block storage, while low-frequency access data is migrated to low-cost object storage; and a data archiving strategy is introduced to migrate long-term inactive but still required data (such as historical transaction records, old versions of market research reports) to cold storage areas to further reduce storage costs. At the same time, search engine technology (such as Elasticsearch) is used to index historical data for quick retrieval and query of these data.

[0092] The effects of the above technical solution are as follows: by compressing multi-source data through compression algorithms, the data volume is reduced, thereby reducing the storage space occupied and the transmission bandwidth requirements, and further reducing the storage and transmission costs; the compressed data packets are redundantly backed up to generate second data packets, and are sent to the cloud space through multi-channel transmission protocols, which increases the redundancy of data transmission, reduces the risk of data loss or damage, and improves the reliability of data transmission; through the real-time data stream processing layer, the stream processing framework (such as Apache Kafka, Apache Flink captures and processes data sources that need to be processed immediately, realizes instant analysis and response of data, supports rapid business decision-making and real-time monitoring; in the process of real-time data processing, eliminating erroneous or invalid data can improve the quality and accuracy of data, and provide a reliable data foundation for subsequent data analysis and application; real-time analysis of processed data, extraction of key indicators (KPIs) or preliminary data aggregation helps to quickly understand business conditions and provide intuitive decision support for management; by building a layered historical data storage architecture, data is automatically migrated according to the access frequency and importance of data, and high-frequency access data is stored in high-performance block storage, while low-frequency access data is migrated to low-cost object storage or cold storage areas, which can optimize storage costs; data archiving strategies are introduced to migrate data that has been inactive for a long time but still needs to be retained to cold storage areas, and search engine technology (such as Elasticsearch) is used to index historical data, which improves the retrieval efficiency and availability of historical data; through data archiving strategies, the secure storage of data that has been inactive for a long time but still needs to be retained (such as historical transaction records and old versions of market research reports) is ensured, meeting the compliance and business continuity requirements of enterprises.

[0093] In one embodiment of the present invention, the S2 includes:

[0094] S21. Through a multi-dimensional data source quality assessment framework, the evaluation framework includes multiple indicators such as data integrity, accuracy, timeliness, reliability, and source authority. Based on preset thresholds and weights, each data source is quantitatively scored to identify the main data source;

[0095] S22. For the primary data source, automatically adjust the extraction time interval based on the frequency of data update, and use regular expressions and machine learning models to identify and extract key information fields;

[0096] S23. During the extraction process, based on the data quality secondary verification mechanism, the logical relationship and consistency between historical data and verification data (such as the cross-check relationship between financial data) are compared;

[0097] S24. When associating related data sources, common identifiers (such as supplier ID, product number) are used in combination with multi-dimensional association strategies, including sequence matching based on timestamps, proximity analysis based on geographic locations, and content association based on text similarity; association rules are dynamically adjusted according to data characteristics and business needs;

[0098] S25. After association, consistency check is performed, including cross-data source cross-validation (such as comparing sales in financial statements with sales in market research reports), rationality check based on business logic (such as consistency check of on-time delivery rate and logistics tracking data), and anomaly detection based on statistical models (such as using clustering algorithms to identify abnormal transaction records). A feedback mechanism is established to feed back the results of the consistency check to the data collection and processing links. At the same time, the association rules and verification algorithms are continuously optimized based on the feedback results to form a closed-loop iterative optimization process.

[0099] The working principle of the above technical solution is as follows: by using a data source quality assessment framework that includes multiple indicators such as data integrity, accuracy, timeliness, reliability and source authority, a comprehensive assessment is conducted on each data source; based on preset thresholds and weights, each data source is quantitatively scored; based on the scoring results, the primary data source with the highest quality and most suitable as the basis for subsequent processing is identified; according to the frequency of updating the primary data source, the time interval for data extraction is automatically adjusted; using regular expressions and machine learning models to identify and extract key information fields from the primary data source, the key information fields include supplier name, product specifications and transaction amount; in the extraction process, by comparing historical data and verifying the logical relationship and consistency between data (such as the cross-reference relationship between financial data), the accuracy and reliability of the data are further verified; any data quality problems found are recorded and fed back to the data collection and processing link in a timely manner; through common identification (such as supplier ID, product number) combined with multi-dimensional association strategies (such as timestamp sequence matching, geographic location proximity analysis, text similarity content association), effective association is performed between different data sources; according to data characteristics and business needs, the association rules are dynamically adjusted to deal with anomalies or special situations in the data. For example, additional verification fields are introduced for auxiliary association, or sentiment analysis model parameters are adjusted to filter noise; data consistency between different data sources is ensured through cross-data source data cross-validation (such as sales comparison between financial statements and market research reports); rationality verification based on business logic (such as consistency between on-time delivery rate and logistics tracking data) and anomaly detection based on statistical models (such as clustering algorithms to identify abnormal transaction records) are used to further improve data quality; a feedback mechanism is established to feed back the results of consistency verification to the data collection and processing links, and association rules and verification algorithms are continuously optimized based on the feedback results. A closed-loop iterative optimization process is formed to continuously improve the accuracy and efficiency of data processing.

[0100] The effects of the above technical solutions are as follows: through a multi-dimensional data source quality assessment framework, comprehensive consideration is given to indicators such as data integrity, accuracy, timeliness, reliability and source authority, and the quality of the data source can be comprehensively assessed, thereby ensuring that the data foundation for subsequent processing is solid and reliable; based on quantitative scoring with preset thresholds and weights, the primary data source can be objectively and scientifically identified, providing high-quality data input for subsequent data extraction and association; the extraction time interval is automatically adjusted according to the frequency of data updates, avoiding unnecessary frequent extraction and improving data processing efficiency; regular expressions and machine learning models are used to automatically identify and extract key information fields, reducing manual intervention and improving the level of automation; in the extraction process, by comparing historical data and verifying data By combining the logical relationship and consistency between data sources, data errors can be discovered and corrected in a timely manner to ensure data accuracy and consistency; through common identification and multi-dimensional association strategies, effective associations between different data sources are achieved, further strengthening the integrity and consistency of the data; according to data characteristics and business needs, association rules are dynamically adjusted to flexibly respond to anomalies or special situations in the data, such as duplicate supplier IDs, social media noise, etc.; anomaly detection based on statistical models, such as using clustering algorithms to identify abnormal transaction records, can promptly discover and process outliers in the data to ensure data accuracy and reliability; by establishing a feedback mechanism, the results of the consistency check are fed back to the data collection and processing links, and the association rules and verification algorithms are continuously optimized based on the feedback results.

[0101] In one embodiment of the present invention, the S22 includes:

[0102] Monitor the update log or timestamp of the primary data source in real time or periodically, record the time interval of each data update, and analyze the historical update frequency data to obtain analysis results, including the periodicity of data updates (such as daily, weekly or monthly updates) and abnormal fluctuations;

[0103] According to the analysis results, the initial data extraction time interval is set to ensure that the latest data can be captured in time while avoiding resource waste caused by over-frequent extraction. Through the dynamic adjustment mechanism, when the data update frequency changes (such as speeding up or slowing down) is monitored, the extraction time interval is automatically adjusted to match the new update speed;

[0104] For structured data (such as table data in a database), regular expressions are used to locate and extract key information fields;

[0105] For unstructured or semi-structured data (such as text reports and social media comments), identify and extract key information through existing named entity recognition models;

[0106] Combine regular expressions and named entity recognition models to build a hybrid extraction strategy; use regular expressions for information that is easy to match through rules, and rely on named entity recognition models to process complex and changeable information.

[0107] Monitor the extraction process in real time and record the key indicators of each extraction task, including execution time, success rate and error information; through the anomaly detection mechanism, when anomalies occur during the extraction process (such as inconsistent data format, network interruption, etc.), trigger an alarm and take corresponding emergency measures (such as retrying extraction, switching to backup data source, etc.);

[0108] Through the extraction result verification mechanism, the extracted data is preliminarily verified, and the effectiveness of the extraction strategy is evaluated based on the data quality analysis of the extraction results; according to the analysis results, the hybrid strategy is optimized and adjusted.

[0109] The working principle of the above technical solution is as follows: by monitoring the update log or timestamp of the main data source, the time interval of each data update is recorded in real time or periodically; the historical update frequency data is analyzed to identify the periodicity of data updates (such as daily, weekly, monthly, etc.) and possible abnormal fluctuations; based on the analysis results, an initial data extraction time interval is set, such as one hour; the changes in data update frequency are continuously monitored; when changes in data update frequency are detected (such as speeding up or slowing down), the data extraction time interval is automatically adjusted to match the new update speed; for structured data (such as database tables), regular expressions are used to accurately locate and extract key information fields, such as dates, amounts, identifiers, etc.; for unstructured or semi-structured data (such as text reports, social media comments), the existing named entity recognition (NER) model is used to identify and extract key information; combining the advantages of regular expressions and named entity recognition models, a hybrid extraction strategy is constructed. Regular expressions are used for information that is easy to match through rules, while named entity recognition models are relied upon for processing complex and changeable information; key indicators of each extraction task, such as execution time, success rate, and error information, are monitored in real time; when anomalies occur during the extraction process (such as data format inconsistency, network interruption, etc.), an alarm is triggered through the anomaly detection mechanism, and corresponding emergency measures are taken automatically or manually, such as retrying the extraction, switching to an alternative data source, etc.; the extracted data is preliminarily verified through the extraction result verification mechanism to evaluate the basic quality of the data; data quality analysis based on the extraction results is used to evaluate the effectiveness of the extraction strategy; the hybrid extraction strategy is optimized and adjusted according to the analysis results to improve the accuracy and efficiency of data extraction.

[0110] The effects of the above technical solution are: by analyzing the update frequency of the main data source and setting the initial extraction time interval, it is possible to ensure the timely capture of the latest data, while avoiding unnecessary frequent extraction, thereby saving computing resources and network resources. When the data update frequency is monitored to change, the extraction time interval is automatically adjusted, further improving the flexibility and efficiency of data extraction; the hybrid extraction strategy combining regular expressions and named entity recognition models can more accurately extract key information from structured, unstructured or semi-structured data. Regular expressions are suitable for data with clear rules and fixed formats, while named entity recognition models are good at processing complex, changeable and semantically rich data. The combination of the two greatly improves the accuracy of data extraction; real-time monitoring of the extraction process and recording of key indicators such as execution time, success rate and error information helps to discover potential problems in a timely manner; through the anomaly detection mechanism, when an anomaly occurs during the extraction process, an alarm can be quickly triggered, and corresponding emergency measures can be taken automatically or manually, such as retrying the extraction, switching to an alternate data source, etc., to ensure the continuity and stability of data extraction and reduce data loss or delays caused by anomalies; preliminary verification of the extracted data is performed to evaluate the data quality and ensure the accuracy and reliability of the extraction results. Based on the results of data quality analysis, the hybrid extraction strategy is optimized and adjusted to continuously improve the accuracy and efficiency of data extraction; data extraction is performed using machine learning models such as named entity recognition to achieve intelligent processing, reduce human intervention, and improve processing efficiency and accuracy; from data update monitoring to extraction time interval adjustment, to data extraction and verification, the entire process is highly automated, reducing the risk of human error and improving overall data processing efficiency; according to data characteristics and business needs, the hybrid extraction strategy can be flexibly adjusted and optimized to adapt to different data sources and data formats, improving the scalability and adaptability of the solution; when the data source or data format changes, the dynamic adjustment mechanism and optimization of the hybrid extraction strategy can quickly adapt to these changes to ensure the stability and accuracy of data extraction.

[0111] In one embodiment of the present invention, S3 includes:

[0112] S31. Identify and handle heterogeneity issues among multi-source data; convert data from different sources into a unified format through methods such as data conversion, standardization and normalization;

[0113] S32. Based on the multi-dimensional indicators of data, the multi-dimensional indicators include importance, reliability, timeliness and source authority, and the data weight distribution mechanism is designed by using the analytic hierarchy process and the fuzzy comprehensive evaluation method in combination with the expert database; and the weight distribution mechanism is dynamically adjusted through the dynamic adjustment strategy of the weight;

[0114] S33. Fuse the data through fusion algorithms (such as weighted average, principal component analysis, neural network, etc.), and perform outlier detection and data cleaning mechanisms during the fusion process to ensure the accuracy and reliability of the fusion results;

[0115] S34. After the integration is completed, the integrated data is verified and evaluated by comparing historical data, industry benchmarks and business logic;

[0116] S35. According to the needs and goals of supplier evaluation, the indicator system is divided into multiple dimensions (such as financial status, production capacity, product quality, delivery on time rate, after-sales service, etc.); under each dimension, the evaluation indicators are further refined to form a multi-level indicator system structure; a dynamic adjustment mechanism for indicators is established to dynamically adjust the indicator system according to influencing factors, including market changes, business needs and data feedback. This includes adding or deleting evaluation indicators, adjusting indicator weights, and optimizing indicator calculation methods, etc.;

[0117] The working principle of the above technical solution is as follows: first, the system identifies the inconsistencies in data types, formats, units and semantics of data from different sources; then, data conversion, standardization and normalization are used to convert data from different sources into a unified format and standard; based on multi-dimensional indicators such as data importance, reliability, timeliness and source authority, the data weight allocation mechanism is designed using the hierarchical analysis method and fuzzy comprehensive evaluation method, combined with the opinions of the expert database; according to the actual performance of the data and changes in market and business needs, the weight allocation mechanism is adjusted in a timely manner through dynamic adjustment strategies; fusion algorithms such as weighted average, principal component analysis, and neural networks are used to integrate data to fully utilize the advantages of each data source. In the process of integration, implement outlier detection and data cleaning mechanisms to eliminate erroneous or unreasonable data; after integration, conduct comprehensive verification and evaluation of the integrated data by comparing historical data, industry benchmarks and business logic, and divide the indicator system into multiple dimensions (such as financial status, production capacity, product quality, delivery on time rate, after-sales service, etc.) according to the needs and goals of supplier evaluation; in each dimension, further refine the evaluation indicators to form a multi-level indicator system structure, for example, product quality is divided into three levels: high, medium and low; establish an indicator dynamic adjustment mechanism, and flexibly adjust the indicator system according to market changes, business needs, data feedback and other influencing factors. This includes adding or deleting evaluation indicators, adjusting indicator weights, and optimizing indicator calculation methods.

[0118] The effects of the above technical solutions are: by identifying and processing the heterogeneity problems between multi-source data, the inconsistency in data types, formats, units and semantics is solved, so that data from different sources can be converted into a unified format, providing a reliable basis for subsequent data analysis and decision-making. By improving the integration capability, information islands can be broken and comprehensive data interconnection and sharing can be achieved; the weight distribution mechanism designed based on multi-dimensional indicators (such as importance, reliability, timeliness and source authority) can reasonably evaluate the contribution of each data source and ensure that the quality differences of data are fully considered in the data fusion process. At the same time, through the dynamic adjustment strategy of weights, the weight distribution can be adjusted in time according to the actual performance of the data and changes in market and business needs to maintain the timeliness and accuracy of the evaluation system; the data is integrated by using fusion algorithms such as weighted average, principal component analysis, and neural networks. At the same time, outlier detection and data cleaning mechanisms are implemented during the fusion process to effectively eliminate erroneous or unreasonable data and ensure the accuracy and reliability of the fusion results. The high-quality data fusion results provide a solid foundation for subsequent decision-making analysis; after the fusion is completed, the fused data is fully verified and evaluated by comparing historical data, industry benchmarks and business logic to ensure the accuracy and applicability of the data. According to the needs and goals of supplier evaluation, the indicator system is divided into multiple dimensions, and the evaluation indicators are further refined under each dimension to form a multi-level indicator system structure. The multi-dimensional indicator system can fully reflect the comprehensive strength of suppliers and provide a rich perspective and basis for evaluation; establish an indicator dynamic adjustment mechanism to flexibly adjust the indicator system according to market changes, business needs, data feedback and other influencing factors. Through the ability of dynamic adjustment, the indicator system can always keep pace with the actual situation, ensuring the timeliness and accuracy of the evaluation.

[0119] In one embodiment of the present invention, the S33 includes:

[0120] According to data characteristics (such as data volume, data distribution, degree of heterogeneity, etc.) and business needs, select fusion algorithms; use statistical methods to re-detect outliers in the data, remove or correct them; and clean the multi-source data after re-detection, including removing duplicate data, correcting erroneous data, processing missing values, etc.

[0121] Formulate fusion strategies based on the selected fusion algorithm and data characteristics, including data alignment, fusion order, weight distribution, etc.; perform data fusion operations according to the fusion strategy and monitor the fusion process in real time;

[0122] The quality of the fusion results is preliminarily evaluated by calculating the statistical indicators of the fusion results (such as mean, variance, correlation coefficient, etc.) and visual analysis (such as scatter plots, heat maps, etc.);

[0123] Based on the preliminary evaluation results, feedback and adjustments are made to the fusion strategy, and the fusion algorithm and strategy are continuously optimized through multiple iterations.

[0124] The working principle of the above technical solution is as follows: First, according to the data characteristics (such as data volume, data distribution, degree of heterogeneity, etc.) and business needs, select the appropriate fusion algorithm. For example, suppose you are processing a financial transaction data set containing millions of records, and you need to analyze this data in real time to detect fraud. Due to the huge amount of data and limited computing resources, you may choose distributed algorithms that can handle large-scale data sets, such as machine learning algorithms supported by Apache Spark. These algorithms can process data in parallel on multiple machines, thereby speeding up processing. This step is the premise of data fusion, because different data and business needs may require different fusion algorithms to achieve the best results; use statistical methods to re-detect outliers in the data to ensure that these values ​​that may affect the accuracy of the results are excluded or corrected before data fusion; conduct a comprehensive cleanup of the multi-source data after re-detection, including removing duplicate data, correcting erroneous data, processing missing values, etc., to improve the quality and consistency of the data; formulate a detailed fusion strategy based on the selected fusion algorithm and data characteristics, including data alignment (ensuring that related data in different data sources can correspond), fusion order (determine the order of data fusion) and weight allocation (allocate weights based on factors such as data importance and reliability); perform data fusion operations according to the formulated fusion strategy to integrate data from multiple data sources into a unified data set; and conduct real-time monitoring during the fusion process; preliminarily evaluate the quality of the fusion results by calculating the statistical indicators of the fusion results (such as mean, variance, correlation coefficient, etc.); use visualization tools (such as scatter plots, heat maps, etc.) to intuitively display the fusion results to help users better understand the relationship and distribution characteristics between data; and provide feedback and adjustments to the fusion strategy based on the preliminary evaluation results. If the fusion result does not meet expectations, it is necessary to analyze the reasons and adjust the fusion algorithm or strategy; through multiple iterations, continuously optimize the fusion algorithm and strategy until a satisfactory fusion result is obtained.

[0125] The effects of the above technical solution are: selecting appropriate fusion algorithms according to data characteristics and business requirements can ensure that the data fusion process is more efficient and the results are more accurate. Different algorithms process different data characteristics and business requirements to ensure the best data integration effect; by re-detecting and processing outliers in the data, and cleaning duplicate, erroneous and missing data, the quality and consistency of the data can be significantly improved. It can reduce noise and errors in the fusion process and improve the accuracy of the fusion results; formulate fusion strategies according to the selected fusion algorithm and data characteristics, including data alignment, fusion order and weight distribution, and can flexibly cope with the complexity and heterogeneity of different data sources. This flexibility enables the technical solution to adapt to a variety of different application scenarios and data environments; real-time monitoring of the fusion process, and feedback and adjustment of the fusion strategy based on the preliminary evaluation results can ensure the controllability and optimizability of the fusion process. By optimizing the fusion algorithm and strategy for multiple iterations, the quality and accuracy of the fusion results can be continuously improved; through the refined data fusion process, high-quality, consistent and accurate fusion results can be generated. Based on the fusion results, it can provide strong support for corporate decision-making, helping decision makers to more accurately grasp market trends, evaluate supplier strength, optimize production processes, etc.; by calculating statistical indicators and performing visual analysis, it can intuitively display the characteristics and trends of the fusion results. Visual analysis helps decision makers better understand the information and rules behind the data, so as to make more informed and accurate decisions; this technical solution includes a detection and processing mechanism for outliers, which can effectively identify and exclude outliers in the data, and reduce system errors and instability caused by data errors or noise. It helps to enhance the robustness and reliability of the system; through multiple iterations of optimizing fusion algorithms and strategies, the performance and stability of the system can be continuously improved. Through the iterative optimization mechanism, the system can continuously adapt to changes in the data environment and changes in business needs, and maintain an efficient and accurate operating state.

[0126] In one embodiment of the present invention, the S4 includes:

[0127] S41. Build advanced models by combining multiple ensemble learning methods (such as Stacking, Boosting, etc.) with multiple machine learning algorithms, and perform parameter tuning and model fusion; use deep learning models (such as LSTM, CNN, etc.) to process complex data (such as time series data, image data, etc.), extract deep features and perform predictive analysis;

[0128] S42. Evaluate the model performance through cross-validation and ROC curve analysis, and select the optimal model for deployment based on the evaluation results;

[0129] S43. According to the evaluation requirements, select evaluation methods (such as analytic hierarchy process, fuzzy comprehensive evaluation method, TOPSIS method, etc.), and form a composite evaluation algorithm through weighted fusion and nonlinear combination. Through the dynamic weight adjustment mechanism, according to the changes in the evaluation objectives and the real-time update of the data, the weights of each evaluation method are automatically adjusted;

[0130] S44. Build a risk assessment model through machine learning to quantitatively assess the possible risks of suppliers; preset warning thresholds based on historical data and business rules. When the risk assessment result exceeds the preset threshold, the warning is automatically triggered and responded through the warning response mechanism, including warning notification, problem tracking, and solution recommendation. The risk assessment result is obtained through the following formula:

[0131]

[0132] Among them, R is the risk assessment result, is the weight of the ith risk factor, is the specific score of the ith risk factor, n is the total number of risk factors, is a function that converts Converted into a numerical value that can be used for calculation, T is the warning threshold, and if R>T, a warning is triggered.

[0133] The working principle of the above technical solution is: using a variety of integrated learning methods (such as Stacking, Boosting) combined with a variety of machine learning algorithms (such as decision trees, random forests, SVM, etc.) to build an advanced model with higher prediction performance and generalization ability. The model is used to combine the prediction results of multiple base models; for each model, parameters are tuned by grid search, random search or Bayesian optimization to find the best combination of model parameters so that the model performs best on the validation set; for complex data (such as time series data, image data, etc.), deep learning models (such as LSTM, CNN, etc.) are used for processing. The advanced model automatically extracts deep feature representations from the original data for predictive analysis. By training a deep learning model, nonlinear relationships and complex patterns in the data can be captured; the prediction results of multiple advanced models and deep learning models are fused by using weighted average, voting or more complex integration strategies, and the performance of the advanced model is evaluated by cross-validation methods (such as K-fold cross validation). By dividing the data set into a training set and a validation set (or test set), the generalization ability of the advanced model is evaluated in multiple training-validation rounds; the classification performance of the advanced model is analyzed using the ROC curve. The ROC curve evaluates the classification ability and robustness of the advanced model by plotting the relationship between the true positive rate (TPR) and the false positive rate (FPR) at different thresholds; based on the results of cross-validation and ROC curve analysis, the advanced model with the best performance is selected for deployment; according to the evaluation requirements, the appropriate evaluation method (such as hierarchical analysis method, fuzzy comprehensive evaluation method, TOPSIS method, etc.) is selected. The results of multiple evaluation methods are integrated into a composite evaluation algorithm by weighted fusion and nonlinear combination. According to the changes in the evaluation objectives and the real-time update of the data, the weights of each evaluation method are automatically adjusted through the dynamic weight adjustment mechanism. A risk assessment model is constructed through a machine learning algorithm to quantitatively evaluate the possible risks of suppliers. Based on historical data and business rules, potential risk factors are automatically identified and predicted; based on historical data and business rules, a reasonable warning threshold is preset, such as starting a warning when the risk assessment results reach 60%. When the risk assessment results exceed the preset threshold, the early warning mechanism is automatically triggered; once the early warning is triggered, the system will promptly notify relevant personnel through early warning notifications (such as emails, SMS, message push, etc.). At the same time, the system will start the problem tracking and solution recommendation mechanism to help relevant personnel quickly locate the problem and take corresponding measures.

[0134] The effects of the above technical solutions are as follows: by combining multiple integrated learning methods and machine learning algorithms, more advanced and complex models can be constructed. By combining the advantages of multiple base models, the bias and variance of a single model can be reduced, thereby improving the accuracy and generalization of predictions. For complex data (such as time series, images, etc.), deep learning models (such as LSTM, CNN) can automatically extract deep feature representations and capture nonlinear relationships and complex patterns in the data. The laws behind the data can be understood more accurately, and the accuracy of predictive analysis can be improved. Through cross-validation and ROC curve analysis, the performance of the model can be comprehensively and objectively evaluated. It helps to avoid overfitting and underfitting problems and ensure that the selected model performs well on unknown data. Based on scientific evaluation results, the optimal model can be selected for deployment. The stability and reliability of the system in practical applications can be ensured. By weighted fusion and nonlinear combination of multiple evaluation methods, a flexible and comprehensive composite evaluation algorithm can be constructed. By fully considering the advantages and applicable scenarios of different evaluation methods, the comprehensiveness and accuracy of the evaluation can be improved. Based on the dynamic weight adjustment mechanism, the evaluation algorithm can automatically adjust the weights of each evaluation method according to the changes in the evaluation target and the real-time update of the data. It can ensure that the evaluation algorithm is always consistent with the actual situation, improve the timeliness and accuracy of the evaluation; the risk assessment model built through machine learning can quantitatively evaluate the possible risks of suppliers. It can help enterprises discover potential risk factors in a timely manner and provide support for decision-making; when the risk assessment results exceed the preset threshold, the early warning mechanism is automatically triggered, and responds through early warning notifications, problem tracking and solution recommendations. It helps enterprises to respond to risk events quickly and reduce losses; by providing advanced model prediction, scientific model evaluation, comprehensive evaluation system, and effective risk assessment and early warning mechanism functions, it provides comprehensive decision-making support for enterprises. This helps enterprises make decisions more efficiently and accurately and improve their market competitiveness. The above risk assessment formula allows different risk factors to be assigned different weights according to specific circumstances, which can achieve flexibility and customizability of the evaluation. The importance of each risk factor may be different in different industries and different business scenarios. By adjusting the weights, the actual situation can be more accurately reflected. The above formula ensures the comprehensiveness of the evaluation by accumulating the weighted scores of all risk factors. Each risk factor is taken into consideration, avoiding the one-sidedness of a single factor determining the overall risk, making the evaluation results more comprehensive and reliable. In the formula The function is used to convert the specific score of the risk factor The formula converts the data into numerical values ​​that can be used for calculation, thus realizing the quantitative assessment of risk. This helps to transform subjective judgment into objective data, and improves the accuracy and comparability of the assessment. The formula ensures the effectiveness of the early warning mechanism by presetting the early warning threshold (T) and automatically triggering the early warning when the risk assessment result (R) exceeds the threshold. This automated early warning response can timely detect potential risks, provide timely response measures for enterprises or organizations, and reduce the probability of risk occurrence and losses. The weights in the formula can be automatically adjusted according to the changes in the evaluation objectives and the real-time update of data through the dynamic weight adjustment mechanism. This dynamic adjustment capability enables the risk assessment model to adapt to different business environments and demand changes, and maintain the accuracy and timeliness of the assessment results. The risk assessment results (R) provide decision makers with quantitative indicators of the supplier's risk level, which helps decision makers to formulate procurement strategies, risk management plans, etc. more scientifically. At the same time, the functions of early warning notification, problem tracking and solution recommendation provided by the early warning response mechanism further support the formulation and implementation of decisions.

[0135] In one embodiment of the present invention, S5 includes:

[0136] S51. Build a trend prediction system based on time series analysis or machine learning models to model suppliers’ historical performance data and predict future development trends;

[0137] S52. Compare and analyze supplier evaluation results with those of other competitors or industry benchmarks to identify their own strengths and weaknesses and provide a basis for formulating competitive strategies; generate personalized improvement suggestions based on the supplier's specific performance; for example, in response to a supplier's shortcomings in product quality, make specific quality control process optimization suggestions; in response to the problem of slow supply chain response, recommend the introduction of advanced supply chain management systems.

[0138] S53. Establish an improvement effect tracking mechanism to regularly collect supplier performance data after adopting improvement suggestions, evaluate the improvement effect, and adjust the suggestions based on feedback;

[0139] S54. Establish a multi-channel user feedback collection mechanism, including online surveys, user interviews, customer service hotlines, etc., to ensure that user opinions can be collected comprehensively and timely, and through an iterative optimization process, organize and analyze the collected user feedback, and formulate corresponding optimization plans; based on the optimization strategy, continuously iterate and optimize system functions and performance.

[0140] The working principle of the above technical solution is as follows: first, collect the supplier's historical performance data, which includes multiple dimensions such as order volume, delivery on time rate, product quality qualification rate, and after-sales service evaluation. Then, clean, denoise and standardize the data; build a trend prediction system based on time series analysis or machine learning algorithms (such as ARIMA, LSTM, XGBoost, etc.). By analyzing historical data, learn the patterns and trends in the data, and thus establish a model that can predict future development trends; use the constructed model to predict the future development trend of the supplier. Among them, the prediction results may include order volume forecasts, delivery on time rate changes, product quality trends, etc. in the future period. For example, the prediction result is that the order volume will increase significantly in the next month. Compare and analyze the evaluation results of the supplier with competitors or industry benchmarks to identify in which aspects the supplier has advantages and in which aspects it has shortcomings. Comparative analysis can help companies understand their position in the market and provide a basis for formulating competitive strategies; generate personalized improvement suggestions based on the specific performance of the supplier. The improvement suggestions include product quality, delivery on time rate, after-sales service and other aspects. For example, for a supplier's shortcomings in product quality, specific suggestions for optimizing the quality control process can be put forward; for the problem of slow supply chain response, it can be recommended to introduce advanced supply chain management systems, etc.; regularly collect performance data of suppliers after adopting improvement suggestions. These data are used to evaluate the implementation effect of the improvement suggestions; by comparing the data before and after the improvement, evaluate whether the improvement effect is significant. If the effect is significant, it means that the improvement suggestion is effective; if the effect is not significant or even worse, it is necessary to re-examine the feasibility and implementation of the improvement suggestion; feedback and adjustment of the improvement suggestion are made according to the evaluation results. If the suggestion is effective, continue to promote it; if the suggestion is ineffective or needs to be improved, formulate new suggestions or adjust the implementation strategy; establish a multi-channel user feedback collection mechanism, including online surveys, user interviews, customer service hotlines, etc.; organize and analyze the collected user feedback. Through data analysis technology (such as text mining, sentiment analysis, etc.), extract key information and opinions from user feedback; formulate corresponding optimization plans based on the analysis results of user feedback. The optimization plan includes multiple aspects such as improvement of system functions, improvement of performance, optimization of user experience, etc.; according to the optimization plan, continuously iterate and optimize system functions and performance. Through continuous user feedback collection and iterative optimization processes, we ensure that the system always meets user needs and maintains a good operating state.

[0141] The effect of the above technical solution is: by building a trend prediction system based on time series analysis or machine learning models, enterprises can accurately predict the future development trends of suppliers. It can help enterprises understand market changes in advance, formulate forward-looking procurement plans and supply chain strategies, so as to avoid potential risks and losses; the prediction results provide strong decision-making support for enterprises. Enterprises can adjust key links such as supplier selection, inventory management, and production plans according to the prediction results to cope with market fluctuations and changes; comparing and analyzing supplier evaluation results with other competitors or industry benchmarks can help enterprises identify their own strengths and weaknesses. Comparative analysis can not only help enterprises understand their position in the supply chain, but also provide a strong basis for enterprises to formulate differentiated competitive strategies; generating personalized improvement suggestions based on the specific performance of suppliers can help enterprises solve problems in the supply chain in a targeted manner. Improvement suggestions can be directly applied to the management and improvement process of suppliers to improve the overall performance level of suppliers; by establishing an improvement effect tracking mechanism, regularly collect performance data of suppliers after adopting improvement suggestions, and evaluate the improvement effect. The tracking mechanism ensures the effectiveness and sustainability of improvement measures, which helps enterprises to continuously optimize supply chain management processes; adjusting the content of suggestions based on evaluation results and feedback can ensure that improvement measures always meet the actual needs of enterprises and market changes. By establishing a multi-channel user feedback collection mechanism, enterprises can ensure that they can collect user opinions in a comprehensive and timely manner. This helps enterprises understand user needs and market trends, and provides strong support for product development and system optimization; the collected user feedback is sorted, analyzed, and optimized through iterative optimization processes. Based on the optimization strategy, the system functions and performance are continuously optimized iteratively to ensure that the system always meets user needs and maintains a leading position; through the comprehensive effects of multiple links such as accurate trend prediction, competitive advantage identification, continuous improvement, and user feedback response, the overall efficiency and competitiveness of the supply chain are significantly improved.

[0142] One embodiment of the present invention, as Figure 2 As shown, a system for implementing a multi-source data processing method for supplier evaluation includes:

[0143] Data collection module: collects multi-source data from different sources and pre-processes the collected multi-source data;

[0144] Data association module: Determine the main data source and related data sources according to the supplier evaluation requirements. The main data source is the core data in the evaluation process, such as financial statements; the related data source is auxiliary data, such as social media evaluations. Based on the preset data screening rules, extract valid data from the main data source and associate the extracted valid data with the data in the related data source. During the association process, match the common identifiers in the data (such as supplier ID, product number, etc.) to ensure the consistency and accuracy of the data.

[0145] Data fusion module: fuses valid data and valid related data to form a comprehensive data set, and builds an indicator system for supplier evaluation based on the fused multi-source data;

[0146] Comprehensive evaluation module: For each evaluation dimension, a data processing model is constructed to conduct in-depth analysis of multi-source data, extract key evaluation indicators and features, and use comprehensive evaluation algorithms (such as hierarchical analysis method, fuzzy comprehensive evaluation method, etc.) to conduct comprehensive evaluation of suppliers based on the evaluation indicators and features of each dimension;

[0147] Result feedback module: Based on the results of the comprehensive evaluation algorithm, the comprehensive evaluation report of the supplier is output, which includes the supplier's score, ranking, advantages and disadvantages, etc. The evaluation results are fed back to the supplier and related personnel to help the supplier understand its own problems and deficiencies and formulate corresponding improvement measures. At the same time, based on the evaluation results and feedback, the data processing model and evaluation algorithm are continuously optimized and improved to improve the accuracy and effectiveness of the evaluation.

[0148] The working principle of the above technical solution is: collect multi-source data related to supplier evaluation from different channels (such as internal enterprise systems, public databases, social media, third-party evaluation platforms, etc.). The multi-source data includes various types such as financial statements, transaction records, customer evaluations, and social media feedback; pre-process the collected multi-source data; and determine the main data source (such as financial statements) and related data sources (such as social media evaluations) according to the evaluation requirements. Among them, the main data source is the core of the evaluation, and the related data source is used to provide auxiliary information; based on the preset data screening rules, extract valid data from the main data source. The data screening rules involve the time range, integrity, and rationality of the data, such as screening the latest data obtained within one week; use the common identifiers in the data (such as supplier ID, product number, etc.) to match and associate the extracted valid data with the data in the related data source. ; Merge the valid data and the valid related data to form a comprehensive data set. Based on the fused multi-source data, construct an indicator system for supplier evaluation. The index system covers multiple aspects of suppliers (such as product quality, delivery capability, service attitude, price competitiveness, etc.), and determines specific evaluation indicators for each aspect, such as product quality must meet national standards or service attitude must be five stars; for each evaluation dimension, a data processing model is constructed to conduct in-depth analysis of multi-source data and extract key evaluation indicators and features. Based on the evaluation indicators and features of each dimension, a comprehensive evaluation algorithm (such as hierarchical analysis method, fuzzy comprehensive evaluation method, etc.) is used to comprehensively evaluate suppliers, comprehensively consider multiple factors, and give a comprehensive score and ranking; according to the results of the comprehensive evaluation algorithm, a comprehensive evaluation report of the supplier is output. The comprehensive evaluation report includes information on the supplier's score, ranking, advantages and disadvantages; the evaluation results are fed back to suppliers and relevant personnel to help them understand their own problems and deficiencies and formulate corresponding improvement measures. At the same time, the data processing model and evaluation algorithm are continuously optimized and improved based on the evaluation results and feedback.

[0149] The effects of the above technical solution are as follows: by collecting and integrating multi-source data from different sources (such as financial statements, social media reviews, customer feedback, etc.), it can more comprehensively reflect the true situation of suppliers and avoid the one-sidedness or limitations brought by a single data source; by building an evaluation index system based on the integrated multi-source data, it can cover multiple aspects of suppliers (such as product quality, price, delivery capability, service attitude, innovation capability, etc.), thereby achieving multi-dimensional and comprehensive evaluation; by preprocessing and screening the collected multi-source data, it can effectively remove noise data and outliers, and improve the accuracy and reliability of the data; in the process of data association, using common identifiers (such as supplier ID, product number, etc.) for precise matching can ensure the consistency and accuracy of the data and reduce the evaluation errors caused by inconsistent data; the output comprehensive evaluation report includes the supplier's score, Rankings, strengths and weaknesses and other information can provide decision makers with a clear and intuitive reference, helping them make more scientific and reasonable decisions; feeding back the evaluation results to suppliers and relevant personnel can help suppliers understand their own problems and shortcomings, so as to formulate targeted improvement measures and improve overall performance; based on the evaluation results and feedback, continuously optimizing and improving data processing models and evaluation algorithms can improve the accuracy and effectiveness of the evaluation and ensure the continuous optimization and improvement of the evaluation system; the process of multi-source data processing and comprehensive evaluation increases the transparency of supplier evaluation, helps to establish a more fair and just evaluation system, and promotes healthy competition among suppliers; through the feedback mechanism, suppliers can promptly understand their own shortcomings and take measures to improve them, which not only helps to enhance the competitiveness of the suppliers themselves, but also helps enterprises to establish long-term and stable cooperative relationships with them and achieve win-win results.

[0150] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims

1. A multi-source data processing method for supplier evaluation, characterized in that: The method comprises: S1. Collect multi-source data from different sources and pre-process the collected multi-source data; S2. Determine the main data source and related data sources according to the supplier evaluation requirements, extract valid data from the main data source based on the preset data screening rules, and associate the extracted valid data with the data in the related data sources; S3. Integrate valid data and valid related data to form a comprehensive data set, and build an indicator system for supplier evaluation based on the integrated multi-source data; S4. For each evaluation dimension, a data processing model is constructed to conduct in-depth analysis of multi-source data, extract key evaluation indicators and features, and use a comprehensive evaluation algorithm to conduct a comprehensive evaluation of suppliers based on the evaluation indicators and features of each dimension; S5. Output the comprehensive evaluation report of the supplier based on the results of the comprehensive evaluation algorithm, and feed back the evaluation results to the supplier and relevant personnel; The S2 comprises: S21. Through a multi-dimensional data source quality assessment framework, quantitatively score each data source based on preset thresholds and weights to identify the primary data source; S22. For the primary data source, automatically adjust the extraction time interval based on the frequency of data update, and use regular expressions and machine learning models to identify and extract key information fields; S23. During the extraction process, the logical relationship and consistency between historical data and verification data are compared based on the data quality secondary verification mechanism; S24. When associating related data sources, dynamically adjust association rules based on data characteristics and business needs through common identification and combined with multi-dimensional association strategies; S25. After association, a consistency check is performed; and a feedback mechanism is established to feed back the consistency check results to the data collection and processing links. At the same time, the association rules and the verification algorithm are continuously optimized according to the feedback results to form a closed-loop iterative optimization process; The S22 comprises: Monitor the update log or timestamp of the main data source in real time or regularly, record the time interval of each data update, and analyze the historical update frequency data to obtain analysis results; According to the analysis results, the initial data extraction time interval is set. Through the dynamic adjustment mechanism, when the data update frequency changes, the extraction time interval is automatically adjusted to match the new update speed; For structured data, regular expressions are used to locate and extract key information fields; For unstructured or semi-structured data, identify and extract key information through existing named entity recognition models; Combine regular expressions and named entity recognition models to build a hybrid extraction strategy; Monitor the extraction process in real time, record the key indicators of each extraction task, and through the anomaly detection mechanism, trigger an alarm and take corresponding emergency measures when an anomaly occurs during the extraction process; Through the extraction result verification mechanism, the extracted data is preliminarily verified, and the effectiveness of the extraction strategy is evaluated based on the data quality analysis of the extraction results; based on the analysis results, the hybrid strategy is optimized and adjusted; S31. Identify and process heterogeneity issues among multi-source data; and convert data from different sources into a unified format; S32. Based on the multi-dimensional indicators of data, the data weight distribution mechanism is designed through the hierarchical analysis method and the fuzzy comprehensive evaluation method combined with the expert database; and the weight distribution mechanism is dynamically adjusted through the dynamic adjustment strategy of the weight; S33, fusing the data through a fusion algorithm, and performing outlier detection and data cleaning mechanisms during the fusion process; S34. After the integration is completed, the integrated data is verified and evaluated by comparing historical data, industry benchmarks and business logic; S35. Divide the indicator system into multiple dimensions according to the needs and goals of supplier evaluation; further refine the evaluation indicators under each dimension to form a multi-level indicator system structure; establish an indicator dynamic adjustment mechanism to dynamically adjust the indicator system according to influencing factors; The S4 comprises: S41. Build advanced models by combining multiple ensemble learning methods with multiple machine learning algorithms, and perform parameter tuning and model fusion; use deep learning models to process complex data, extract deep features and perform predictive analysis; S42. Evaluate the model performance through cross-validation and ROC curve analysis, and select the model for deployment based on the evaluation results; S43. According to the evaluation requirements, an evaluation method is selected, and a composite evaluation algorithm is formed through weighted fusion and nonlinear combination. The weight of each evaluation method is automatically adjusted according to the changes in the evaluation objectives and the real-time update of the data through a dynamic weight adjustment mechanism; S44. Build a risk assessment model through machine learning to quantitatively assess the risks faced by suppliers; by presetting early warning thresholds, when the risk assessment results exceed the preset thresholds, an early warning is automatically triggered and responded to through the early warning response mechanism.

2. According to claim 1, a multi-source data processing method for supplier evaluation is characterized in that: Said S1 comprises: S11. Collect multi-source data from different sources through collection scripts or API interfaces; S12, sending the collected multi-source data to a cloud space, where the cloud space stores the received multi-source data through a hierarchical storage structure; S13. Preprocess the stored multi-source data, encrypt the sensitive data, and desensitize the non-sensitive but privacy-related data.

3. A multi-source data processing method for supplier evaluation according to claim 2, characterized in that: The S12 comprises: The uploaded multi-source data is compressed by a compression algorithm to obtain a compressed first data packet, and the compressed data packet is redundantly backed up to obtain a second data packet, and the first data packet and the second data packet are respectively sent to the cloud space through a multi-channel transmission protocol; After receiving the first data packet and the second data packet, the cloud space stores the second data packet in the second storage space; and stores the first data packet in the first storage space; Decompressing the first data packet in the first storage space, and capturing the data source in the first storage space that needs to be processed immediately by using a stream processing framework through a real-time data stream processing layer; And eliminate erroneous or invalid data, analyze the processed data in real time, extract key indicators or perform preliminary data aggregation; Build a layered historical data storage architecture in the first storage space, using a combination of object storage and block storage to automatically migrate data based on data access frequency and importance; Introduce a data archiving strategy to migrate data that has been inactive for a long time but still needs to be retained to a cold storage area, and use search engine technology to index historical data.

4. According to claim 1, a multi-source data processing method for supplier evaluation is characterized in that: The S33 comprises: Select fusion algorithms based on data characteristics and business needs; use statistical methods to re-detect outliers in the data, remove or correct them; and clean the multi-source data after re-detection; Formulate fusion strategies based on the selected fusion algorithm and data characteristics, perform data fusion operations according to the fusion strategies, and monitor the fusion process in real time; By calculating the statistical indicators and visual analysis of the fusion results, the quality of the fusion results is preliminarily evaluated; Based on the preliminary evaluation results, feedback and adjustments are made to the fusion strategy, and the fusion algorithm and strategy are continuously optimized through multiple iterations.

5. The multi-source data processing method for supplier evaluation according to claim 1, characterized in that: The S5 comprises: S51. Build a trend prediction system based on time series analysis or machine learning models to model suppliers’ historical performance data and predict future development trends; S52. Compare and analyze supplier evaluation results with industry benchmarks, and generate personalized improvement suggestions based on the supplier's specific performance; S53. Establish an improvement effect tracking mechanism to regularly collect supplier performance data after adopting improvement suggestions, evaluate the improvement effect, and adjust the suggestions based on feedback; S54. Establish a multi-channel user feedback collection mechanism, and organize and analyze the collected user feedback through an iterative optimization process, and formulate corresponding optimization plans; continuously iterate and optimize system functions and performance based on optimization strategies.

6. A system for implementing the multi-source data processing method for supplier evaluation as claimed in claim 1, characterized in that: The system comprises: Data collection module: collects multi-source data from different sources and pre-processes the collected multi-source data; Data association module: Determine the main data source and related data sources according to the supplier evaluation requirements, extract valid data from the main data source based on the preset data screening rules, and associate the extracted valid data with the data in the related data sources; Data fusion module: fuses valid data and valid related data to form a comprehensive data set, and builds an indicator system for supplier evaluation based on the fused multi-source data; Comprehensive evaluation module: For each evaluation dimension, a data processing model is constructed to conduct in-depth analysis of multi-source data, extract key evaluation indicators and features, and use a comprehensive evaluation algorithm to conduct a comprehensive evaluation of suppliers based on the evaluation indicators and features of each dimension; Result feedback module: Based on the results of the comprehensive evaluation algorithm, output the supplier's comprehensive evaluation report and feedback the evaluation results to the supplier and relevant personnel.

Images