Adaptive acquisition processing method and device for multi-source data, equipment and medium

Abstract

The application discloses a kind of multi-source data's adaptive acquisition processing method, device, equipment and medium, method includes: obtaining data source image, wherein the data source image includes the dynamic characteristic of data source;Based on the data source image and the load state of target database, determine acquisition strategy;According to the acquisition strategy, execute data acquisition task;In response to the abnormal information generated in acquisition process, update the dynamic characteristic in the data source image.Effective effect: effectively avoid the system collapse caused by high concurrent writing when database is high load, and invalid acquisition when network delay is high, realize the load balancing and efficiency optimization of acquisition process, realize the adaptive iteration optimization of acquisition strategy, improve the robustness and automation level of multi-source data access, reduce artificial operation and maintenance cost, guarantee the timeliness and reliability of warehousing data.

Description

Technical Field

[0001] This invention relates to the field of data processing acceleration technology, specifically to an adaptive acquisition and processing method, apparatus, device, and medium for multi-source data. Background Technology

[0002] As enterprise informatization deepens, data sources are becoming increasingly diversified. Data is typically stored across multiple business systems, with data sources including relational databases, non-relational databases, business API interfaces, log files, message queues, and IoT devices. Currently, to achieve unified access to multi-source data, ETL tools or custom scripts are commonly used.

[0003] However, existing technologies have significant shortcomings in practical applications. On the one hand, existing data acquisition scheduling strategies are usually statically configured, such as using fixed acquisition cycles or batch sizes, and cannot be dynamically adjusted according to the real-time status of the data source (such as update frequency fluctuations and network latency) and the load of the target database. When the data source updates frequently or the network conditions are poor, fixed strategies can easily lead to data acquisition delays or interface blocking; while when the target database is overloaded, continuous large-batch writes may cause database crashes. On the other hand, existing acquisition processes lack effective feedback mechanisms. When anomalies occur during acquisition (such as connection timeouts or data format errors), the system often only records or issues alarms, failing to adaptively adjust subsequent acquisition behavior based on the anomaly, leading to repeated occurrences of the same errors and reducing the reliability and efficiency of data access. Therefore, there is an urgent need for a multi-source data acquisition and processing solution that can dynamically adjust the acquisition strategy according to the data source status and target database load, and has adaptive feedback capabilities. Summary of the Invention

[0004] To address the problems of rigid acquisition strategies and lack of adaptive feedback mechanisms in existing technologies, this application proposes an adaptive acquisition and processing method, apparatus, device, and medium for multi-source data. This method constructs a data source profile containing dynamic features, dynamically determines the acquisition strategy based on the target database load status, and uses anomaly information during the acquisition process to correct the profile. This achieves adaptive optimization of the acquisition strategy and improves the efficiency and reliability of multi-source data access.

[0005] To achieve the above objectives, the present invention adopts the following technical solution: An adaptive data acquisition and processing method for multi-source data includes: acquiring a data source profile, wherein the data source profile includes dynamic features of the data source; determining an acquisition strategy based on the data source profile and the load status of a target database; executing a data acquisition task according to the acquisition strategy; and updating the dynamic features in the data source profile in response to abnormal information generated during the acquisition process.

[0006] As one implementation, the dynamic characteristics include at least one of the following: data source update frequency, historical collection error rate, response latency, and data source trust level.

[0007] As one implementation method, determining the collection strategy based on the data source profile and the load status of the target database includes: calculating a collection priority score, which is determined based on a weighted average of positive and negative factors. The positive factors include the data source update frequency and the data source trust level, and the negative factors include response latency, historical collection error rate, and the load status of the target database.

[0008] As one implementation method, the abnormal information includes at least one of the following: data source connection abnormality, field mapping abnormality, data quality abnormality, and data entry failure abnormality.

[0009] As one implementation, updating the dynamic features in the data source profile includes performing at least one of the following operations based on the abnormal information: increasing the historical collection error rate, reducing the data source trust level, and adjusting the collection priority.

[0010] As one implementation, the method further includes: performing field mapping on the collected data, wherein the field mapping is generated based on at least one of field name similarity, field type similarity, field value distribution similarity, and field context relationship.

[0011] As one implementation, the method further includes: scoring the quality of the collected data, and performing conflict fusion based on at least one of the following factors: quality score, data source trust level, and update time.

[0012] Furthermore, the present invention also provides an adaptive acquisition and processing device for multi-source data, comprising: a profile generation module configured to acquire a data source profile, wherein the data source profile includes dynamic features of the data source; a scheduling module configured to determine an acquisition strategy based on the data source profile and the load status of the target database; an acquisition module configured to execute a data acquisition task according to the acquisition strategy; and a feedback update module configured to update the dynamic features in the data source profile in response to abnormal information generated during the acquisition process.

[0013] Furthermore, the present invention also provides an electronic device, comprising: one or more processors; and a memory for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors perform the method described above.

[0014] In addition, the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method described above.

[0015] Beneficial effects: This invention constructs a data source profile that includes dynamic features such as data source update frequency, historical collection error rate, and response latency. This allows collection decisions to move beyond relying solely on static connection configurations and instead adjust based on the real-time operational status of the data source, thereby reducing scheduling lag caused by mismatches between static configurations and actual conditions. Furthermore, this invention combines the data source profile with the target database's load status for determining the collection strategy. It can dynamically adjust the collection frequency, collection batch size, and execution order based on the source's status and the target's capacity, ensuring timely data collection while reducing the risk of response latency, write failures, or service unavailability in high-load scenarios. Simultaneously, this invention establishes an anomaly feedback mechanism, using anomaly information generated during the collection process to update the dynamic features in the data source profile and correct subsequent collection strategies accordingly. This enables the system to adaptively adjust to changes in the data source status, reducing invalid access to abnormal data sources and repeated retries. Therefore, this invention improves the stability, adaptability, and automation of multi-source data collection processes while reducing manual maintenance costs. Attached Figure Description

[0016] Figure 1 This is a schematic diagram of the method flow according to an embodiment of the present invention; Figure 2 This is a schematic diagram illustrating the process of determining the collection strategy based on data source profiling, as exemplified by an embodiment of the present invention. Figure 3 This is a flowchart illustrating the data source profile and related parameter updates driven by the anomaly feedback of the present invention. Figure 4 This is a schematic diagram of an adaptive acquisition and processing system architecture for multi-source data according to an embodiment of the present invention; Figure 5 This is a schematic diagram of the structure of an electronic device according to an embodiment of the present invention. Detailed Implementation

[0017] To make the objectives, technical solutions, and beneficial effects of this invention clearer, the invention will be further described below in conjunction with the accompanying drawings and specific embodiments. It should be understood that the following embodiments are for illustrative purposes only and are not intended to limit the scope of protection of this invention. Other embodiments obtained by those skilled in the art based on the embodiments of this invention without inventive effort should all fall within the scope of protection of this invention.

[0018] Unless otherwise defined, the technical terms used in this specification have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. The terminology used in this specification is for describing specific embodiments only and is not intended to limit the invention.

[0019] Example 1: like Figure 1 As shown, this embodiment provides an adaptive acquisition and processing method for multi-source data. This method constructs a data source profile containing dynamic features and dynamically adjusts the acquisition strategy by combining feedback information from the acquisition process. The method specifically includes the following steps: Step S100: Obtain a data source profile, wherein the data source profile includes dynamic features of the data source.

[0020] Specifically, the data source profile differs from traditional configuration files that only include static connection configuration information such as IP address, port, and access account. It is used to characterize the data source's operational status, data activity level, and historical collection performance during the data collection process. The dynamic characteristics refer to attribute parameters that change over time, depending on changes in the data source's operational status or collection behavior.

[0021] For example, a data source may have a higher data update frequency during peak business hours and a lower update frequency at night; or, when network conditions fluctuate or interface services malfunction, the response latency of the data source may increase, and the historical collection error rate may also rise. This embodiment continuously monitors and acquires the above dynamic characteristics to generate a data source profile that reflects the current operating status and historical collection performance of the data source, providing a data basis for determining subsequent collection strategies, thereby reducing the decision-making lag problem caused by relying solely on static configuration for collection scheduling.

[0022] Step S200: Determine the collection strategy based on the data source profile and the load status of the target database.

[0023] After obtaining the data source profile, this embodiment performs joint processing with the current load status of the target database. The process of determining the collection strategy can be understood as a process of resource allocation and task scheduling based on the source and target status.

[0024] Specifically, the system can determine at least one of the following parameters for the data collection task: collection frequency, collection batch, execution order, retry parameters, or rate limiting parameters, based on dynamic characteristics reflected in the data source profile, such as data source update frequency, response latency, historical collection error rate, data source trust level, or availability score, combined with the current load status of the target database.

[0025] For example, for data sources with high profile update frequency, low response latency, and low historical collection error rate, the system can assign them higher collection priority and larger collection batches when the target database load is low. For data sources with high profile response latency, high historical collection error rate, or low availability score, the system can reduce their collection frequency or postpone their collection tasks. When the target database is under high load, the system can reduce the collection frequency of non-critical data sources or reduce the single write batch to reduce the risk of response latency, write failure, or service unavailability of the target database due to continuous high-concurrency writes.

[0026] Therefore, by simultaneously considering the status of the source data source and the load status of the target database, a balance can be struck between the timeliness of data acquisition and the stability of system operation.

[0027] Step S300: Execute the data acquisition task according to the acquisition strategy.

[0028] After determining the data acquisition strategy, the system executes data acquisition tasks on external data sources according to the strategy. These data acquisition tasks may include at least one of the following: data extraction, data transformation, data cleaning, data caching, data writing, or data verification.

[0029] Specifically, the system can acquire data from at least one external data source according to the acquisition frequency, batch size, and execution order determined in the acquisition strategy, and write the acquired data to the target database or intermediate cache. During the data acquisition task, the system can also record information such as the execution status, response time, error type, and writing results of the acquisition task in real time, so as to provide a basis for subsequent anomaly feedback and profile updates.

[0030] Step S400: In response to abnormal information generated during the acquisition process, update the dynamic features in the data source profile.

[0031] During the execution of data acquisition tasks, exceptions may occur, such as data source connection errors, field mapping errors, data quality errors, and data import failures. This embodiment does not merely log these exceptions; instead, it uses them as feedback to correct data source profiles, field mapping rules, acquisition strategies, or target database status parameters.

[0032] Specifically, the system can respond to abnormal information generated during the data collection process, determine the abnormal type and attribution object corresponding to the abnormal information, and update the corresponding parameters based on the abnormal type and attribution object. For abnormalities attributable to the running status of the data source, such as connection timeout, access denial, authentication failure, or interface unavailability, the system can update the response latency, historical collection error rate, or availability score in the corresponding data source profile; for abnormalities attributable to data quality, such as a large number of null values, format errors, or business rule validation failures, the system can update the data quality score or data source trust level in the corresponding data source profile; for field mapping abnormalities, the system can update the field mapping rules or reduce the mapping confidence of the corresponding mapping relationship; for database entry failure abnormalities, the system can update the target database load status, adjust the write batch, or adjust the retry strategy.

[0033] The updated data source profile can be used to determine the next round of data collection strategy, enabling the system to adjust its collection behavior based on historical collection performance and real-time anomaly feedback. This establishes a closed-loop processing mechanism encompassing data source status awareness, collection strategy determination, data collection execution, and anomaly feedback correction, improving the robustness and automation of the multi-source data access process.

[0034] Example 2: like Figure 2 As shown, based on the above embodiments, this embodiment further illustrates the specific composition of dynamic features in the data source profile and their application in the process of determining the collection strategy.

[0035] Specifically, the dynamic features contained in the data source profile can serve as the basis for the system to make adaptive data collection decisions. These dynamic features may include at least one of the following: data source update frequency, historical data collection error rate, response latency, data source trust level, availability score, and data quality score.

[0036] The data source update frequency characterizes the rate at which a data source generates new data. A higher data source update frequency indicates a higher level of data activity. Under the condition of meeting the target database load requirements, the data collection frequency can be appropriately increased.

[0037] Historical data collection error rate is used to characterize the anomalies that occurred during the historical data collection process. Data sources with high historical data collection error rates may have issues such as unstable interfaces, network fluctuations, abnormal permissions, or changes in data format. Therefore, their data collection priority can be reduced or the retry interval can be increased.

[0038] Response latency is used to characterize the length of time between initiating a data collection request and receiving a response. Data sources with high response latency may cause data collection tasks to block, affecting overall data collection efficiency. Therefore, data collection concurrency, timeout, or execution order can be adjusted based on response latency.

[0039] Data source trustworthiness rating characterizes the authenticity, accuracy, or business credibility of the data provided by the data source. In the event of data conflicts, data from data sources with higher trustworthiness ratings may have a higher fusion weight. Availability score characterizes the accessibility and stability of the data source interface or service, while data quality score characterizes the quality level of the data provided by the data source in terms of completeness, legality, and standardization.

[0040] After acquiring a data source profile containing the aforementioned dynamic characteristics, the system can calculate a collection priority score based on these dynamic characteristics and the load status of the target database. The collection priority score can be determined based on a weighted calculation of positive and negative factors.

[0041] Positive factors may include at least one of the following: data source update frequency, data source trust level, data quality score, or business importance. Negative factors may include at least one of the following: response latency, historical data collection error rate, target database load status, number of availability anomalies, or number of write failures.

[0042] As an optional implementation, the system can normalize each positive and negative factor and calculate the collection priority score according to preset weights. The higher the data source update frequency, the higher the data source trust level, or the higher the data quality score, the higher the corresponding collection priority score can be; the higher the response latency, the higher the historical collection error rate, or the higher the target database load, the lower the corresponding collection priority score can be.

[0043] In practical applications, the weighting coefficients can be configured according to business needs. For example, in scenarios with high real-time requirements, the weight of the data source update frequency can be increased; in scenarios with high stability requirements, the penalty weight of historical collection error rate, response latency, or target database load status can be increased; and in scenarios with high data quality requirements, the weight of the data source trust level or data quality score can be increased. Through these methods, the system can generate a collection strategy adapted to the current operating state, realizing the dynamic allocation of collection resources.

[0044] Example 3: like Figure 3 As shown, based on the above embodiments, this embodiment further illustrates the data source profile update mechanism driven by anomaly feedback.

[0045] Specifically, during the execution of data acquisition tasks, the system can monitor and record abnormal information. This abnormal information may include at least one of the following: data source connection abnormality, field mapping abnormality, data quality abnormality, and data import failure.

[0046] Data source connection anomalies can include network timeouts, connection failures, access denials, authentication failures, and interface unavailability, causing the data collection task to fail to start or be interrupted. Field mapping anomalies can include source and target fields not matching, field types not convertible, required fields missing, or mapping confidence levels below a preset threshold. Data quality anomalies can include a large number of null values, garbled characters, non-standard formats, abnormal value ranges, or violations of business rules in the collected data. Data import failure anomalies can include primary key conflicts, constraint violations, target database connection failures, target database write timeouts, transaction lock conflicts, or changes to the target table structure.

[0047] After obtaining the abnormal information, the system can determine the abnormal type and the abnormal attribution object corresponding to the abnormal information, and perform corresponding update operations based on the determination results.

[0048] As an optional implementation, when the system detects a data source connection anomaly, it can determine that the anomaly is related to data source availability or network connectivity. In this case, the system can increase the historical collection error rate in the corresponding data source profile, add response latency records, or decrease the availability score of the data source. During subsequent collection strategy determination, the historical collection error rate, response latency, or availability score can be used as negative factors in calculating the collection priority score, thereby reducing the collection frequency of the data source in a short period or extending its retry interval, reducing invalid access to the abnormal data source.

[0049] As an alternative implementation, when the system detects a data quality anomaly, it can determine that the anomaly is related to the standardization, completeness, or business validity of the data provided by the data source. In this case, the system can lower the data quality score in the corresponding data source profile, or lower the data source trust level if preset conditions are met. The updated data quality score or data source trust level can be used for subsequent data conflict fusion and collection strategy determination to reduce the probability of low-quality data entering the target database.

[0050] As an alternative implementation, when the system detects a field mapping anomaly, it can determine that the anomaly may be related to differences in field naming, incompatible field types, changes in the target table structure, or incomplete mapping rules. In this case, the system can lower the mapping confidence of the corresponding field mapping relationship, mark the relevant fields as fields to be verified, or trigger a field mapping rule update process. For field mapping anomalies, it is not necessary to directly lower the data source trust level unless the field mapping anomaly is related to long-standing data structure anomalies or data standardization issues in the data source.

[0051] As an alternative implementation, when the system detects an inbound failure exception, it can determine that the exception may be related to the target database load status, target table constraints, primary key conflicts, transaction locks, or changes in the target table structure. In this case, the system can update the target database load status and adjust the write batch size, write frequency, or retry strategy. For inbound failure exceptions that can be clearly attributed to errors in the data source, the system can also update the data quality score or historical collection error rate of that data source accordingly.

[0052] Through the above mechanism, the system can transform abnormal information during the data collection process into the basis for updating data source profiles, field mapping rules, collection strategies, or target database status parameters. The updated parameters can be used to determine the next round of collection strategies, allowing the collection frequency, collection batch, execution order, and retry parameters to be adjusted according to the collection execution results, thereby improving the stability and adaptability of the multi-source data access process.

[0053] Example 4: Based on the above embodiments, this embodiment further illustrates the process of intelligent field mapping of the collected data after the data collection task is performed.

[0054] Specifically, since multi-source data originates from different business systems, the naming, data types, or value rules for the same business object may differ between these systems. For example, for the business meaning of "customer number," a customer relationship management system might name it "customer_id," an order management system might name it "cust_no," and a financial system might name it "clientCode." Relying solely on manual configuration of field mapping tables could result in a large workload, high maintenance costs, and untimely updates.

[0055] Therefore, this embodiment provides a multi-dimensional intelligent field mapping method. The system can perform field mapping on the collected data, and the field mapping can be generated based on at least one of field name similarity, field type similarity, field value distribution similarity, and field context relationship.

[0056] Field name similarity measures the degree of closeness between source and target field names at the character or semantic level. The system can calculate field name similarity using edit distance algorithms, keyword matching, pinyin matching, abbreviation matching, or semantic vector models. For example, "cust_no" and "customer_id" are not identical in character form, but through abbreviation recognition or semantic analysis, the correspondence between "cust" and "customer" can be identified, thus obtaining the corresponding name similarity score.

[0057] Field type similarity is used to determine whether the data types of the source and target fields are consistent or convertible. Specifically, if both the source and target fields are date / time types, a high field type similarity can be assigned; if the source field is an integer and the target field is a string, but they are convertible in business logic, a medium field type similarity can be assigned; if the types are incompatible, the field type similarity can be reduced. By introducing field type similarity, the probability of incorrectly mapping numeric fields to text description fields can be reduced.

[0058] Field value distribution similarity is used to determine the correspondence between a source field and a target field based on the characteristics of their values. The system can analyze the data distribution characteristics of the source and target fields through sampling. These data distribution characteristics can include at least one of the following: numerical range, enumerated value set, mean, variance, null value rate, unique value proportion, or value length distribution. For example, if the source field "sex" has values ​​of {0, 1} and the target field "gender" has values ​​of {M, F}, although their field names and value forms are different, they both have a binary enumerated distribution characteristic, and the system can determine that there is a potential mapping relationship between them.

[0059] Field context relationships are used to help determine field mapping relationships by leveraging the association information within the data table or data structure where the field resides. The system can analyze the similarity between other fields in the source field's table and other fields in the target field's table. If the surrounding fields in the source and target tables have high similarity, the likelihood of a mapping relationship between the source and target fields increases accordingly. For example, if the source table contains fields such as "order number," "order amount," and "order date," and the target table also contains fields with similar business meanings, then the "amount" field in the source table and the "amount" field in the target table can obtain a high context relationship score.

[0060] In practical applications, the system can perform weighted calculations on field name similarity, field type similarity, field value distribution similarity, and field contextual relationships to generate a comprehensive mapping similarity. When the comprehensive mapping similarity is higher than a preset threshold, a mapping relationship between the source field and the target field can be established; when the comprehensive mapping similarity is lower than the preset threshold but higher than the manual verification threshold, the mapping relationship can be marked as pending verification; when the comprehensive mapping similarity is lower than the manual verification threshold, no mapping relationship needs to be established.

[0061] By using the above multi-dimensional field mapping method, the manual configuration cost caused by inconsistent field naming can be reduced, and the automation and accuracy of field mapping in the process of multi-source data access can be improved.

[0062] Example 5: Based on the above embodiments, this embodiment further illustrates the process of quality scoring and conflict fusion of the collected data after the data collection task is performed.

[0063] Specifically, when multiple data sources provide data for the same business object, differences in update time, data quality, and reliability may exist among the data sources. Directly overwriting earlier-written data with later-written data could result in lower-quality data overwriting higher-quality data. Therefore, this embodiment performs a quality score on the collected data and performs conflict fusion based on at least one of the following factors: quality score, data source reliability level, and update time.

[0064] Before performing conflict fusion, the system can first determine whether multiple candidate records from different data sources correspond to the same business object. Specifically, the system can determine the object correspondence between multiple candidate records based on business primary keys, unique identifiers, field combination matching rules, or similarity matching rules. For example, it can determine whether records in different data sources correspond to the same customer, the same order, or the same business entity based on the matching results of customer ID, order ID, ID number, mobile phone number, unified social credit code, or a combination of multiple fields.

[0065] After determining that candidate records correspond to the same business object, the system can perform a quality score on the collected data. The quality score may include at least one of the following: completeness score, accuracy score, timeliness score, legality score, or consistency score.

[0066] The integrity score can be determined based on the ratio of the number of non-empty fields to the total number of fields. The accuracy score can be determined based on whether the data conforms to predefined format rules, business rules, or validation rules. The timeliness score can be determined based on the difference between the data generation time, update time, or collection time and the current time; the smaller the difference, the higher the timeliness score. The legality score can be determined based on whether the data meets the requirements of field type, value range, enumeration value rules, or database constraints. The consistency score can be determined based on the degree of consistency of field values ​​for the same business object across different data sources.

[0067] After obtaining the quality score, the system can perform conflict fusion based on at least one of the following factors: quality score, data source trust level, and update time. As an optional implementation, the system can calculate a comprehensive score for each candidate record, which can be obtained by weighting the quality score, data source trust level, and normalized update time. For candidate records from high-trust-level data sources, with higher quality scores and more recent updates, their comprehensive scores can be higher than other candidate records. The system can select candidate records with higher comprehensive scores as the primary records to write into the target database, thereby reducing the probability of low-quality data overwriting high-quality data.

[0068] As another optional implementation, the system can also employ a field-level fusion strategy. When candidate records from different data sources have different strengths in different fields, the system can select higher-quality data values ​​for each target field. For example, if the "customer phone number" field provided by data source A has high completeness and passes format validation, while the "customer address" field provided by data source B has a more recent update time and a higher reliability level, the system can fuse the "customer phone number" from data source A and the "customer address" from data source B into a single target record.

[0069] Through the aforementioned quality scoring and conflict fusion mechanism, when inconsistencies exist among multi-source data, the data content to be written into the target database can be determined based on factors such as data quality, data source trust level, and update time, thereby improving the consistency and trustworthiness of the data entering the database.

[0070] Example 6: like Figure 4 As shown, this embodiment provides an adaptive acquisition and processing device for multi-source data. This device is used to implement the adaptive acquisition and processing method for multi-source data described in any one of embodiments 1 to 5 above. The device may include a profile generation module 601, a scheduling module 602, an acquisition module 603, and a feedback update module 604.

[0071] The profile generation module 601 is used to acquire a data source profile, which includes dynamic characteristics of the data source. Specifically, the profile generation module can establish a connection with an external data source and generate or update the data source profile based on the data source's response information, collection records, abnormal records, or data quality detection results. The dynamic characteristics in the data source profile may include at least one of the following: data source update frequency, historical collection error rate, response latency, data source trust level, availability score, or data quality score.

[0072] The scheduling module 602 is used to determine a collection strategy based on the data source profile and the load status of the target database. The collection strategy may include at least one of the following: collection frequency, collection batch size, execution order, retry parameters, or rate limiting parameters. Specifically, the scheduling module can obtain the data source profile from the profile generation module and the load status of the target database from the target database monitoring interface. The scheduling module can calculate a collection priority score based on positive and negative factors, and generate a collection task queue based on the collection priority score. The positive factors may include at least one of the following: data source update frequency, data source trust level, data quality score, or business importance; the negative factors may include at least one of the following: response latency, historical collection error rate, target database load status, or number of availability anomalies.

[0073] The acquisition module 603 is used to execute data acquisition tasks according to the acquisition strategy. Specifically, the acquisition module can receive acquisition instructions issued by the scheduling module and extract data from the corresponding data source based on the acquisition instructions. The acquisition module may include at least one of a protocol adaptation unit, a data extraction unit, a data conversion unit, a data cleaning unit, and a data writing unit to support data acquisition from relational databases, interface services, file systems, message queues, or other types of data sources. During the execution of the acquisition task, the acquisition module can also record the task execution status and generate exception information when connection failure, field mapping failure, data quality verification failure, or database entry failure occurs.

[0074] The feedback update module 604 is used to update the dynamic features in the data source profile in response to abnormal information generated during the data acquisition process. Specifically, the feedback update module can receive abnormal information generated by the acquisition module, determine the abnormal type and attribution object corresponding to the abnormal information, and generate update instructions based on the determination results. For data source connection abnormalities, the feedback update module can update the historical acquisition error rate, response latency, or availability score in the corresponding data source profile; for data quality abnormalities, the feedback update module can update the data quality score or data source trust level in the corresponding data source profile; for field mapping abnormalities, the feedback update module can update the field mapping rules or reduce the mapping confidence of the corresponding mapping relationship; for database entry failure abnormalities, the feedback update module can update the target database load status, adjust the write batch, or adjust the retry strategy.

[0075] Through the coordinated operation of the above modules, the device provided in this embodiment can determine the acquisition strategy based on the data source profile and the target database load status, and update the relevant parameters according to the abnormal information in the acquisition process, thereby realizing the dynamic adjustment of the multi-source data acquisition strategy and improving the efficiency and reliability of the multi-source data access process.

[0076] Example 7 like Figure 5 As shown, this embodiment provides an electronic device 700, which is used to execute the adaptive acquisition and processing method for multi-source data described in any of the above embodiments. The electronic device 700 may be a standalone server, a server cluster, a cloud platform computing node, an edge computing gateway, or a front-end acquisition server deployed between the data source and the target database.

[0077] Specifically, the electronic device 700 may include one or more processors 710, a memory 720, and a communication interface 730, which can be connected via a bus 740. The processor 710 executes a computer program stored in the memory 720 to implement the method steps described in any of embodiments 1 to 5. The memory 720 stores one or more programs, as well as intermediate data generated during the acquisition process, data source profile information, anomaly records, field mapping rules, or acquisition strategy parameters. The communication interface 730 communicates with external data sources, target databases, or other network devices.

[0078] The memory 720 may include high-speed random access memory or non-volatile memory, such as disk storage devices, flash memory devices, or other non-transitory solid-state storage devices. In some embodiments, the memory 720 may also include a memory remotely located relative to the processor 710, which can be connected to the processor 710 via a network for storing historical acquisition data, data source profile backup information, or scheduling strategy records.

[0079] During operation, the processor 710 can read the pre-stored or real-time acquired data source connection configuration in the memory 720, generate a data source profile, and store it in a designated area of ​​the memory 720. Subsequently, the processor 710 can read the data source profile and the target database load status, calculate the acquisition priority score, and generate an acquisition strategy. During the execution of the acquisition task, the processor 710 can communicate with the external data source through the communication interface 730, temporarily storing the acquired data in the buffer of the memory 720 or writing it to the target database. If abnormal information is generated during the acquisition process, the processor 710 can update the corresponding data source profile, field mapping rules, acquisition strategy parameters, or target database status parameters according to the abnormality type and abnormality attribution object, thereby achieving adaptive acquisition control.

[0080] This embodiment also provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, it can implement the adaptive acquisition and processing method for multi-source data described in any one of embodiments 1 to 5 above.

[0081] Specifically, the computer-readable storage medium can be any tangible medium that contains or stores a program that can be used by, or in conjunction with, an instruction execution system, apparatus, or device. For example, the computer-readable storage medium may include a portable computer disk, hard disk, random access memory, read-only memory, erasable programmable read-only memory, flash memory, portable compact disk read-only memory, optical storage device, magnetic storage device, or any combination of the above media.

[0082] When the computer program is loaded into the electronic device 700 and executed by the processor 710, the electronic device 700 can execute at least one of the following processing logics: profile generation, strategy scheduling, data acquisition, anomaly feedback, field mapping, quality scoring, or conflict fusion, thereby achieving adaptive acquisition and processing of multi-source data.

[0083] Example 8: To further illustrate the application of this embodiment of the invention, this embodiment uses a multi-source access scenario for enterprise business databases as an example. In this scenario, external data sources may include customer relationship management systems, order management systems, warehouse management systems, financial systems, and third-party logistics interfaces. Each data source may use different types of databases or interface protocols, and the field naming rules, field types, and data update frequencies may differ.

[0084] During system initialization, the profile generation module can acquire initial profiles for each data source. Taking a customer relationship management system (CRM) system as an example, the system can detect that the data source is a relational database and set the dynamic features in its data source profile to initial values. For example, the data source update frequency can be marked as high-frequency, the historical collection error rate can be set to an initial value, the response latency can be determined based on the initial detection results, and the data source trust level can be determined based on business preset rules or historical data quality evaluation results. The scheduling module can determine the collection strategy for the CRM system based on the initial profile and the current load status of the target database. For example, when the target database load is low, the collection frequency of the data source can be increased, and its collection task can be placed at a higher position in the collection queue.

[0085] During the data collection process, the system may encounter inconsistencies in field naming. For example, the same business object, "Customer Number," might be named "customer_id" in a customer relationship management system, "cust_no" in an order management system, and "clientCode" in a financial system. To address this, the data collection module can perform field mapping on the collected data and calculate the overall mapping similarity between the source and target fields.

[0086] Specifically, for the "cust_no" field, although its name is not exactly the same as "customer_id", the system can identify the semantic correspondence between "cust" and "customer". Simultaneously, the system can detect that the data type of this field is compatible with the target field, and through sampling analysis, it finds that its value distribution has a high similarity to the target field. Furthermore, the system can improve the mapping confidence between this field and the target field by considering the contextual relationships between surrounding fields such as "order amount" and "order date" in the table containing this field and related fields in the target table. After comprehensively considering field name similarity, field type similarity, field value distribution similarity, and field contextual relationships, the system can determine that a mapping relationship exists between "cust_no" and "customer_id" and automatically establish the field mapping.

[0087] During data collection, if the network environment of the third-party logistics interface fluctuates, causing multiple consecutive connection timeouts, the collection module can capture the data source connection anomaly and generate an error message. The feedback update module responds to this error message by increasing the historical collection error rate in the data source profile corresponding to the logistics interface and decreasing its availability score. In the next round of scheduling decisions, the scheduling module can recalculate the collection priority score based on the updated data source profile. Due to the increased historical collection error rate and decreased availability score, the collection priority score of this logistics interface can be reduced. The system can then reduce the collection frequency of this logistics interface and postpone its execution order in the collection queue. For example, the system can adjust the collection frequency of this logistics interface from once every 5 minutes to once every 30 minutes.

[0088] Furthermore, for the same customer, the Customer Relationship Management System (CRM) and the Order Management System may provide inconsistent data. For example, the two systems may provide different customer phone numbers. The system can first determine whether records in different data sources correspond to the same customer based on customer ID, mobile phone number, ID number, or field combination matching rules. After determining that they correspond to the same customer, the system can perform quality scoring on relevant candidate records and perform conflict fusion based on the quality score, data source trust level, and update time. If the CRM data source has a high trust level and its provided phone number field integrity score and format validation results are excellent, the system can select the phone number provided by the CRM as the phone number field value in the target record and write it into the target database. If the customer address field provided by the Order Management System has a more recent update time and a higher quality score, the system can also use a field-level fusion method to write the customer address field value provided by the Order Management System into the fused target record.

[0089] As can be seen from the above processing of multi-source access scenarios for enterprise business databases, the method provided by the embodiments of the present invention can realize data source profile generation, dynamic determination of collection strategy, intelligent field mapping, anomaly feedback correction, and conflict fusion processing during multi-source data access, thereby improving the automation, stability, and data consistency of multi-source data access.

[0090] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can make changes, substitutions, or combinations to the above embodiments within the technical scope disclosed in the present invention, and such changes, substitutions, or combinations should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. An adaptive acquisition and processing method for multi-source data, characterized in that, include: Obtain a data source profile, wherein the data source profile includes dynamic characteristics of the data source; Based on the data source profile and the load status of the target database, a collection strategy is determined; Execute the data acquisition task according to the acquisition strategy; In response to abnormal information generated during the data collection process, the dynamic features in the data source profile are updated.

2. The method according to claim 1, characterized in that, The dynamic characteristics include at least one of the following: data source update frequency, historical collection error rate, response latency, and data source trust level.

3. The method according to claim 2, characterized in that, The step of determining the collection strategy based on the data source profile and the load status of the target database includes: Calculate the collection priority score, which is determined based on a weighted average of positive and negative factors. The positive factors include the data source update frequency and the data source trust level, while the negative factors include response latency, historical collection error rate, and the target database load status.

4. The method according to claim 1, characterized in that, The abnormal information includes at least one of the following: data source connection abnormality, field mapping abnormality, data quality abnormality, and data entry failure abnormality.

5. The method according to claim 4, characterized in that, Updating the dynamic features in the data source profile includes: Based on the abnormal information, perform at least one of the following operations: increase the historical data collection error rate, reduce the data source trust level, or adjust the data collection priority.

6. The method according to claim 1, characterized in that, The method further includes: The collected data is mapped to fields, and the field mapping is generated based on at least one of field name similarity, field type similarity, field value distribution similarity, and field context relationship.

7. The method according to claim 1, characterized in that, The method further includes: The collected data is scored for quality, and conflict fusion is performed based on at least one of the following factors: quality score, data source trust level, and update time.

8. An adaptive acquisition and processing device for multi-source data, characterized in that, include: The profile generation module is configured to acquire a profile from a data source, wherein the profile from the data source includes dynamic features of the data source. The scheduling module is configured to determine the collection strategy based on the data source profile and the load status of the target database; The data acquisition module is configured to perform data acquisition tasks according to the acquisition strategy. The feedback update module is configured to update the dynamic features in the data source profile in response to abnormal information generated during the collection process.

9. An electronic device, characterized in that, include: One or more processors; A memory for storing one or more programs that, when executed by one or more processors, cause the one or more processors to implement the method as described in any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method as described in any one of claims 1 to 7.

Images