Self-adaptive real-time data acquisition system and acquisition method thereof

Abstract

The invention belongs to the technical field of intelligent control, and particularly relates to a self-adaptive real-time data acquisition system and an acquisition method thereof. The system comprises an access control module used for controlling access of a target data source and a target database; the protocol adaptation module is used for being connected with the target data source based on the protocol configuration information of the target data source; the data analysis module is used for analyzing the data information of the target data source and constructing a unified data model; the event model module is used for responding to the event triggering condition and generating a real-time event model; the self-adaptive strategy module is used for generating a self-adaptive acquisition strategy based on the unified data model and the real-time event model; the acquisition scheduling module is used for generating an acquisition task based on the acquisition configuration information and the point location configuration information of the target data source; and based on a preset acquisition strategy or / and an adaptive strategy, executing the acquisition task to acquire data information of the target data source. According to the invention, automatic configuration can be realized, and the flexibility and adaptivity of data acquisition are improved, so that the data acquisition efficiency is improved.

Description

Technical Field

[0001] The invention belongs to the technical field of intelligent control, and in particular relates to an adaptive real-time data acquisition system and an acquisition method thereof. Background Art

[0002] Data acquisition technology refers to the technology of collecting, organizing and analyzing data from various data sources (relational database / non-relational database / big data / message queue) such as various industrial PLCs, DCS sensor devices, IoT devices and PC computing through specific methods and tools. Data acquisition technology plays a vital role in many fields such as Industry 4.0, autonomous driving, smart cities, emergency management, smart logistics, etc.

[0003] Traditional data collection tools can manually or automatically extract data from specific data sources (such as databases, files, etc.) based on a single data format and data parsing protocol. These tools usually have fixed data collection templates and parameter settings and can only be applied to specific types of data collection tasks.

[0004] However, in the complex and ever-changing environment of actual industrial production, facing mixed fields and diverse business needs, traditional data collection tools show obvious limitations. These tools usually require manual configuration of collection parameters, which not only increases the complexity of operation, but also brings the risk of errors. Faced with data sources of different types and formats, traditional tools often require additional custom development work to adapt, which greatly limits their flexibility and versatility. Furthermore, when the data environment changes dynamically, traditional tools often cannot respond quickly, resulting in the timeliness of data collection being affected or even delayed. In addition, the fixed collection frequency setting also brings new problems: when the data changes slowly, frequent collection may lead to unnecessary waste of resources; when the data fluctuates greatly, the fixed frequency may miss key information, affecting the integrity and accuracy of the data.

[0005] In summary, traditional data collection tools are obviously unable to meet the needs of efficient, flexible, and real-time data collection when faced with complex scenarios in actual industrial production. New solutions are urgently needed to meet these challenges. Summary of the invention

[0006] In view of the shortcomings of the prior art described above, the purpose of the present invention is to provide an adaptive real-time data collection system and a collection method thereof, which are used to solve the problems existing in traditional data collection tools, and automatically identify and adapt to the data structure and business logic of new fields through a built-in domain knowledge base and an adaptive learning algorithm. According to system performance, data importance and environmental changes, the collection strategy is dynamically adjusted to ensure the real-time and high efficiency of data collection.

[0007] To achieve the above objectives and other related objectives, the present invention provides an adaptive real-time data acquisition method, comprising:

[0008] Acquire point data and rule configuration from the target database, and obtain protocol configuration information, acquisition configuration information, and point configuration information that match each target data source;

[0009] Generate a collection task based on the collection configuration information and point configuration information of the target data source;

[0010] Establishing a connection with the target data source based on the protocol configuration information of the target data source;

[0011] According to the preset collection strategy, the collection task is executed to collect data information of the target data source;

[0012] According to the adaptive collection strategy, the collection task is executed to collect data information of the target data source; the adaptive collection strategy is generated based on the real-time event model, and the real-time event model is generated by the event triggering condition;

[0013] Parse the data information of the target data source and store it in the target database.

[0014] Furthermore, after obtaining the point data and rule configuration from the target database, the following steps are also included:

[0015] Subscribe to the data information of the target database;

[0016] Synchronizing a clock of the target database;

[0017] Based on the data information of the subscribed target database and the clock of the synchronized target database, the protocol configuration information, collection configuration information and point configuration information matching each target data source are obtained.

[0018] Furthermore, executing the collection task to collect data information of a target data source according to a preset collection strategy includes:

[0019] Read the current clock;

[0020] Based on the current clock, the acquisition task is allocated to the corresponding execution unit according to the preset acquisition strategy;

[0021] The execution unit executes the collection task to collect data information from a target data source.

[0022] Further, according to the adaptive collection strategy, executing the collection task to collect data information of the target data source includes:

[0023] Read the current clock;

[0024] Based on the current clock, the acquisition task is allocated to the corresponding execution unit according to the adaptive acquisition strategy;

[0025] The execution unit executes the collection task to collect data information from a target data source.

[0026] Further, allocating the acquisition task to a corresponding execution unit includes:

[0027] Creating a collection subtask for the collection task;

[0028] The acquisition subtasks are allocated to corresponding execution units according to their priorities; the priorities of the acquisition subtasks are determined by the importance and urgency of the acquisition subtasks and the availability of processing resources.

[0029] Furthermore, the adaptive acquisition strategy includes:

[0030] Adaptive strategy, when a delay occurs, the collection task is performed according to the adaptive collection interval; the adaptive collection interval is determined by the system status and business needs;

[0031] Burst strategy: when a delay occurs, the collection task is executed according to the burst collection interval;

[0032] The skip strategy skips the missed collection interval when a delay occurs, and performs the collection task at the preset collection interval in the next multiple cycle.

[0033] Furthermore, the data information of the target data source is parsed and stored in the target database including:

[0034] The unified data model is archived and then deposited into the target database.

[0035] Analyze the data information of the target data source and build a unified data model;

[0036] The unified data model is stored in the target database through data archiving.

[0037] To achieve the above object, the present invention also provides an adaptive real-time data acquisition system, comprising:

[0038] An access control module, used to control access to a target data source and a target database;

[0039] A protocol adapter module, configured to connect to the target data source based on the protocol configuration information of the target data source;

[0040] The data analysis module is used to analyze and build a unified data model based on the data information of the target data source;

[0041] An event model module, for generating a real-time event model in response to an event triggering condition;

[0042] An adaptive strategy module, used for generating an adaptive acquisition strategy based on the real-time event model;

[0043] The collection scheduling module is used to generate a collection task based on the collection configuration information and point configuration information of the target data source; and to execute the collection task to collect data information of the target data source based on a preset collection strategy and / or an adaptive strategy.

[0044] Furthermore, the adaptive strategy module includes:

[0045] An adaptive strategy unit is used to execute the collection task according to the adaptive collection interval when a delay occurs; the adaptive collection interval is determined by the system state and business requirements;

[0046] The burst strategy unit is used to complete the next task as quickly as possible when a delay occurs until a task catches up with the normal frequency;

[0047] The skip strategy unit is used to skip the missed collection interval when a delay occurs, and perform the collection task according to the preset collection interval in the next multiple cycle.

[0048] The present invention has the function of automatically configuring protocol configuration information and collection configuration information, which effectively reduces the complexity of operation and the risk of error. Through the protocol configuration information, the present invention can automatically adapt to target data sources of different types and different data formats. When an emergency occurs, the present invention can respond quickly and adjust the collection strategy. The present invention avoids the waste of resources caused by the use of a fixed collection frequency when the data changes slightly, and the omission of key information when the data changes significantly. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 It is a schematic diagram of the structure of an adaptive real-time data acquisition system according to an embodiment of the present invention;

[0050] Figure 2 is a schematic diagram of the structure of an adaptive strategy module according to an embodiment of the present invention;

[0051] Figure 3 is a flow chart of an adaptive real-time data collection method according to an embodiment of the present invention;

[0052] Figure 4 is a schematic diagram of an outbreak strategy according to an embodiment of the present invention;

[0053] Figure 5 It is a schematic diagram of the skip strategy of an embodiment of the present invention. DETAILED DESCRIPTION

[0054] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described implementation is only a part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0055] Traditional data collection tools have limitations in data formats and parsing protocols. They can only process a single type and are difficult to cope with diverse data needs. Moreover, the setting of collection parameters relies on manual operation, which is not only cumbersome but also prone to configuration errors due to human error. For example, when configuring a data source, if the wrong address or port information is manually entered, the collection will fail directly. It is also difficult to accurately optimize and adjust the collection frequency according to the dynamic characteristics of the data, which may result in waste of resources or untimely data collection.

[0056] Based on the above problems, the present invention proposes an adaptive real-time data acquisition system and a method thereof to solve the problem that traditional data acquisition tools cannot meet the needs of efficient, flexible and real-time data acquisition when facing complex scenarios in actual industrial production.

[0057] In an exemplary scenario, the target data source is complex and diverse, and may include data from different types of devices and stored in different formats, such as real-time monitoring data generated by various sensors, which are distributed in different environments, including temperature, humidity, and pressure sensors in industrial production workshops, and wind speed, wind direction, and air quality sensors in outdoor environment monitoring; there may also be data from different software systems, such as data generated by the company's financial system, sales system, inventory management system, etc. The formats and transmission methods of these data are different, some are structured database data, and some are unstructured text or log data. The target database may be a diverse data storage and processing unit, such as a traditional relational database for long-term storage and management of data with a strict structure; there may also be a non-relational database, such as a NoSQL database suitable for storing semi-structured and unstructured data, for processing massive log data, user behavior data, etc.; it may also involve a data warehouse for integrating and analyzing data from multiple data sources to support corporate decision-making; in addition, the data sink may also include some real-time data processing platforms for rapid processing and response to the collected real-time data. The adaptive real-time data acquisition system and method of the present invention can automatically identify and adapt different target data sources and different data formats, realize comprehensive data acquisition, monitoring and transmission from the target data source to the target database, and maintain efficient data acquisition performance.

[0058] Figure 1 Schematic diagram of the structure of an adaptive real-time data acquisition system according to an embodiment of the present invention. Figure 1 As shown, the system comprises:

[0059] An access control module, used to control access to a target data source and a target database;

[0060] A protocol adapter module, configured to connect to the target data source based on the protocol configuration information of the target data source;

[0061] Data parsing module, used for data information of target data source, parsing and building unified data model;

[0062] An event model module, for generating a real-time event model in response to an event triggering condition;

[0063] An adaptive strategy module, used to generate an adaptive acquisition strategy based on the unified data model and the real-time event model;

[0064] The collection scheduling module is used to generate a collection task based on the collection configuration information and point configuration information of the target data source; and to execute the collection task to collect data information of the target data source based on a preset collection strategy and / or an adaptive strategy.

[0065] In an embodiment of the present invention, the access control module is an important component of the adaptive real-time data acquisition system of the embodiment of the present invention, and is responsible for coordinating the functions of the two submodules, the data parsing module and the protocol adapter module. The access control module can manage the access of the target data source to ensure the stability and effectiveness of data acquisition. The adaptive real-time data acquisition system reads the data source and the data sink configuration, and the access control module can automatically connect and identify various target data sources, wherein the target data source includes but is not limited to industrial production equipment, industrial sensing equipment, industrial control equipment, smart water, electricity and gas meters, smart pressure sensors and smart helmets, etc. The integrity and consistency of the data are guaranteed by unified management of the accessed target data sources.

[0066] After successfully establishing a connection with the target data source and the target database, the adaptive real-time data acquisition system will read two key contents from the database: one is the point data, which covers the detailed parameter conditions corresponding to each monitoring point. For example, in the environmental monitoring scenario, the point data may contain specific values ​​such as temperature, humidity, and pollutant concentration at different monitoring locations. These data are the core objects for subsequent analysis, processing, and adaptive adjustment based on their changes; the second is all rule configurations. The rule configuration contains a lot of key information, such as which point data are the focus of attention, the collection accuracy requirements for different types of data, and what collection strategies should be adopted under what circumstances. These rules will fully guide the specific operation of subsequent collection tools to ensure that the entire data collection process can be carried out in a standardized and orderly manner.

[0067] In the embodiment of the present invention, the protocol adapter module can seamlessly connect and integrate information flows from various heterogeneous data sources. The protocol adapter module can widely support the collection protocol set, including but not limited to international standards and industry specifications such as OPC, MQTT, MODBUS, IEC 101 and IEC 104. The selection and application of these protocols ensure the efficiency, accuracy and wide compatibility of the data collection process.

[0068] In an embodiment of the present invention, the data analysis module can convert the collected data information into a unified data model. For example, for the collected data, the data analysis module can identify its data format, such as structured data (such as tabular data in a relational database, XML files), semi-structured data (such as JSON format data) and unstructured data (such as text files, images, audio, etc.). For different formats, corresponding analysis methods and techniques are adopted. For example, for structured data, SQL query statements or specific database reading tools can be used to extract data; for JSON data, JSON parsing libraries can be used for parsing; for text data, natural language processing technology can be used for information extraction and structured processing.

[0069] It should be noted that, in the process of parsing the data, the data parsing module converts and maps the extracted data according to a predefined unified data model. For example, operations such as renaming of data fields, conversion of data types, and adjustment of data structures ensure that the final data meets the specifications of the unified data model. For example, an embodiment of the present invention uniformly converts the fields representing dates in different target data sources into a specific date format, and uniformly converts different encoding methods into standard encoding. The data parsing module of the present invention parses the data of different target data sources into a unified data model, thereby eliminating the differences in data formats and structures, so that the data is consistent and standardized in the entire system, which is conducive to real-time analysis of data conditions, so as to provide more accurate collection.

[0070] In some embodiments, after the access control module reads the rule configuration, the adaptive real-time data acquisition system can uniformly manage the target data source (source) and the data sink (sink). When data is collected from a certain industrial device (data source), the access control module obtains the relevant configuration of the device and submits the data collection work to the protocol adapter module. If the data communication protocol used by the industrial device is Modbus, the protocol adapter module will perform protocol adaptation on the collected data information. For example, the Modbus protocol is divided into Modbus RTU, Modbus ASCII, and ModbusTCP / IP protocols. These specific protocols will be specifically adapted through the protocol adapter in combination with the characteristics of the protocol. If the adapted protocol is Modbus ASCII, the next step is to enter the data parsing module, and build a unified data model after data parsing for the Modbus ASCII protocol. At this time, the parsed unified data model can be archived (sink) and stored in the target database.

[0071] In some embodiments, the user can quickly generate a configuration template through the command line. After simply modifying the configuration parameters according to the content of the configuration template, the adaptive real-time data acquisition system can be started. The adaptive real-time data acquisition system can automatically adapt the protocol of the data source so as to connect with the target data source, thereby facilitating the execution of the acquisition task in the subsequent process.

[0072] The embodiment of the present invention can configure multiple target data sources, and configure the acquisition protocol under each target data source, so that when data is collected from multiple different devices (data sources), protocol adaptation is automatically achieved.

[0073] In an embodiment of the present invention, the event model module can generate a real-time event model in response to event triggering conditions. The event model can include information such as time, point, point name, point value, and state. This method can build an efficient and flexible event processing framework. The event model module can comprehensively capture and accurately process various events occurring in the system. These events can be the arrival of data, the start or end of a collection task, changes in the state of a data source (such as connection interruption, recovery), etc. It can also include but is not limited to abnormal data fluctuations, equipment failure warnings, system performance bottlenecks, etc. Through carefully defined event triggering conditions, the adaptive real-time data acquisition system can instantly identify and respond to these key events to ensure that problems are promptly addressed and processed. Each type of event corresponds to a set of detailed response mechanisms, which automatically trigger corresponding processing procedures or operating instructions according to the nature and severity of the event.

[0074] In the embodiment of the present invention, the event model module can monitor and process events such as equipment status changes and production data anomalies. For example, when a device fails, a "equipment failure event" is triggered, and the data collection strategy is adjusted to focus on collecting data related to the failure in order to perform failure analysis.

[0075] In the embodiment of the present invention, the adaptive strategy module can keenly perceive the dynamic changes of the target data source, carefully interpret the multi-dimensional requirements of the system configuration, and deeply understand the personalized needs of users. On this basis, it can autonomously and dynamically adjust its collection strategy and parameter configuration to ensure that the data collection process is always kept in the optimal state.

[0076] Exemplarily, the algorithm interface of the adaptive strategy module generates an adaptive collection strategy through thresholds (high alarm, high-high alarm, low alarm, low-low alarm) or real-time events (such as switches), wherein the algorithm interface generates an adaptive algorithm through thresholds or real-time event models based on specific business logic, and then adjusts the data collection strategy and performs collection scheduling in the next collection cycle.

[0077] In some embodiments, by monitoring real-time events, such as collecting data from a device, when a parameter (such as temperature) in the device approaches a certain value, an event is triggered and the device is regulated accordingly. In the adaptive data acquisition system, when the temperature parameter approaches a threshold value when collecting data from the device, an event is triggered and the acquisition frequency is adjusted (it may be faster or slower, assuming that more frequent acquisition is required) so as to monitor and find out in time that the device parameter has reached the threshold value, and control the device to regulate the device in time to avoid errors caused by time.

[0078] In some embodiments of the present invention, Figure 2 is a schematic diagram of the structure of an adaptive strategy module according to an embodiment of the present invention; Figure 2 As shown, the adaptive strategy module includes:

[0079] An adaptive strategy unit is used to execute the collection task according to the adaptive collection interval when a delay occurs; the adaptive collection interval is determined by the system state and business requirements;

[0080] The burst strategy unit is used to execute the collection task according to the burst collection interval when a delay occurs;

[0081] The skip strategy unit is used to skip the missed collection interval when a delay occurs, and perform the collection task according to the preset collection interval in the next multiple cycle.

[0082] The embodiment of the present invention implements adaptive collection strategies through multiple strategy units, and can quickly adjust the collection strategy for the target data source, including speeding up the collection frequency (reducing the collection interval), slowing down the collection frequency (increasing the collection interval), changing the target data source, etc. These methods can adjust the target data source to avoid data errors caused by the collection time and affect the real-time performance of the collection.

[0083] In the embodiment of the present invention, the acquisition scheduling module, as the core commander of data flow, is responsible for the accurate scheduling of the entire process of data acquisition, processing and transmission. It is not only a time management tool, but also the key to ensure data timeliness and system response speed. Different data sources have different characteristics, such as data format, data volume, data generation rules, etc., and the update frequency is also different.

[0084] It should be noted that the collection scheduling module is mainly used to dynamically adjust the collection frequency according to business needs. In a complex production environment, when encountering emergencies, it can also ensure that key data is collected without omission at the frequency that users are concerned about.

[0085] In some preferred embodiments, the adaptive real-time data acquisition system intelligently assigns processing priorities to acquisition tasks based on the importance and urgency of the acquired data and the availability of processing resources, ensuring that high-priority data is processed first, thereby maximizing the satisfaction of real-time requirements.

[0086] Figure 3 is a flow chart of an adaptive real-time data collection method according to an embodiment of the present invention. Figure 3 As shown, the method includes:

[0087] 101. Acquire point data and rule configuration from the target database, and obtain protocol configuration information, acquisition configuration information, and point configuration information matching each target data source;

[0088] 102. Generate a collection task based on the collection configuration information and point configuration information of the target data source;

[0089] 103. Establishing a connection with a target data source based on the protocol configuration information of the target data source;

[0090] 104. Execute the collection task to collect data information from the target data source according to the preset collection strategy;

[0091] 105. Execute the collection task to collect data information of a target data source according to an adaptive collection strategy; the adaptive collection strategy is generated based on a real-time event model, and the real-time event model is generated by the event triggering condition;

[0092] 106. Analyze the data information of the target data source and store it in the target database.

[0093] Through steps 101 to 106 of the embodiment of the present invention, the functions of automatically configuring protocol configuration information and acquisition configuration information are provided, which effectively reduces the complexity of operation and the risk of error. Through the protocol configuration information, the present invention can automatically adapt to target data sources of different types and different data formats. When an emergency occurs, the present invention can respond quickly and adjust the acquisition strategy. The present invention avoids the waste of resources caused by the use of a fixed acquisition frequency when the data changes slightly, and the omission of key information when the data changes significantly.

[0094] In an embodiment of the present invention, after starting the adaptive real-time data acquisition system, it can be connected to the target database so that the adaptive real-time data acquisition system can subscribe to the data information of the target database and synchronize the clock of the target database; based on the subscribed data information of the target database and the synchronized clock of the target database, the protocol configuration information, acquisition configuration information and point configuration information matching each target data source are obtained.

[0095] Exemplarily, the adaptive real-time data acquisition system can send a subscription request to the target database. This request will clearly state the data content, format, time range and other requirements to be obtained. After receiving the subscription request, the target database verifies and processes the subscription request. If the request is legal and meets the conditions, a subscription relationship will be established between the target database and the adaptive real-time data acquisition system. After the subscription relationship is established, the target database will push data to the adaptive real-time data acquisition system in an agreed manner and frequency. The push method can be real-time push, that is, once new data is generated, it is immediately sent to the adaptive real-time data acquisition system; it can also be pushed at fixed time intervals, such as every few minutes or hours.

[0096] In an embodiment of the present invention, the collection configuration information is a series of settings and rules that need to be followed when collecting data from the target data source. It contains many key elements, the first of which is the setting of the collection frequency. Different business scenarios and data requirements determine different collection frequencies. For example, for industrial equipment data with extremely high real-time requirements, high-frequency collection in seconds or even milliseconds may be required in order to capture subtle changes in the market in a timely manner; for some relatively stable and slowly changing data, such as basic information of corporate employees, the collection frequency may be lower, such as once a month or once a quarter.

[0097] In the embodiment of the present invention, the point configuration information is to clarify which data is collected in the target data source and where it is collected. Taking the data source of the sensor network as an example, the point configuration information will specify in detail the sensor number, location and parameters monitored by the sensor for which data needs to be collected. Taking the data source of the database as an example, the point configuration information will determine the table, field and related query conditions that need to be collected.

[0098] Based on the collection configuration information and point configuration information, the collection resources are arranged reasonably, and the execution order and time nodes of the collection tasks are determined to ensure that the collection tasks can be completed efficiently and accurately. The required resource information covers the hardware equipment, network bandwidth, storage capacity and other resources required to perform the collection tasks, ensuring that the task will not fail due to insufficient resources during the execution of the task.

[0099] It should be noted that by generating collection tasks based on the collection configuration information and point configuration information of the target data source, it is possible to achieve precise and standardized data collection, provide a reliable data foundation for subsequent data processing, analysis and application, and strongly support the company's decision-making and business development.

[0100] In the embodiment of the present invention, executing the collection task to collect data information of the target data source according to the preset collection strategy includes:

[0101] Read the current clock;

[0102] Based on the current clock, the acquisition task is allocated to the corresponding execution unit according to the preset acquisition strategy;

[0103] Among them, by reading the current clock, the system can accurately record the time point of data collection, which is crucial to ensure the timeliness and accuracy of data. In some time-sensitive application scenarios, such as financial transaction data collection and real-time monitoring systems, accurate time records can help analysts accurately trace the time when data was generated.

[0104] The execution unit executes the collection task to collect data information from a target data source.

[0105] In an embodiment of the present invention, the preset collection strategy is a set of rules formulated in advance according to business needs and data characteristics, which specifies how to collect data in different situations, such as the frequency, time interval, and priority of data. The system reasonably allocates the collection tasks to the corresponding execution units according to the current clock and these strategies. The execution unit can be a hardware device (such as a sensor, a data acquisition card, etc.) or a software module (such as a program segment specifically responsible for data collection). This clock- and strategy-based task allocation method can ensure the orderly progress of the collection tasks, avoid task conflicts and waste of resources, and can also flexibly adjust the execution method of the collection tasks according to different business needs.

[0106] In the embodiment of the present invention, executing the collection task to collect data information of the target data source according to the adaptive collection strategy includes:

[0107] Read the current clock;

[0108] Based on the current clock, the acquisition task is allocated to the corresponding execution unit according to the adaptive acquisition strategy;

[0109] In an embodiment of the present invention, the adaptive acquisition strategy includes:

[0110] Adaptive strategy, when a delay occurs, the collection task is performed according to the adaptive collection interval; the adaptive collection interval is determined by the system status and business needs;

[0111] Among them, the system status covers multiple aspects of information, including but not limited to the system resource utilization (such as CPU utilization, memory occupancy, network bandwidth utilization, etc.), the current load of the data source, the stability of data transmission, and the degree of competition for resources by other running tasks in the system. For example, if the CPU utilization of the system is too high, it means that the system processing capacity is limited. At this time, in order to avoid the collection task further increasing the burden on the system, the adaptive collection interval may be appropriately extended; on the contrary, if the system resources are relatively abundant and the data source load is low, the adaptive collection interval can be shortened accordingly to obtain data more frequently and improve the real-time nature of the data. Business requirements are also an important basis for determining the adaptive collection interval. Different business scenarios have different requirements for the real-time nature and accuracy of data. In some business scenarios with extremely high real-time requirements, such as financial transaction monitoring, industrial automation control, etc., even if there is a delay, the collection interval needs to be shortened as much as possible to ensure that key data can be captured in time and provide accurate support for decision-making. In some business scenarios with relatively low real-time requirements, such as market trend analysis, historical data statistics, etc., the collection interval can be appropriately relaxed to balance the utilization of system resources and the efficiency of data collection.

[0112] In the preferred embodiment of the present invention, by real-time monitoring of various indicators of the system status and combining with pre-set business demand parameters, the most suitable adaptive collection interval for the current situation is dynamically calculated. During the calculation process, the algorithm will comprehensively consider the influence of multiple factors, and through continuous adjustment and optimization, ensure that the collection interval can always match the system status and business needs.

[0113] like Figure 4 As shown, the burst strategy directly executes the next cycle of collection tasks when a delay occurs until the collection task returns to the normal collection interval;

[0114] For example, in a collection task sequence, such as Figure 4 As shown in the figure, the second task is delayed (marked as "delay"), and then starting from the third task, the task execution speed will be accelerated, and multiple tasks will be executed continuously (the third, fourth, fifth, and sixth tasks are all marked as "work"), in order to make the overall task progress catch up with the normal frequency as soon as possible.

[0115] like Figure 5 As shown, the skip strategy skips the missed collection interval when a delay occurs, and performs the collection task according to the preset collection interval in the next multiple cycle.

[0116] In the embodiment of the present invention, after skipping the missed collection interval, the adaptive real-time data collection system will enter the next key step: perform the collection task in the next multiple period according to the preset collection interval. Among them, the "multiple period" is a concept of a time period determined based on the preset collection interval. It is an integer multiple of the preset collection interval. This multiple is set to ensure that the system can return to the normal collection rhythm at the appropriate time node, and also takes into account that the system needs a certain buffer and adjustment time after processing the delay.

[0117] For example, if the preset collection interval is 5 minutes, when a delay occurs and causes 3 collection intervals (i.e. 15 minutes) to be missed, the system will not start collection immediately after the delay ends, but wait until the next multiple period. If the multiple is set to 2, the system will restart the collection task at the preset 5-minute collection interval at the 10th minute after the delay ends (i.e. 2 times of 5 minutes).

[0118] This embodiment effectively ensures the smooth progress of data collection work and improves the stability and reliability of the system by skipping the missed collection interval and executing the collection task at the next multiple cycle according to the preset collection interval.

[0119] For example, Figure 5 As shown, in the above acquisition task sequence, after the second task is delayed, the second task will not be made up, but will be skipped directly, and then the task will continue to be executed in the next multiple cycle (for example, here it may be a multiple cycle of 2, that is, the fourth task) (the fourth, fifth, and sixth tasks are marked as "work").

[0120] The embodiments of the present invention provide different adaptive acquisition strategies to handle the delay situation occurring during the task execution process, effectively ensuring the smooth progress of data acquisition work and improving the stability and reliability of the system.

[0121] In the embodiment of the present invention, allocating the acquisition task to the corresponding execution unit includes:

[0122] Creating a collection subtask for the collection task;

[0123] The acquisition subtasks are allocated to corresponding execution units according to their priorities; the priorities of the acquisition subtasks are determined by the importance and urgency of the acquisition subtasks and the availability of processing resources.

[0124] The execution unit executes the collection task to collect data information from a target data source.

[0125] The collection method in the embodiment of the present invention can solve the problems existing in traditional data collection tools. Through the built-in domain knowledge base and adaptive learning algorithm, the data structure and business logic of the new domain can be automatically identified and adapted. According to system performance, data importance and environmental changes, the collection strategy can be dynamically adjusted to ensure the real-time and high efficiency of data collection.

[0126] A person skilled in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing the relevant hardware through a program, and the program can be stored in a computer-readable storage medium, which can include: ROM, RAM, disk or CD, etc.

[0127] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. An adaptive real-time data collection method, characterized in that: The method comprises: Acquire point data and rule configuration from the target database, and obtain protocol configuration information, acquisition configuration information, and point configuration information that match each target data source; Generate a collection task based on the collection configuration information and point configuration information of the target data source; Establishing a connection with the target data source based on the protocol configuration information of the target data source; According to the preset collection strategy, the collection task is executed to collect data information of the target data source; According to the adaptive collection strategy, the collection task is executed to collect data information of the target data source; the adaptive collection strategy is generated based on the real-time event model, and the real-time event model is generated by the event triggering condition; Parse the data information of the target data source and store it in the target database.

2. The adaptive real-time data acquisition method according to claim 1, characterized in that: After obtaining point data and rule configuration from the target database, it also includes: Subscribe to the data information of the target database; Synchronizing a clock of the target database; Based on the data information of the subscribed target database and the clock of the synchronized target database, the protocol configuration information, collection configuration information and point configuration information matching each target data source are obtained.

3. The adaptive real-time data acquisition method according to claim 1, characterized in that: The executing the acquisition task to acquire data information of the target data source according to the preset acquisition strategy includes: Read the current clock; Based on the current clock, the acquisition task is allocated to the corresponding execution unit according to the preset acquisition strategy; The execution unit executes the collection task to collect data information from a target data source.

4. The adaptive real-time data acquisition method according to claim 1, characterized in that: According to the adaptive collection strategy, executing the collection task to collect data information of the target data source includes: Read the current clock; Based on the current clock, the acquisition task is allocated to the corresponding execution unit according to the adaptive acquisition strategy; The execution unit executes the collection task to collect data information from a target data source.

5. An adaptive real-time data acquisition method according to claim 3 or 4, characterized in that: Allocating the acquisition task to the corresponding execution unit includes: Creating a collection subtask for the collection task; The acquisition subtasks are allocated to corresponding execution units according to their priorities; the priorities of the acquisition subtasks are determined by the importance and urgency of the acquisition subtasks and the availability of processing resources.

6. The adaptive real-time data acquisition method according to claim 1 or 4, characterized in that: The adaptive acquisition strategy includes: Adaptive strategy, when a delay occurs, the collection task is performed according to the adaptive collection interval; the adaptive collection interval is determined by the system status and business needs; Burst strategy: when a delay occurs, the next cycle of collection tasks is directly executed until the collection task returns to the normal collection interval; The skip strategy skips the missed collection interval when a delay occurs, and performs the collection task at the preset collection interval in the next multiple cycle.

7. The adaptive real-time data acquisition method according to claim 1, characterized in that: Parsing the data information of the target data source and storing it in the target database includes: Analyze the data information of the target data source and build a unified data model; The unified data model is stored in the target database through data archiving.

8. An adaptive real-time data acquisition system, characterized in that: The system comprises: An access control module, used to control access to a target data source and a target database; A protocol adapter module, configured to connect to the target data source based on the protocol configuration information of the target data source; The data analysis module is used to analyze and build a unified data model based on the data information of the target data source; An event model module, for generating a real-time event model in response to an event triggering condition; An adaptive strategy module, used for generating an adaptive acquisition strategy based on the real-time event model; The collection scheduling module is used to generate a collection task based on the collection configuration information and point configuration information of the target data source; and to execute the collection task to collect data information of the target data source based on a preset collection strategy and / or an adaptive strategy.

9. The adaptive real-time data acquisition system according to claim 8, characterized in that: The adaptive strategy module includes: An adaptive strategy unit is used to execute the collection task according to the adaptive collection interval when a delay occurs; the adaptive collection interval is determined by the system state and business requirements; The burst strategy unit is used to execute the collection task according to the burst collection interval when a delay occurs; The skip strategy unit is used to skip the missed collection interval when a delay occurs, and perform the collection task according to the preset collection interval in the next multiple cycle.

Images