A real-time data stream intelligent scheduling and dynamic loop push method and system

Abstract

The present method relates to the technical field of real-time data processing and push of time series databases. The present invention discloses a method and system for intelligent scheduling and dynamic loopback push of real-time data streams, including setting a push topic and configuring measuring points according to specific business needs; integrating the measuring point requirements of push tasks, implementing unified query management, and centrally managing the query and push tasks of the measuring points, and intelligently scheduling according to priority and data change frequency; when network anomalies or data loss occur, the data sequence is resent through the full resend function, and a cache mechanism is introduced to reduce direct queries to the time series library and directly obtain data by caching commonly used measuring point data; supporting dynamic adjustment of push tasks, measuring points and query strategies, and providing API interfaces and configuration options for integration and data exchange with external systems. The present method can adapt to various complex business scenarios and provide users with more efficient and reliable real-time data push services.

Description

Technical Field

[0001] The present invention relates to the technical field of real-time data processing and push of time series databases, and in particular to a method and system for intelligent scheduling and dynamic loopback push of real-time data streams. Background Art

[0002] With the rapid development of the Internet of Things and the Industrial Internet, time series data, as an important information resource, is crucial for business decision-making and monitoring in terms of timeliness, accuracy, and reliability. In particular, in the fields of industrial automation, intelligent monitoring, and remote management, the push function of real-time data has become an indispensable part. In order to meet the urgent needs of these fields for real-time data processing, we propose a time series library real-time data push technology based on the MQTT (Message Queue Telemetry Transport) protocol.

[0003] Currently, although there are a variety of data push technologies, they often face some challenges when processing time series data. First, real-time performance is a key issue. Traditional polling or long polling methods will cause high latency when the data volume is large or the network conditions are poor, and cannot meet the real-time requirements. Secondly, security and privacy protection are also aspects that cannot be ignored. During the data transmission process, it is necessary to ensure that the data is not illegally obtained or tampered with to ensure the security and reliability of the data.

[0004] To overcome these challenges, we proposed a real-time data push technology for time series libraries based on the MQTT protocol. As a lightweight publish / subscribe message transmission protocol, MQTT has the characteristics of small messages, low overhead, support for multiple QoS (quality of service) levels and publish / subscribe modes, and is very suitable for real-time data push. By optimizing the implementation method and data transmission mechanism of the MQTT protocol, we can ensure that time series data can be pushed to the target end in real time and accurately, while introducing advanced data encryption and authentication mechanisms to ensure security and privacy protection during data transmission.

[0005] The development of this technology will greatly improve the real-time and reliability of time series data, and meet the urgent needs for real-time data processing and push in fields such as the Internet of Things and Industrial Internet. We believe that with the continuous advancement of technology and the continuous expansion of application scenarios, this technology will be widely used and promoted in more fields. Summary of the invention

[0006] In view of the above existing problems, the present invention is proposed.

[0007] Therefore, the present invention provides a real-time data stream intelligent scheduling and dynamic loopback push method, which can adapt to various complex business scenarios and provide users with more efficient and reliable real-time data push services.

[0008] To solve the above technical problems, the present invention provides the following technical solutions: a real-time data stream intelligent scheduling and dynamic loop push method, comprising: setting a push topic and configuring a measuring point according to specific business needs, and the user can create a push instance, define a push topic, and intuitively operate the measuring point through the provided graphical management interface; integrating the measuring point requirements of the push task, implementing unified query management, and centrally managing the query and push tasks of the measuring point, and intelligently scheduling according to the priority and data change frequency; when a network anomaly or data loss occurs, the data sequence is resent through the full resend function, and a cache mechanism is introduced to cache the commonly used measuring point data to reduce the direct query of the time series library and directly obtain data; supporting dynamic adjustment of push tasks, measuring points and query strategies, and providing API interfaces and configuration options for integration with external systems and data exchange.

[0009] As a preferred solution of the method for intelligent scheduling and dynamic loopback push of real-time data streams described in the present invention, the configuration of the measuring points includes starting a time series database management system and logging into the system using user credentials;

[0010] Set the unique identifier of the measurement point and provide a detailed description of the measurement point, select the measurement point data type; specify the measurement point data from the device ID, sensor address or API endpoint, set the measurement point data to be sampled at equal intervals, transfer it to the time series library for storage and follow the set storage strategy;

[0011] The storage policy includes data retention rules and data compression rules. The data compression rules are expressed as:

[0012] ,

[0013] ,

[0014] in, represents the compressed data set, represents the number of data points in the original dataset, represents the i-th data point in the original data set, represents the compression threshold of the i-th data point, represents the standard deviation of the ith data point, represents the compressive strength parameter, represents the mean of the data set, represents the compression index of the i-th data point;

[0015] After data compression, select from the existing push topic list or create a new push topic, map the measurement point to the selected push topic, and after confirming that the settings are correct, save the measurement point configuration and apply it to the time series database.

[0016] As a preferred solution of the real-time data stream intelligent scheduling and dynamic loop push method described in the present invention, the unified query management includes analyzing the requirements of the push task, determining the common measurement point data, clustering the measurement point requirements, and classifying the measurement points with similar requirements into one category.

[0017] ;

[0018] ;

[0019] ;

[0020] in, represents the integrated demand clustering quality function, Represents the set of measurement point requirements, Represents a push task collection. represents the satisfaction of the requirement of the lth task, represents the data availability of measurement point s, represents the demand sensitivity coefficient of the lth task, represents the resource consumption function of the lth task, represents the consumption of resource j by the lth task, Indicates the number of measurement points, Indicates the number of push tasks. represents a very small positive number, represents the normalization parameter, represents the availability of resource j;

[0021] According to the clustering results, the measurement point demand matrix is ​​constructed and the matrix dimension is defined as And initialize the matrix, for each task l and each measurement point s, calculate and , the calculated values ​​fill the matrix M:

[0022] ,

[0023] get:

[0024] ,

[0025] in, It represents the ratio of the demand satisfaction of the lth task to the sth measurement point and the resource consumption, 1≤l≤n, 1≤s≤m, represents the demand satisfaction of the measuring point s, represents the resource consumption function of the measuring point s, It means getting the element in the mth column of the nth row in the matrix.

[0026] As a preferred solution of the real-time data stream intelligent scheduling and dynamic loop push method described in the present invention, the intelligent scheduling includes: analyzing the measurement point demand matrix, identifying the measurement points commonly required by the tasks, creating new query tasks for the measurement points commonly required, integrating the requirements of the corresponding tasks, determining the priority of the merged query tasks, assigning a priority weight to each task according to the importance and urgency of the task, and obtaining the frequency of changes in the evaluation data of each measurement point through historical data analysis;

[0027] Construct an n×1 task priority matrix P:

[0028] ,

[0029] in, Indicates the priority weight of the task;

[0030] Construct an m×1 data change frequency vector F:

[0031] ,

[0032] in, Indicates the average data change frequency of task-related measurement points;

[0033] When regular tasks appear, a comprehensive scheduling based on priority and data change frequency is adopted. For each task l, the comprehensive priority is calculated :

[0034] ,

[0035] In the formula, represents the priority weight of task l, represents the average data change frequency of the measurement points s related to task l;

[0036] according to Sort the tasks, schedule the tasks with higher priority first, and when task l has not been scheduled for a long time, dynamically increase the priority weight and set the time threshold , when the waiting time of task l exceeds , then according to the proportion Increase:

[0037] ,

[0038] in represents the waiting time of task l;

[0039] When resource competing tasks appear, compare the comprehensive priorities of competing tasks , priority scheduling High tasks, If they are the same, the task will be scheduled based on its historical resource usage.

[0040] When an urgent task occurs, immediately increase the priority weight of the urgent task. Set to the highest and recalculate the combined priority of all tasks .

[0041] As a preferred solution of the method for intelligent scheduling and dynamic loopback push of real-time data streams described in the present invention, the full retransmission function includes assigning a unique identifier to each data sequence, monitoring the network status in real time during the data transmission process through the system log, and detecting network anomalies;

[0042] When network anomalies occur, data loss is detected through the receiver confirmation and timeout mechanism. At the data receiving end, if the received data sequence is incomplete or does not meet expectations, it is judged as data loss; if the sent data does not receive confirmation within the predetermined time, it is judged as data loss;

[0043] Once a network anomaly or data loss is detected, the system immediately triggers the full retransmission mechanism. When full retransmission is triggered, the system resends the entire data sequence. After resending the data sequence, the sender waits for the receiver to confirm through an ACK message or data integrity check. After the network returns to normal, the system automatically or manually checks whether there is an unconfirmed data sequence and performs necessary retransmission operations.

[0044] As a preferred solution of the method for intelligent scheduling and dynamic loopback push of real-time data streams described in the present invention, the cache mechanism includes setting a cache layer to store frequently queried measurement point data, and the data in the cache is synchronously updated according to the update frequency of the time series library;

[0045] When a push task needs to query a measurement point, the system first checks the cache layer. If the required data exists in the cache layer, the system directly obtains it from the cache without accessing the time series library. If the required data does not exist in the cache layer, the system accesses the time series library to obtain the data and synchronizes it to the cache layer.

[0046] When the cache reaches its maximum capacity, old data is eliminated according to the elimination strategy to make room for new data:

[0047] ;

[0048] in, represents the elimination strategy function, represents the time decay constant, Indicates the residence time of data in the cache. Indicates the access frequency of data. Indicates the frequency of data modification. represents the adjustment constant for modifying the frequency;

[0049] The data synchronization mechanism of the cache layer is consistent with the data in the time series library. For real-time updated data, the system adopts a real-time synchronization mechanism. The system monitors the performance indicators of the cache layer, including hit rate and response time, and adjusts and optimizes the cache mechanism based on the monitoring results.

[0050] As a preferred solution of the real-time data stream intelligent scheduling and dynamic loopback push method described in the present invention, the dynamic adjustment includes dynamically adding and deleting push tasks, measurement points and query strategies to meet the needs of different users and changes in business scenarios. Users customize the priority and push frequency parameters of push tasks as needed to meet specific business needs.

[0051] Allow users to adjust the priority of tasks and the execution interval of tasks according to actual conditions; adjust the parameters of cache strategies according to the needs of cache strategies; adjust the parameters of data synchronization strategies according to the needs of data synchronization;

[0052] Authentication and authorization are performed on the API. Only authorized users can access and modify data. Version control is performed on the API to support the addition of new functions and the improvement of old functions. Configuration files are configured in a standard data format to store configuration information of push tasks, measurement points, and query strategies.

[0053] When the system starts, it automatically loads the configuration file and adjusts the system behavior according to the configuration information, allowing users to manually load the configuration file and modify the system behavior;

[0054] In terms of the client subscription mode, the user software sends a subscription message to the control topic, specifies the destination topic for push and the required list of measuring points. When the push instance receives the subscription message, it will push the measuring point value to the user in real time when the corresponding measuring point value changes. At the same time, the user can send an unsubscribe message at any time. On the management side, the user sets the subscription topic through the graphical management tool, and specifies the destination topic for push and the subscribed measuring point list. The push instance automatically sends real-time data when the measuring point changes according to the configuration.

[0055] Another object of the present invention is to provide a real-time data stream intelligent scheduling and dynamic loop push system, which can be applied to complex environments that require rapid response and dynamic adjustment, and can significantly improve data processing capabilities and service quality.

[0056] As a preferred solution of the real-time data stream intelligent scheduling and dynamic loop push system described in the present invention, it includes: data acquisition and preprocessing module, intelligent scheduling and optimization module, dynamic loop push module, system monitoring and management module, user interaction and API module;

[0057] The data acquisition and preprocessing module is responsible for the acquisition and preprocessing of real-time data;

[0058] The intelligent scheduling and optimization module intelligently schedules data processing tasks and optimizes system performance according to real-time data and system status;

[0059] The dynamic loopback push module dynamically pushes data to the target system or user according to the data processing results and system status;

[0060] The system monitoring and management module monitors the system operation status, manages system resources, and ensures stable operation of the system;

[0061] The user interaction and API module provides an interface for users to interact with the system, and an interface for external systems to interact with the system.

[0062] Beneficial effects of the present invention: Through intelligent scheduling and optimization modules, system resources can be reasonably allocated according to the frequency of data changes and the priority of tasks, thereby improving the efficiency of data processing and push. The full retransmission function and cache mechanism can ensure that the system can recover and continue to operate stably in the event of network anomalies or data loss, ensuring the integrity and accuracy of the data. The system supports dynamic adjustment of push tasks, measurement points and query strategies, can adapt to different business needs and environmental changes, and is easy to expand and maintain.

[0063] The authentication and authorization mechanism of API ensures that only authorized users can access and modify data, which enhances data security. The graphical management interface and automatic loading function of configuration files simplify the management and maintenance of the system and reduce management costs. It is suitable for application scenarios that require real-time monitoring and rapid response, such as the Internet of Things, financial data analysis, and real-time monitoring systems. By providing an API interface, it can be easily integrated with external systems, promoting data exchange and resource sharing between different systems. This method is not only applicable to various types of data streams, but also can be customized according to different business rules and requirements, and has strong adaptability. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for describing the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work. Among them:

[0065] Figure 1 A flowchart of a method for intelligent scheduling and dynamic loopback push of real-time data streams provided by an embodiment of the present invention.

[0066] Figure 2 A schematic diagram of working modules of a real-time data stream intelligent scheduling and dynamic loopback push system provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0067] In order to make the above-mentioned purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are described in detail below in conjunction with the drawings of the specification. Obviously, the described embodiments are part of the embodiments of the present invention, but not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary persons in the art without creative work should fall within the scope of protection of the present invention.

[0068] In the following description, many specific details are set forth to facilitate a full understanding of the present invention, but the present invention may also be implemented in other ways different from those described herein, and those skilled in the art may make similar generalizations without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0069] Secondly, the term "one embodiment" or "embodiment" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The term "in one embodiment" that appears in different places in this specification does not necessarily refer to the same embodiment, nor does it refer to a separate or selective embodiment that is mutually exclusive with other embodiments.

[0070] The present invention is described in detail with reference to schematic diagrams. When describing the embodiments of the present invention, for the sake of convenience, the cross-sectional diagrams showing the device structure will not be partially enlarged according to the general scale, and the schematic diagrams are only examples, which should not limit the scope of protection of the present invention. In addition, in actual production, the three-dimensional dimensions of length, width and depth should be included.

[0071] At the same time, in the description of the present invention, it should be noted that the directions or positional relationships indicated by the terms "upper, lower, inner and outer" are based on the directions or positional relationships shown in the drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific direction, be constructed and operated in a specific direction, and therefore cannot be understood as limiting the present invention. In addition, the terms "first, second or third" are only used for descriptive purposes and cannot be understood as indicating or implying relative importance.

[0072] In the present invention, unless otherwise clearly specified and limited, the terms "install, connect, connect" should be understood in a broad sense, for example: it can be a fixed connection, a detachable connection or an integral connection; it can also be a mechanical connection, an electrical connection or a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0073] Example 1, reference Figure 1 , which is the first embodiment of the present invention, provides a real-time data stream intelligent scheduling and dynamic loopback push method, including:

[0074] S1: Set push topics and configure measurement points according to specific business needs. Users can create push instances, define push topics, and intuitively operate measurement points through the provided graphical management interface.

[0075] Furthermore, the configuration of the measuring points includes starting a time series database management system and logging into the system using user credentials;

[0076] Set the unique identifier of the measurement point and provide a detailed description of the measurement point. Select the measurement point data type, including integer, floating point, and string. Specify the measurement point data from the device ID, sensor address, or API endpoint. Set the measurement point data to be sampled at equal intervals, once every 10 minutes, and transfer it to the time series library for storage and follow the set storage strategy.

[0077] The storage policy includes data retention rules and data compression rules. It defines that the data is retained in the time series library for 30 days and is automatically cleared when the data reaches the retention period. The data compression rule is expressed as:

[0078] ;

[0079] ;

[0080] in, represents the compressed data set, represents the number of data points in the original dataset, represents the i-th data point in the original data set, represents the compression threshold of the i-th data point, represents the standard deviation of the ith data point, represents the compressive strength parameter, represents the mean of the data set, represents the compression index of the i-th data point;

[0081] After data compression, select from the existing push topic list or create a new push topic, map the measurement point to the selected push topic, and after confirming that the settings are correct, save the measurement point configuration and apply it to the time series database.

[0082] S2: Integrate the measurement point requirements of the push tasks and implement unified query management. At the same time, centrally manage the query and push tasks of the measurement points, and intelligently schedule them according to priority and frequency of data changes.

[0083] Furthermore, the unified query management includes analyzing the requirements of the push tasks, determining the common measurement point data, performing cluster analysis on the measurement point requirements, and grouping the measurement points with similar requirements into one category.

[0084] ;

[0085] ;

[0086] ;

[0087] in, represents the integrated demand clustering quality function, Represents the set of measurement point requirements, Represents a push task collection. represents the satisfaction of the requirement of the lth task, represents the data availability of measurement point s, represents the demand sensitivity coefficient of the lth task, represents the resource consumption function of the lth task, represents the consumption of resource j by the lth task, Indicates the number of measurement points, Indicates the number of push tasks. represents a very small positive number, represents the normalization parameter, represents the availability of resource j;

[0088] According to the clustering results, the measurement point demand matrix is ​​constructed and the matrix dimension is defined as And initialize the matrix, for each task l and each measurement point s, calculate and , the calculated values ​​fill the matrix M:

[0089] ,

[0090] get:

[0091] ,

[0092] in, It represents the ratio of the demand satisfaction of the lth task to the sth measurement point and the resource consumption, 1≤l≤n, 1≤s≤m, represents the demand satisfaction of the measuring point s, represents the resource consumption function of the measuring point s, It means getting the element in the mth column of the nth row in the matrix.

[0093] It should be noted that the intelligent scheduling includes analyzing the measurement point demand matrix, identifying the measurement points commonly required by multiple tasks, creating a new query task for the measurement points commonly required, integrating the requirements of all related tasks, determining the priority of the merged query task, assigning a priority weight to each task according to the importance and urgency of the task, and obtaining the frequency of changes in the evaluation data of each measurement point through historical data analysis;

[0094] Construct an n×1 task priority matrix P:

[0095] ,

[0096] in, Indicates the priority weight of the task;

[0097] Construct an m×1 data change frequency vector F:

[0098] ,

[0099] in, Indicates the average data change frequency of task-related measurement points;

[0100] When regular tasks appear, a comprehensive scheduling based on priority and data change frequency is adopted. For each task l, the comprehensive priority is calculated :

[0101] ,

[0102] In the formula, represents the priority weight of task l, represents the average data change frequency of the measurement points s related to task l;

[0103] according to Sort the tasks, schedule the tasks with higher priority first, and when task l has not been scheduled for a long time, dynamically increase the priority weight and set the time threshold , when the waiting time of task l exceeds , then according to the proportion Increase:

[0104] ,

[0105] in represents the waiting time of task l;

[0106] When resource competing tasks appear, compare the comprehensive priorities of competing tasks , priority scheduling High tasks, If they are the same, the task will be scheduled based on its historical resource usage.

[0107] When an urgent task occurs, immediately increase the priority weight of the urgent task. Set to the highest and recalculate the combined priority of all tasks .

[0108] S3: When network anomalies or data loss occur, the data sequence is resent through the full resend function. A cache mechanism is introduced to cache the data of commonly used measurement points, reduce direct queries to the time series library, and directly obtain data.

[0109] Furthermore, the full retransmission function includes assigning a unique identifier to each data sequence, monitoring the network status in real time during data transmission through system logs, and detecting network anomalies;

[0110] When network anomalies occur, data loss is detected through the receiver confirmation and timeout mechanism. At the data receiving end, if the received data sequence is incomplete or does not meet expectations, it is judged as data loss; if the sent data does not receive confirmation within the predetermined time, it is judged as data loss;

[0111] Once a network anomaly or data loss is detected, the system immediately triggers the full retransmission mechanism. When full retransmission is triggered, the system resends the entire data sequence. After resending the data sequence, the sender waits for the receiver to confirm through an ACK message or data integrity check. After the network returns to normal, the system automatically or manually checks whether there is an unconfirmed data sequence and performs necessary retransmission operations.

[0112] It should be noted that the cache mechanism includes setting a cache layer to store frequently queried measurement point data, and the data in the cache is synchronously updated according to the update frequency of the time series library;

[0113] When a push task needs to query a measurement point, the system first checks the cache layer. If the required data exists in the cache layer, the system directly obtains it from the cache without accessing the time series library. If the required data does not exist in the cache layer, the system accesses the time series library to obtain the data and synchronizes it to the cache layer.

[0114] When the cache reaches its maximum capacity, old data is eliminated according to the elimination strategy to make room for new data:

[0115] ;

[0116] in, represents the elimination strategy function, represents the time decay constant, Indicates the residence time of data in the cache. Indicates the access frequency of data. Indicates the frequency of data modification. represents the adjustment constant for modifying the frequency;

[0117] The data synchronization mechanism of the cache layer is consistent with the data in the time series library. For real-time updated data, the system adopts a real-time synchronization mechanism. The system monitors the performance indicators of the cache layer, including hit rate and response time, and adjusts and optimizes the cache mechanism based on the monitoring results.

[0118] S4: Supports dynamic adjustment of push tasks, measurement points, and query strategies, and provides API interfaces and configuration options for integration and data exchange with external systems.

[0119] Furthermore, the dynamic adjustment includes dynamically adding and deleting push tasks, measurement points, and query strategies to meet the needs of different users and changes in business scenarios. Users can customize the priority and push frequency parameters of push tasks as needed to meet specific business needs.

[0120] Allow users to adjust the priority of tasks and the execution interval of tasks according to actual conditions; adjust the parameters of cache strategies according to the needs of cache strategies; adjust the parameters of data synchronization strategies according to the needs of data synchronization;

[0121] Authentication and authorization are performed on the API. Only authorized users can access and modify data. Version control is performed on the API to support the addition of new functions and the improvement of old functions. Configuration files are configured in a standard data format to store configuration information of push tasks, measurement points, and query strategies.

[0122] When the system starts, it automatically loads the configuration file and adjusts the system behavior according to the configuration information, allowing users to manually load the configuration file and modify the system behavior;

[0123] In terms of the client subscription mode, the user software sends a subscription message to the control topic, specifies the destination topic for push and the required list of measuring points. When the push instance receives the subscription message, it will push the measuring point value to the user in real time when the corresponding measuring point value changes. At the same time, the user can send an unsubscribe message at any time. On the management side, the user sets the subscription topic through the graphical management tool, and specifies the destination topic for push and the subscribed measuring point list. The push instance automatically sends real-time data when the measuring point changes according to the configuration.

[0124] Example 2, an embodiment of the present invention, provides a method for intelligent scheduling and dynamic loopback push of real-time data streams. In order to verify the beneficial effects of the present invention, scientific demonstration is carried out through experiments.

[0125] A certain factory encountered a performance bottleneck when using the traditional push method. The number of single-task measurement points is 20,000. When the number of push tasks gradually increases, especially when the number of tasks reaches 12 and the total number of measurement points reaches 240,000, the system response time will increase significantly, and the average response time will exceed 1 second. This seriously affects the real-time performance and efficiency of the push. In order to solve the performance problems of the traditional push method, a new push method was introduced. Even if the number of tasks increases to 20 and the total number of measurement points reaches 400,000, the new method can still maintain the performance of real-time push in seconds, with a response time of less than 500 milliseconds. This new method effectively reduces the system response time and improves the efficiency and real-time performance of the push by integrating multiple push task measurement points, uniformly managing measurement point queries and push tasks, and introducing cache mechanisms.

[0126] Table 1 Comparison of push task quantity and response time

[0127] ,

[0128] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the present invention, which should all be included in the scope of the claims of the present invention.

[0129] Embodiment 3, the third embodiment of the present invention, is different from the first two embodiments in that:

[0130] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc., which can store program codes.

[0131] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in conjunction with such instruction execution systems, devices or apparatuses. For the purposes of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in conjunction with such instruction execution systems, devices or apparatuses.

[0132] More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or more wires (electronic device), a portable computer disk case (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be a paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, deciphering or, if necessary, processing in another suitable manner, and then stored in a computer memory.

[0133] It should be understood that the various parts of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0134] Example 4, reference Figure 2 , which is an embodiment of the present invention, provides a real-time data stream intelligent scheduling and dynamic loop push system, which is characterized by: including a data acquisition and preprocessing module, an intelligent scheduling and optimization module, a dynamic loop push module, a system monitoring and management module, and a user interaction and API module.

[0135] The data acquisition and preprocessing module is responsible for the acquisition and preprocessing of real-time data.

[0136] The intelligent scheduling and optimization module intelligently schedules data processing tasks and optimizes system performance based on real-time data and system status.

[0137] The dynamic loopback push module dynamically pushes data to the target system or user based on the data processing results and system status.

[0138] The system monitoring and management module monitors the system operation status, manages system resources, and ensures stable system operation.

[0139] The user interaction and API module provides an interface for users to interact with the system, as well as an interface for external systems to interact with the system.

[0140] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the present invention, which should all be included in the scope of the claims of the present invention.

Claims

1. A real-time data stream intelligent scheduling and dynamic loopback push method, characterized in that: include, According to specific business needs, set the push topic and configure the measurement points. Users can create push instances, define push topics, and intuitively operate measurement points through the provided graphical management interface; Integrate the measurement point requirements of push tasks, implement unified query management, centrally manage the query and push tasks of measurement points, and intelligently schedule them according to priority and data change frequency; The unified query management includes analyzing the requirements of the push tasks, determining the common measurement point data, performing cluster analysis on the measurement point requirements, and grouping the measurement points with similar requirements into one category, as follows: Among them, Q(S,T) represents the integrated demand clustering quality function, S represents the set of measurement point demands, T represents the set of push tasks, G l (S,T) represents the satisfaction of the requirement of the lth task, g l (s) represents the data availability of measurement point s, k l represents the demand sensitivity coefficient of the lth task, H l (S,T) represents the resource consumption function of the lth task, h lj represents the consumption of resource j by the lth task, m represents the number of measurement points, n represents the number of pushed tasks, ∈ represents a very small positive number, σ represents a normalization parameter, and p lj represents the availability of resource j; According to the clustering results, the measurement point demand matrix is ​​constructed and the matrix dimension is defined as And initialize the matrix, for each task l and each measurement point s, calculate G l and H l , the calculated values ​​fill the matrix M: get: Among them, M l,s represents the ratio of the demand satisfaction of the lth task to the sth measurement point and the resource consumption, 1≤l≤n, 1≤s≤m, G l (s) represents the demand satisfaction of the measuring point s, H l (s) represents the resource consumption function of the measuring point s, M n,m It means that the element in the mth column of the nth row in the matrix is ​​obtained; The intelligent scheduling includes analyzing the measurement point demand matrix, identifying the measurement points commonly required by the tasks, creating new query tasks for the measurement points commonly required, integrating the requirements of all corresponding tasks, determining the priority of the merged query tasks, assigning a priority weight to each task according to the importance and urgency of the task, and obtaining the frequency of changes in the evaluation data of each measurement point through historical data analysis; Construct an n×1 task priority matrix P: Among them, w1…w n Indicates the priority weight of the task; Construct an m×1 data change frequency vector F: Among them, f1…f m Indicates the average data change frequency of task-related measurement points; When regular tasks appear, a comprehensive scheduling based on priority and data change frequency is adopted. For each task l, the comprehensive priority Π is calculated. l : P l =w l ·f s In the formula, w l represents the priority weight of task l, f s represents the average data change frequency of the measurement points s related to task l; According to l Sort the tasks, schedule the tasks with higher priority first, and when task l has not been scheduled for a long time, dynamically increase the priority weight and set the time threshold T th , when the waiting time of task l exceeds T th , then it increases according to the proportion α: w l =w l +α·(T’-T th ) Where T' represents the waiting time of task l; When resource competition tasks occur, compare the comprehensive priorities of the competing tasks. l , priority scheduling Π l High mission, l If they are the same, the task will be scheduled based on its historical resource usage. When an urgent task occurs, immediately increase the priority weight of the urgent task to w l Set to the highest and recalculate the comprehensive priority of all tasks Π l ; When network anomalies or data loss occur, the data sequence is resent through the full resend function. The cache mechanism is introduced to cache the data of commonly used measurement points, reduce direct queries to the time series library, and directly obtain data; The cache mechanism includes setting a cache layer to store frequently queried measurement point data, and the data in the cache is synchronously updated according to the update frequency of the time series library; When a push task needs to query a measurement point, the system first checks the cache layer. If the required data exists in the cache layer, the system directly obtains it from the cache without accessing the time series library. If the required data does not exist in the cache layer, the system accesses the time series library to obtain the data and synchronizes it to the cache layer. When the cache reaches its maximum capacity, old data is eliminated according to the elimination strategy to make room for new data: Where F' represents the elimination policy function, R represents the time decay constant, x represents the residence time of data in the cache, y represents the access frequency of data, z represents the modification frequency of data, and q represents the adjustment constant of the modification frequency; The data synchronization mechanism of the cache layer is consistent with the data in the time series database. For real-time updated data, the system uses a real-time synchronization mechanism. The system monitors the performance indicators of the cache layer, including hit rate and response time, and adjusts and optimizes the cache mechanism based on the monitoring results. It supports dynamic adjustment of push tasks, measurement points and query strategies, and provides API interfaces and configuration options for integration and data exchange with external systems.

2. A real-time data stream intelligent scheduling and dynamic loopback push method as claimed in claim 1, characterized in that: The configuration of the measuring points includes starting the time series database management system and logging into the system using user credentials; Set the unique identifier of the measurement point and provide a detailed description of the measurement point, select the measurement point data type; specify the measurement point data from the device ID, sensor address or API endpoint, set the measurement point data to be sampled at equal intervals, transfer it to the time series library for storage and follow the set storage strategy; The storage policy includes data retention rules and data compression rules. The data compression rules are expressed as: Among them, D c represents the compressed data set, k represents the number of data points in the original data set, and D o,i represents the i-th data point in the original data set, H c,i represents the compression threshold of the i-th data point, σ(D o,i ) represents the standard deviation of the i-th data point, λ represents the compression strength parameter, μ represents the mean of the data set, and p i represents the compression index of the i-th data point; After data compression, select from the existing push topic list or create a new push topic, map the measurement point to the selected push topic, and after confirming that the settings are correct, save the measurement point configuration and apply it to the time series database.

3. A real-time data stream intelligent scheduling and dynamic loopback push method as claimed in claim 2, characterized in that: The full retransmission function includes assigning a unique identifier to each data sequence, monitoring the network status in real time during data transmission through system logs, and detecting network anomalies; When network anomalies occur, data loss is detected through the receiver confirmation and timeout mechanism. At the data receiving end, if the received data sequence is incomplete or does not meet expectations, it is judged as data loss; if the sent data does not receive confirmation within the predetermined time, it is judged as data loss; Once a network anomaly or data loss is detected, the system immediately triggers the full retransmission mechanism. When full retransmission is triggered, the system resends the entire data sequence. After resending the data sequence, the sender waits for the receiver to confirm through an ACK message or data integrity check. After the network returns to normal, the system automatically or manually checks whether there is an unconfirmed data sequence and performs necessary retransmission operations.

4. A method for intelligent scheduling and dynamic loopback push of real-time data streams as claimed in claim 3, characterized in that: The dynamic adjustment includes dynamically adding and deleting push tasks, measurement points, and query strategies to meet the needs of different users and changes in business scenarios. Users can customize the priority and push frequency parameters of push tasks as needed to meet business needs. Allow users to adjust the priority of tasks and the execution interval of tasks according to actual conditions; adjust the parameters of cache strategies according to the needs of cache strategies; adjust the parameters of data synchronization strategies according to the needs of data synchronization; Authentication and authorization are performed on the API. Only authorized users can access and modify data. Version control is performed on the API to support the addition of new functions and the improvement of old functions. Configuration files are configured in a standard data format to store configuration information of push tasks, measurement points, and query strategies. When the system starts, it automatically loads the configuration file and adjusts the system behavior according to the configuration information, allowing users to manually load the configuration file and modify the system behavior; In terms of the client subscription mode, the user software sends a subscription message to the control topic, specifies the destination topic for push and the required list of measuring points. When the push instance receives the subscription message, it will push the measuring point value to the user in real time when the corresponding measuring point value changes. At the same time, the user can send an unsubscribe message at any time. On the management side, the user sets the subscription topic through the graphical management tool, and specifies the destination topic for push and the subscribed measuring point list. The push instance automatically sends real-time data when the measuring point changes according to the configuration.

5. A system using a real-time data stream intelligent scheduling and dynamic loopback push method as claimed in any one of claims 1 to 4, characterized in that: It includes data collection and preprocessing module, intelligent scheduling and optimization module, dynamic loop push module, system monitoring and management module, user interaction and API module; The data acquisition and preprocessing module is responsible for the acquisition and preprocessing of real-time data; The intelligent scheduling and optimization module intelligently schedules data processing tasks and optimizes system performance according to real-time data and system status; The dynamic loopback push module dynamically pushes data to the target system or user according to the data processing results and system status; The system monitoring and management module monitors the system operation status, manages system resources, and ensures stable operation of the system; The user interaction and API module provides an interface for users to interact with the system, and an interface for external systems to interact with the system.

6. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, the steps of a real-time data stream intelligent scheduling and dynamic loop push method described in any one of claims 1 to 4 are implemented.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of a real-time data stream intelligent scheduling and dynamic loopback push method described in any one of claims 1 to 4 are implemented.

Images