Containerized SCADA (supervisory control and data acquisition) data processing elastic scaling method and system based on dynamic perception

Abstract

The invention discloses a containerized SCADA (supervisory control and data acquisition) data processing elastic scaling method and system based on dynamic perception. The method comprises the following steps: carrying out standardized container mirror image packaging on an SCADA data processing unit; after the container is started, multi-dimensional indexes of the container, a host machine, a service queue and a message bus are collected in real time and written into a real-time library and a time sequence library; dynamically updating a channel routing table through secondary routing mapping according to index changes, and completing binding of the FES collection container and the SCADA data processing container; normalizing the indexes, inputting the normalized indexes into a load evaluation model, obtaining a unified load score and trend prediction, and outputting an elastic scaling decision; carrying out capacity expansion or capacity reduction on the container according to a decision to form a new container set matched with the current load; real-time indexes of a new container are reinjected into the monitoring system, the load evaluation model is iteratively optimized, and elastic expansion and contraction of the container are continuously completed. Compared with the prior art, the method has the advantages that on-demand dynamic allocation of resources is realized in the SCADA processing system, and the method has elasticity and self-adaptive capability.

Description

Technical Field

[0001] This invention belongs to the field of power system automation and cloud computing technology, and in particular relates to a containerized SCADA data processing elastic scaling method and system based on dynamic perception. Background Technology

[0002] As the energy transition deepens, the penetration rate of distributed new energy sources such as wind power and photovoltaics in the power grid has significantly increased. However, the output of these energy sources is highly random, intermittent, and volatile, resulting in massive, high-frequency, and highly sudden measurement data, posing a severe challenge to the real-time data acquisition and processing capabilities of the power system.

[0003] To address these challenges, cloud-edge converged intelligent scheduling and operation platforms have become a core development direction in the industry. This platform achieves precise perception and intelligent control of widely distributed new energy sources through localized processing at the edge and global collaborative scheduling in the cloud. Under this architecture, the containerized SCADA processing system located at the main station undertakes the critical responsibilities.

[0004] The containerized SCADA system architecture commonly used in existing technologies has inherent structural flaws: the data acquisition unit (FES container) and the data processing unit (SCADA data processing container) are hard-coded, statically bound one-to-one, meaning that data acquired by one FES is consistently sent to a specific SCADA data processing container. Limited by hardware I / O and communication protocols, a single FES container typically supports only 1000 data channels. Faced with the explosive growth of new energy data, maintenance personnel must horizontally scale the FES containers. Due to this strong coupling, the SCADA data processing containers are forced to passively scale up synchronously at a 1:1 ratio. This scaling mode leads to serious resource mismatch and system stability issues, including:

[0005] 1) Low resource utilization: The actual data processing capacity of a single SCADA data processing container far exceeds 1,000 channels. Passive expansion has resulted in a large number of SCADA data processing containers being in a state of extremely low load, with computing resources (CPU, memory) being seriously idle and wasted.

[0006] 2) Deterioration of system stability: The disorderly growth in the number of containers causes the cluster scheduling complexity to increase exponentially, the number of system failure points to increase, and the reliability to decrease.

[0007] 3) Scalability bottleneck: The statically coupled architecture cannot adapt to the dynamic changes in the amount of new energy data, and the system scalability faces a ceiling, making it difficult to continue to evolve.

[0008] Therefore, breaking through the strong coupling constraints of the existing architecture and designing a new type of SCADA processing system that can achieve dynamic allocation of resources on demand and has elasticity and adaptability has become an urgent technical requirement for promoting the construction of new power systems. Summary of the Invention

[0009] Purpose of the invention: This invention provides a dynamically sensing-based containerized SCADA data processing elastic scaling method and system, which solves the core defects such as low resource utilization, poor system stability and insufficient scalability caused by the strong coupling between the front-end acquisition and processing containers. It breaks through the strong coupling constraints of the existing architecture, realizes on-demand dynamic allocation of resources in the SCADA processing system, and has elasticity and adaptability.

[0010] Technical solution: This invention provides a dynamically sensing-based containerized SCADA data processing elastic scaling method, comprising:

[0011] S1. Standardize the container image encapsulation of the SCADA data processing unit to obtain a container image that can be flexibly scheduled.

[0012] S2. Start the SCADA data processing container using the container image, and collect multi-dimensional perception indicator data in real time from the SCADA data processing container, the host machine, the SCADA service queue and the message bus channel. Establish an indicator collection and monitoring system, and write it into the real-time library and the time-series library simultaneously.

[0013] S3. Start the FES container, generate a channel routing table using a two-level routing mapping mechanism, initialize the channel routing table based on multi-dimensional perception index data, and dynamically bind the FES container to the SCADA data processing container.

[0014] S4. After normalizing the multi-dimensional perception index data, input it into the load assessment model to obtain the load score and short-term load trend prediction value, and output the elastic scaling decision in combination with the preset threshold.

[0015] S5. Based on the elastic scaling decision, expand or shrink the SCADA data processing container to form a new set of SCADA data processing containers that matches the current load.

[0016] S6. Inject the real-time indicator data of the new SCADA data processing container set back into the indicator acquisition and monitoring system, iteratively optimize the load assessment model, and continuously expand or shrink the SCADA data processing container.

[0017] Furthermore, S1 includes:

[0018] S1.1 Determine the set of materials required for mirroring based on the operational dependencies of the SCADA data processing unit;

[0019] S1.2 Write the material set into the standardized configuration;

[0020] S1.3 Based on the standardized configuration, generate a directory structure adapted for container operation on the host machine;

[0021] S1.4 Place all types of materials into the corresponding directories according to the rules to form reusable mirrored packaged content;

[0022] S1.5. Perform permission and version processing on the image packaged content;

[0023] S1.6. Generate a standardized container image from the processed image package contents.

[0024] Furthermore, S2 includes:

[0025] S2.1. Start the SCADA data processing container based on the standardized container image, and deploy the real-time library, time-series library and distributed agent to establish an indicator collection and monitoring system;

[0026] S2.2. Through the distributed agent, real-time collection of multi-dimensional indicator data is performed on SCADA data processing container resources, host resources, SCADA business queues and message bus channels.

[0027] S2.3. Synchronously write the collected indicator data into the real-time database and the time-series database.

[0028] Furthermore, S3 includes:

[0029] S3.1 Establish a two-level channel routing table in the real-time library. The first level records the fixed correspondence between the FES container and the channel interval consisting of consecutive channel numbers, and the second level records the dynamic correspondence between the channel interval and the SCADA data processing container.

[0030] S3.2 Start the FES container and initialize the channel routing table. Distribute the data collected and synchronized by the FES container to the real-time library and time-series library to the SCADA data processing container corresponding to the FES container in the channel routing table.

[0031] S3.3 When the channel routing table is updated, a routing table change event is sent to each FES container through the message bus. The FES container queries the local cached channel routing table with the channel number and re-matches the channel range with the SCADA data processing container.

[0032] Furthermore, S4 includes:

[0033] S4.1 Normalize the multi-dimensional perception indicator data in the real-time library and the time-series library;

[0034] S4.2 Input the normalized multi-dimensional perception index data into the load assessment model, use the weighted linear fusion algorithm to calculate the current load score, and use the ARIMA or LSTM model to perform short-term trend prediction on the normalized multi-dimensional perception index data to obtain the load trend prediction value within a preset time period in the future.

[0035] S4.3. Based on the preset expansion and contraction thresholds, combined with the load score and trend prediction value, output the elastic scaling decision, including the number of expansions and the number of contractions.

[0036] Furthermore, S5 includes:

[0037] S5.1 Based on the elastic scaling decision, call the container orchestration platform to issue a list of containers to be expanded or shrunk, and update the resource quotas simultaneously.

[0038] S5.2 During expansion, a new SCADA data processing container is created using the container image. The FES container is filtered, the corresponding channel range of the FES container is queried in the channel routing table, the new SCADA data processing container is bound to the corresponding channel range, and the data distributed by the FES container is received.

[0039] When scaling down, the channel range bound to the SCADA data processing container to be scaled down is redirected to other SCADA data processing containers, triggering a synchronous update of the channel routing table of all FES containers. After all FES containers have been updated, the data in the SCADA data processing container to be scaled down is cleared and the SCADA data processing container to be scaled down is destroyed.

[0040] S5.3 After the number of SCADA data processing containers is adjusted, the mapping relationship between the channel range and the SCADA data processing containers is recalibrated to form a new set of SCADA data processing containers that matches the current load.

[0041] Furthermore, S6 includes:

[0042] S6.1. Inject real-time multi-dimensional indicator data from the new SCADA data processing container set back into the indicator acquisition and monitoring system;

[0043] S6.2. Based on the back-injection index data, calibrate the load assessment model and update the weight coefficients of the weighted linear fusion algorithm and the short-term trend prediction weights.

[0044] S6.3 Continuously monitor load changes using the calibrated load assessment model, dynamically trigger the expansion or contraction of SCADA data processing containers, and maintain a dynamic match between the number of containers and the real-time load.

[0045] This invention also provides a dynamically sensing-based containerized SCADA data processing elastic scaling system, comprising:

[0046] The image generation module is used to encapsulate the SCADA data processing unit into a standardized container image to obtain a container image that can be flexibly scheduled.

[0047] The data acquisition module is used to start the SCADA data processing container using the container image, and to collect multi-dimensional perception indicator data in real time from the SCADA data processing container, the host machine, the SCADA business queue and the message bus channel, establish an indicator acquisition monitoring system, and synchronously write the data into the real-time library and the time-series library.

[0048] The dynamic binding module is used to start the FES container, generate a channel routing table using a two-level routing mapping mechanism, initialize the channel routing table based on multi-dimensional perception index data, and dynamically bind the FES container to the SCADA data processing container.

[0049] The elastic scaling decision module is used to normalize the multi-dimensional perception index data and input it into the load assessment model to obtain the load score and short-term load trend prediction value, and output the elastic scaling decision in combination with the preset threshold.

[0050] The new container forming module is used to expand or shrink the SCADA data processing container according to the elastic scaling decision to form a new set of SCADA data processing containers that matches the current load.

[0051] The dynamic execution module is used to inject real-time indicator data of the new SCADA data processing container set back into the indicator acquisition and monitoring system, iteratively optimize the load assessment model, and continuously expand or shrink the SCADA data processing container.

[0052] The present invention also provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the above-described method.

[0053] The present invention also provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the above-described method.

[0054] Beneficial effects: This invention provides a containerized distributed SCADA data processing and intelligent elastic scaling method and system, bringing significant advancements including:

[0055] 1) Based on self-developed key technologies such as container management, in-container service management, and real-time library cluster deployment, it realizes automated scheduling and containerized operation of real-time processing tasks, reducing the need for manual intervention and lowering the complexity and cost of operation and maintenance;

[0056] 2) By adopting an elastic resource scheduling method based on dynamic perception and intelligent routing, resource utilization is improved, the ability to cope with sudden traffic surges is enhanced, and predictive elastic scaling can prepare resources in advance before traffic peaks arrive. Combined with dynamic routing, traffic is quickly distributed to avoid system overload.

[0057] 3) Improved fault recovery efficiency: The dynamic routing mechanism enables automatic isolation and traffic migration of faulty units, reducing fault recovery time. This invention adopts a new architecture that combines physical distribution and logical unification, introducing cloud computing and containerization technologies to drive industrial technology transformation, address the channel expansion challenges brought about by new energy grid connection, and enhance system deployment flexibility and scenario adaptability. Attached Figure Description

[0058] Figure 1 Add a branch execution logic graph for the increment.

[0059] Figure 2 This is a diagram illustrating the collection of container resource metrics.

[0060] Figure 3 This is a schematic diagram of business queue metric collection.

[0061] Figure 4 Flowchart for the initial state of containerized data acquisition and processing.

[0062] Figure 5 Extended container flowchart for containerized data acquisition and processing.

[0063] Figure 6 To reduce the container flow of containerized data acquisition and processing. Detailed Implementation

[0064] The present invention will be further illustrated below with reference to the accompanying drawings and specific embodiments. It should be understood that the following specific embodiments are for illustrative purposes only and are not intended to limit the scope of the invention. After reading the present invention, any modifications of the present invention in various equivalent forms by those skilled in the art will fall within the scope defined by the appended claims.

[0065] Example 1

[0066] This invention provides a dynamically sensing-based containerized SCADA data processing elastic scaling method, comprising:

[0067] S1. Standardize the container image encapsulation of the SCADA data processing unit to obtain a container image that can be flexibly scheduled.

[0068] S2. Start the SCADA data processing container using the container image, and collect multi-dimensional perception indicator data in real time from the SCADA data processing container, the host machine, the SCADA service queue and the message bus channel. Establish an indicator collection and monitoring system, and write it into the real-time library and the time-series library simultaneously.

[0069] S3. Start the FES container, generate a channel routing table using a two-level routing mapping mechanism, initialize the channel routing table based on multi-dimensional perception index data, and dynamically bind the FES container to the SCADA data processing container.

[0070] S4. After normalizing the multi-dimensional perception index data, input it into the load assessment model to obtain the load score and short-term load trend prediction value, and output the elastic scaling decision in combination with the preset threshold.

[0071] S5. Based on the elastic scaling decision, expand or shrink the SCADA data processing container to form a new set of SCADA data processing containers that matches the current load.

[0072] S6. Inject the real-time indicator data of the new SCADA data processing container set back into the indicator acquisition and monitoring system, iteratively optimize the load assessment model, and continuously expand or shrink the SCADA data processing container.

[0073] In this embodiment,

[0074] The specific process of S1 includes:

[0075] S1.1 During the debugging phase, comprehensively review the missing system library materials (e.g., libprotobuf), system tool library materials, core process materials, configuration file materials, etc. (main core process scada_core, auxiliary process sca_log_agent, configuration file scada.ini), and configure the missing materials in the corresponding configuration items of the docbuild.ini file, clarifying the storage path and association relationship of each type of file; the docbuild.ini configuration example is as follows:

[0076]

[0077] The configuration items in docbuild.ini include the defined core processes, auxiliary processes, shared libraries, system libraries, configuration files and their path information. These need to be sorted out and configured into the corresponding options in advance during the program coding or deployment phase. The configuration items in docbuild.ini are shown in Table 1.

[0078] Table 1 docbuild.ini Configuration Items

[0079]

[0080] S1.2 Write the builddoc.sh SHELL script to initialize the script configuration array variable to store the parsing container build standard configuration file (docbuild.ini), including the system missing library list img_notexist_sysso, the configuration file path list img_config_file, the helper process list img_worker_process, and the system tools and their dependency mapping arrays img_sys_tools_exec and img_sys_tools_so, etc.

[0081] The `mkdir -p` command creates the complete directory structure required for image packaging, including `docbuild / user / conf`, `docbuild / user / bin`, `docbuild / user / lib`, `docbuild / system`, and `docbuild / other-file` (used for temporary storage of intermediate materials), ensuring that subsequent file copies have a clear target path;

[0082] S1.3 Execute the builddoc.sh script on the host machine to trigger the standard directory pre-creation process. The script automatically creates a directory structure that conforms to the container build specification, including the core business directory and the system dependency directory. The core business directory docbuild / user / {conf,bin,lib} stores the core materials related to SCADA business. conf stores configuration files, bin stores binary programs, and lib stores custom dynamic libraries. The system dependency directory docbuild / system creates subdirectories according to the original system path structure (such as / lib64, / usr / lib) to store the system-level materials necessary for container operation, and saves missing system libraries and system tool configuration files according to the original system path structure.

[0083] S1.4 The script parses the docbuild.ini file segment by segment according to the [SECTION] tag, mapping the configuration data to the corresponding array variables, including:

[0084] Parse the system library names configured under the [img_notexist_sysso] segment and store them in the img_notexist_sysso array;

[0085] Parse the KEY value under the [bool_nusp_conf] segment, and set the configuration file source identifier is_nusp_conf based on the KEY value being true / false;

[0086] Parse the filename and path of the configuration under the [img_config_file] segment, combine it with the is_nusp_conf identifier to concatenate the full path of the configuration file, parse it in the format of "filename=path", and store it in the img_config_file array;

[0087] Parse the tool names and dependent libraries configured under the [img_sys_tools] segment, and store them in img_sys_tools_exec (key: tool name) and img_sys_tools_so (value: dependent library) respectively, using the tool name=path resolution method;

[0088] Parse the KEY values ​​under the [img_sys_tools_conf] segment, split the system tool paths and dependent configuration files in the format KEY=VAL, and store them in the associative array img_sys_tools_conf_map (with tool paths as keys and configuration file lists as values).

[0089] S1.5 parses the number of script parameters and execution type, and executes different business logic, divided into two main branches: basic construction and incremental addition.

[0090] Based on the parameter type passed to the script, the basic build branch logic is executed (the parameter is the path to the scada_core main program). The specific process includes:

[0091] Determine the source path (~ / conf or current directory conf) based on the is_nusp_conf identifier, iterate through the img_config_file array, copy the core configuration file to the docbuild / user / conf directory, and overwrite if the file already exists;

[0092] Copy the main kernel program binary file (i.e., script parameter $1) to the docbuild / user / bin / directory, and use the chmod command to set executable permissions for all binary programs under docbuild / user / bin (e.g., chmod 755).

[0093] Collect the custom dependency libraries of the main core program, use the `ldd` command to parse the main program's dependency libraries, filter out custom libraries with the `awk` attribute, extract the full path of the library file, and then copy them in batches to `docbuild / user / lib` using `xargs`; for example, by executing `ldd "$1" | awk ' / libscada / {print $3}' | xargs -I {} cp {}docbuild / user / lib / `, filter for custom libraries with the `libscada` attribute; set executable permissions for the dynamic libraries under `docbuild / user / lib` (e.g., `chmod 755`) to ensure that the program and library files in the container can be loaded and run normally;

[0094] The system dependencies of the main core program are collected. The results of `ldd` are used to filter out non-custom shared system libraries. A string matching function is then called to compare these libraries against the `img_notexist_sysso` whitelist. For libraries that meet the criteria, target paths are created according to the principle of path relativity (e.g., ` / lib / libxxx.so` corresponds to `docbuild / system / lib / libxxx.so`). The parent directory is created using `mkdir -p`, and the library files are copied. Alternatively, by executing `ldd "$1" | grep -v "libscada" | awk '{print $3}'`, the missing system libraries array `img_notexist_sysso` is compared and copied; libraries not on the whitelist (defined in `img_notexist_sysso`) are not processed.

[0095] S1.6, Preprocess the configuration files and boot environment within the image, including:

[0096] Format conversion: The dos2unix tool is used to remove Windows format line breaks from cshrc (container environment variables), dockerfile (image build instructions), and shell (startup script) in the docbuild / docme directory to ensure cross-system compatibility.

[0097] Version information is generated by executing `. / docbuild / user / bin / scada_core -v` to extract version information to `docbuild / version.info`.

[0098] Copy the script files: By copying the preprocessed startup script (service.sh, which contains program start-up and stop logic and process protection configuration), debugging scripts (such as log viewing and process status detection scripts), and container build files (Dockerfile, Nari_Dockerfile) to the docbuild root directory, a complete container build and runtime support system is formed.

[0099] S1.7. To add new auxiliary materials, append the `add` command and target parameters (auxiliary process binary path name or `ALL`) to the SHELL script to execute the incremental addition branch logic, such as... Figure 1 As shown, the specific process includes:

[0100] S1.7.1 If the parameter is the full path to the auxiliary process binary file, then execute the logic for adding a single auxiliary process; if the parameter is ALL, then iterate through img_worker_process (a list of auxiliary processes) and img_sys_tools_exec (a list of system tools) and execute the logic for adding in batches.

[0101] S1.7.2 The process for adding a single auxiliary material is as follows: Copy the binary file of the auxiliary process or system tool to docbuild / user / bin. If the file already exists, overwrite it (ensuring the latest version is used) and output the copy log; parse the target binary file using ldd, filter out custom shared libraries with feature identifiers, and copy them to docbuild / user / lib to supplement the custom dependencies for the program running in the container; based on the ldd parsing results, filter out non-custom system libraries, compare them with the list of dependency libraries corresponding to the tool in the img_sys_tools_so array, and copy those that meet the conditions to docbuild / system according to the original path structure; query the img_sys_tools_conf_map associative array to obtain the list of configuration files corresponding to the current auxiliary material; create the target path according to the principle of path relativity (e.g., / etc / tools.conf corresponds to docbuild / system / etc / tools.conf), copy the configuration file to the target directory, and ensure that the configuration is available when the auxiliary material is running;

[0102] S1.7.3 The batch addition process is as follows: the single addition logic is called in a loop, and all auxiliary processes in img_worker_process and all system tools in img_sys_tools_exec are processed in sequence. After completion, a batch addition success message is output.

[0103] For example: If you need to add a new auxiliary process (such as log_agent), execute . / builddoc.sh add / opt / scada / bin / log_agent to trigger the "incremental addition" branch, copy the target binary (script parameter $2) to docbuild / user / bin; add the auxiliary process $2's dependency library to docbuild / user / lib / ; and inject the associated configuration into the docbuild / user / conf directory;

[0104] S1.8 Execute the container build command to generate a standardized data processing container image; switch to the docbuild directory to execute the container build command and submit it to the image repository.

[0105] S2 constructs a multi-dimensional perception indicator system for SCADA services. This system includes container system resource indicators (CPU, memory), host machine resource indicators (CPU, memory, disk I / O, network bandwidth), SCADA service processing indicators (message processing rate, queue backlog depth, processing latency), and channel status indicators (connection count, traffic rate, health status). The above indicator data is collected in real-time through a distributed agent client and persistently stored in a time-series database. The specific process includes:

[0106] S2.1 Deploy a real-time database (referred to as the real-time database, used to record real-time metric data) and a Prometheus time-series database (referred to as the time-series database, used to record metric data in terms of time dimension).

[0107] Create a real-time metric statistics table for containers in the real-time library to record real-time metric data for each container and business process. The table includes the following fields: 1) Container number (set as primary key), 2) Metric name, 3) Measurement type (remote signaling / telemetry), 4) Processing speed per second (number of measurements processed per second), 5) Queue backlog per second (number of measurements backlogged in the current container process per second), 6) Bus channel backlog per second (number of measurements backlogged in the current channel), 7) Container CPU utilization, 8) Container memory utilization, 9) Host to which the container belongs;

[0108] The time-series database uses a triplet structure of tags, timestamps, and values ​​to store various monitoring metrics related to containers. The tags in the triplet structure are as follows: 1) Container number (slice), 2) Metric name, 3) Measurement type (remote signaling / telemetry), 4) Processing speed per second (number of measurements processed per second), 5) Queue backlog per second (number of measurements backlogged in the current container process per second), 6) Bus channel backlog per second (number of measurements backlogged in the current channel), 7) Container CPU utilization, 8) Container memory utilization, and 9) Host of the container. The timestamp attribute value in the triplet structure is written to the time when the distributed agent client publishes data to the time-series database. The value in the triplet structure is obtained by the distributed agent client from the corresponding attribute metrics in the container's real-time metric statistics table.

[0109] S2.2 Configure the collection rules for container resources and system resources, adopting a distributed real-time library collection and time-series library aggregation mode to collect four types of core data indicators, forming a closed-loop monitoring link. The collection rules include:

[0110] Collecting container resource metrics includes CPU and memory utilization metrics for teleindication / telemetry processing containers. These metrics are collected by the distributed agent client, which uses regular expressions to match container names and container numbers, and invokes container resource viewing commands to collect the resource utilization of this type of container. The collected data is then published to the time-series database. Figure 2 As shown;

[0111] The collected business queue metrics include the number of items in the circular queue within the core process and processing rate metrics. These metrics are calculated in real time by the metric processing thread in the core process, and the calculated metrics are written to the corresponding attributes (such as queue backlog per second, processing speed per second) in the real-time metric statistics table of the real-time database container. Then, the metric publishing process running on the SCADA host machine collects the corresponding attributes (such as queue backlog per second, processing speed per second) from the real-time metric statistics table in the real-time database in real time and publishes them to the time-series database, such as... Figure 3 As shown;

[0112] S2.3 Develop and deploy a distributed proxy client that matches container names and container numbers using regular expressions. The process includes:

[0113] The collection process of the distributed agent client sca_agent is deployed on the host machine where the SCADA data processing container is located;

[0114] In sca_agent, the container resource collection configuration matches the SCADA data processing container (name prefixed with sca) using regular expressions. The regex pattern(R"(^sca_[a-zA-Z_]+_\d+$)") matches the SCADA data processing container and container number. In this case, the expression match[1] is the container name and match[2] is the container number.

[0115] S2.4 Develop and deploy the sca_agent container metric publishing function, call docker stats to collect container metrics, collect metrics (container CPU utilization, container memory utilization, host I / O busy rate) from the real-time library container metric information table and publish them to the time series library. The metric collection frequency is once every 5 seconds.

[0116] For example, in sca_agent, the system resource collection configuration uses the built-in node_disk_io and node_network_transmit_bytes metrics to collect disk I / O (unit: KB / s) and network bandwidth utilization (unit: %), and publishes them to the time series library. The metric collection frequency is once every 5 seconds.

[0117] S2.5 Develop a core process indicator processing thread to collect business queue and message bus indicators (such as internal loop queue backlog, data processing throughput, average data processing latency, and message bus channel backlog), and write the collected indicators into the real-time library container indicator information table. The collection frequency is once every 5 seconds.

[0118] S3 employs a two-level routing mapping and traffic scheduling mechanism. The first level is static configuration, recording a fixed correspondence between data acquisition containers (i.e., FES containers) and channel intervals composed of consecutive channel numbers. This assigns one or more data acquisition units (FES containers) to a specific channel interval, enabling business-level group management that remains unchanged regardless of the scaling up or down of backend physical containers. The second level dynamically maintains the correspondence between channel intervals and SCADA data processing containers. It defines the data traffic within a channel interval and determines which physical SCADA data processing containers should share the data traffic, achieving elastic resource allocation. By dynamically updating the second-level mapping, it achieves lossless traffic migration and load balancing. The migration process uses a double-buffering mechanism to ensure zero data loss. When the system needs to expand, new processing containers are automatically added to the channel interval; when scaling down, the SCADA data processing containers in the channel interval are removed. The specific process includes:

[0119] S3.1 Deploy the routing table storage environment, initialize the mapping data, and use a real-time library to store the channel routing table. The channel routing table fields include channel ID (channel_id, primary key), target SCADA data processing container ID (dist_dsca_id), routing status (active / inactive), associated FES container ID, and routing version number. Import the initial data in batches according to the mapping relationship of channel range - FES container - SCADA data processing container, adapting to one-to-one and flexible expansion association modes. The channel routing table is essentially a distributed routing table stored in a high-speed in-memory database (such as Redis) or a real-time SCADA library.

[0120] S3.2 Start the FES container, initialize and load the channel routing table, use a thread-safe hash structure to form a cache in local memory, and start a real-time incremental synchronization mechanism.

[0121] S3.3 The FES container collects measurement data (such as remote signaling changes and telemetry values) through the TCP port, parses the protocol header to extract the channel ID as the route matching key value, queries the locally cached channel routing table with the channel ID, and quickly matches the target SCADA data processing container. If the matched route is in an active state, the measurement data is distributed to the target SCADA data processing container. If no matched route is in an active state, a temporary routing mechanism is triggered to send the measurement data to the default buffer container of the SCADA cluster, synchronously record alarms and start route repair.

[0122] S3.4 A route cache synchronization mechanism is constructed. By configuring real-time library triggers, when changes such as additions, modifications, or deletions occur in the real-time library channel routing table, a change notification event is immediately triggered, ensuring a single query response time of ≤1ms. The notification is pushed to the corresponding FES container based on the associated FES container ID. Upon receiving the notification, the FES container only updates the corresponding channel routing table information and synchronously increments its local cache version number. Simultaneously, a timed full-scale verification mechanism is configured, performing version alignment every 3 minutes. The FES container compares its local cache version with the global version in the real-time library; if they are inconsistent, it incrementally pulls the differing routes, avoiding the risk of missed incremental synchronization.

[0123] The specific operation process of S4 includes:

[0124] S4.1 Deploy a negative feedback module on the SCADA container host, access the indicator data in the time series library, design a comprehensive load assessment model, and use a weighted fusion algorithm to calculate the comprehensive load score. The formula is as follows:

[0125] ;

[0126] Among them, the weighting coefficient It can also be dynamically adjusted according to the container memory situation; each indicator is mapped to the [0,1] range to complete normalization, and the upper limit threshold (such as 0.8) and lower limit threshold (such as 0.3) are preset as the triggering benchmark for elastic scaling action, with a preset 5-second collection cycle;

[0127] The dimensions and meanings of the comprehensive load assessment indicators are shown in Table 2.

[0128] Table 2 Three-dimensional load assessment

[0129]

[0130] A time series forecasting model (such as ARIMA or LSTM) is introduced to predict short-term load trends. This model employs a deployment approach combining online learning and offline training. Initially, a general-purpose model is used, and later, an LSTM model is trained based on collected historical data to achieve scenario adaptation. The predicted values ​​are used to trigger early warning-based pre-scaling and optimize scaling. A collaborative decision-making process, considering prediction confidence and real-time load values, is implemented, enabling a shift from passive response to proactive prediction. Specifically, this includes:

[0131] In the initial stage of system deployment, due to the lack of historical data, a lightweight, general-purpose benchmark model (such as Seasonal ARIMA) was used as the initial predictor. The model parameters were pre-trained on publicly available datasets of typical renewable energy power plants (such as the intraday fluctuation patterns of photovoltaic power plants). Simultaneously, a shadow mode was activated, where the model's predictions were used only for log recording and performance evaluation, not directly driving scaling decisions. Decisions were still primarily based on the current real-time load. This stage was mainly used to rapidly accumulate the system's specific historical data.

[0132] Once the system has accumulated enough historical data (e.g., after running continuously for 2 weeks), it will automatically trigger the scenario-adaptive training process. The training data will use multi-dimensional indicator data collected by the FES container.

[0133] Based on load scoring and trend prediction results, an elastic scaling strategy is generated. The strategy considers multiple factors such as expansion threshold, shrinkage threshold, cooldown time, and scaling range. It is deployed on the SCADA container host and combined with the closed loop of load monitoring system - decision-making - routing update to realize dynamic adjustment of resources and load and elastic scaling of containers.

[0134] The predicted values ​​obtained from the time series forecasting model are used to trigger early warning-based pre-expansion / pre-shrinkage.

[0135] When the predictive model outputs that the load score in the next 5-10 minutes will exceed the expansion threshold, the system will not immediately trigger formal expansion, but will instead enter an expansion preparation state. In this state, the system performs resource pre-checks; these checks whether there are sufficient idle resources in the resource pool. If not, it requests and initializes virtual machines / containers from the resource pool in advance, putting them in a hot standby state. Only when the real-time load also reaches the threshold will the system immediately add the hot standby resources to the service cluster and import traffic through a secondary routing mechanism. Pre-expansion reduces resource preparation time from minutes to seconds, ensuring resources wait for traffic, rather than traffic waiting for resources. Furthermore, a decision arbitrator is set up, whose inputs include the real-time load score (L_real), the predicted load score (L_pred), and the prediction confidence. When the prediction confidence is high (e.g., sunny weather, stable photovoltaic output), the system relies more on the predicted value and is more likely to execute pre-expansion in advance. When the prediction confidence is low (e.g., sudden weather changes), the decision weights shift towards the real-time value, prioritizing a conservative, passive response strategy to ensure system safety.

[0136] In pre-scaling, scaling down decisions are more cautious than scaling up. Even if a future load decrease is predicted, a hard condition must be met that the real-time load remains below the scaling down threshold for several consecutive periods, and a scaling down delay window is introduced. For example, if the system predicts that the load will decrease in 30 minutes, but the real-time load remains high, the system will wait until the real-time load reaches the threshold, and then, based on the predicted trend, confirm that the load will not rebound quickly before executing scaling down to prevent frequent scaling.

[0137] S4.2 Define the triggering rules for the elastic scaling strategy. The triggering rules include two types: expansion triggering and shrinkage triggering. The condition for expansion triggering is that the Load Score is ≥ the upper limit threshold for 3 consecutive collection cycles, or a single indicator exceeds the limit (such as queue backlog > 1000 and continues to grow). The condition for shrinkage triggering is that the Load Score is ≤ the lower limit threshold for 10 consecutive collection cycles, and the number of associated collection points is ≤ 50% of the initial value.

[0138] S4.3. A progressive scaling algorithm is adopted. After the adjustment is triggered, the adjustment action calculation is performed to determine the number of containers to be scaled up or down. The container orchestration platform is linked to perform container creation / destruction operations to avoid resource fluctuations. During the execution process, idle resources are scheduled first, and new resources are automatically requested from the resource pool when they are insufficient.

[0139] The expansion capacity is dynamically calculated based on the proportion of load exceeding the threshold. The single container's capacity is calibrated using historical data, and the calculation formula is as follows:

[0140]

[0141] The reduction in capacity is specified as follows: each reduction shall not exceed 20% of the current number of containers, and at least N basic containers shall be retained to avoid single point of failure; N is greater than or equal to 2, and N is 2 in this embodiment;

[0142] S4.4 After the scaling action is completed, load balancing and dynamic routing adjustment are triggered to achieve coordinated adaptation of routes and resources. When expanding, the load balancer migrates the collection points associated with the high-load container to the new FES container according to the principle of data type consistency (telemetry data of the same type are preferentially allocated to the same container), and synchronously updates the target container mapping of the corresponding channel_id in the channel routing table.

[0143] When scaling down, a three-step process is executed: route first → data collection → container destruction. First, the channel routing table is updated to redirect the channel_id associated with the container to be destroyed to other normal containers, triggering the synchronization and caching of the channel routing table of all FES containers. After all FES containers confirm that the route switch is completed, the container to be destroyed is notified to stop receiving new data. The queue backlog in the container is monitored to drop to 0 to ensure that all data has been processed. After a 30-second delay, the container destruction operation is performed.

[0144] S4.5. Add anti-jitter processing to the elastic scaling strategy generation and execution mechanism as a measure to ensure mechanism stability, including:

[0145] Configure anti-jitter processing and set a minimum adjustment interval (e.g., 5 minutes) to avoid frequent scaling in a short period of time and ensure the smoothness of load adjustment and routing adjustment;

[0146] The linked channel routing table updates the routing mapping in real time after the container scales up or down, ensuring continuous data distribution. The scaling-out verification mechanism requires that the number of containers in the secondary verification queue be zero and all FES container channel routing tables be switched before destruction can proceed.

[0147] The elastic scaling strategy generation and execution can also support dynamically adjusting the number of containers based on the number of preceding containers. This includes:

[0148] 1) Start the data processing container elastic scaling task and initialize the channel routing table. The FES subsystem contains all FES containers. Initially map the mapping relationship between the FES subsystem and the SCADA data processing container to ensure that each data processing container corresponds to a front-end container and processes the measurement data collected by the corresponding FES container.

[0149] 2) Triggering SCADA data processing container scaling down during front-end acquisition: When the number of containers in the FES subsystem decreases, the associated data processing container scaling down process is triggered synchronously. The corresponding data processing container is removed according to the one-to-one mapping relationship. At the same time, the dist_dsca_id mapping relationship of the channel_id in the channel routing table of the FES subsystem is updated to ensure continuous data routing.

[0150] When expanding the front-end acquisition capacity, the expansion of the SCADA data processing container is triggered: When a new container is added to the FES subsystem, a corresponding data processing container is automatically added, a new mapping relationship is established in the FES subsystem channel routing table, and a dedicated channel_id range is allocated to realize the independent routing and processing of the new measurement data.

[0151] The specific operation process of S5 includes:

[0152] S5.1 Based on the elastic scaling decision, call the container orchestration platform to issue a list of containers to be expanded or shrunk, and update the resource quota synchronously; before triggering, check the anti-jitter interval (e.g., 5 minutes), and delay the decision if the interval is not reached.

[0153] S5.2, Figure 4 This represents the initial state for container data processing. In this initial state, the data processing container environment and the FES container environment are based on standardized container images, deployed and started through a container orchestration platform, and configured with runtime parameters (e.g., 1 core 2GB for a single FES container, 2 cores 4GB for a single SCADA container), dynamic application state, scenario, sub-scenario, number of threads, and other parameters. Routing data is written in batches according to a preset association mode (e.g., one-to-one or flexible expansion). In the one-to-one mode, the FES container, ID-SCADA data processing container, and channel range are bound one-to-one. In the flexible expansion mode, dynamically allocated channel segments are reserved. The initial routing state is uniformly set to "act". The system initializes the routing version number to 1, associates it with the FES container ID, and precisely matches the corresponding data collection node; it triggers the FES container to load the channel routing cache and build a local memory hash cache; it starts the FES route real-time incremental synchronization and 3-minute full verification mechanism; it enables the monitoring load adjustment mechanism, connects the negative feedback module to the monitoring interface of each container, verifies that the initial CPU and memory utilization is ≤30%, and confirms that the container resource status is normal; it sends test measurement data (including each channel identifier) ​​to verify that the entire link of FES container route distribution, SCADA data processing container reception and processing is smooth, and the route matching accuracy is 100%.

[0154] During expansion, such as Figure 5As shown, a new SCADA data processing container is created based on a standardized image by calling the container orchestration API. The load balancer filters the collection points of high-load containers according to data type consistency and updates the target container ID corresponding to the channel_id in the channel routing table. After the FES container cache is synchronized, the new SCADA data processing container starts to receive and process data, and the expansion is completed.

[0155] During volume reduction, such as Figure 6 As shown, by prioritizing the updating of the channel routing table, the channel associated with the container to be destroyed is redirected to other normal containers, and a change notification is pushed to the associated FES subsystem; after receiving the routing switch confirmation from the FES subsystem, the container to be destroyed is notified to stop receiving new data, and its business queue backlog is monitored to drop to 0; after the dual conditions of queue clearing and FES subsystem routing switch completion are met, the container orchestration API is called to destroy the container.

[0156] S5.3 After the number of SCADA data processing containers is adjusted, the mapping relationship between the channel range and the SCADA data processing containers is recalibrated to form a new set of SCADA data processing containers that matches the current load.

[0157] The specific operation process of S6 includes:

[0158] S6.1 Inject the real-time multi-dimensional indicator data of the new SCADA data processing container set back into the indicator acquisition and monitoring system;

[0159] S6.2. Based on the back-injection index data, calibrate the load assessment model and update the weight coefficients of the weighted linear fusion algorithm and the short-term trend prediction weights.

[0160] S6.3 Continuously monitor load changes using the calibrated load assessment model, dynamically trigger the expansion or contraction of SCADA data processing containers, and maintain a dynamic match between the number of containers and the real-time load.

[0161] Example 2

[0162] Based on Embodiment 1, the present invention also provides a dynamically sensing-based containerized SCADA data processing elastic scaling system, comprising:

[0163] The image generation module is used to encapsulate the SCADA data processing unit into a standardized container image to obtain a container image that can be flexibly scheduled.

[0164] The data acquisition module is used to start the SCADA data processing container using the container image, and to collect multi-dimensional perception indicator data in real time from the SCADA data processing container, the host machine, the SCADA business queue and the message bus channel, establish an indicator acquisition monitoring system, and synchronously write the data into the real-time library and the time-series library.

[0165] The dynamic binding module is used to start the FES container, generate a channel routing table using a two-level routing mapping mechanism, initialize the channel routing table based on multi-dimensional perception index data, and dynamically bind the FES container to the SCADA data processing container.

[0166] The elastic scaling decision module is used to normalize the multi-dimensional perception index data and input it into the load assessment model to obtain the load score and short-term load trend prediction value, and output the elastic scaling decision in combination with the preset threshold.

[0167] The new container forming module is used to expand or shrink the SCADA data processing container according to the elastic scaling decision to form a new set of SCADA data processing containers that matches the current load.

[0168] The dynamic execution module is used to inject real-time indicator data of the new SCADA data processing container set back into the indicator acquisition and monitoring system, iteratively optimize the load assessment model, and continuously expand or shrink the SCADA data processing container.

[0169] In this embodiment,

[0170] The image generation module includes:

[0171] The sorting module is used during the debugging phase to comprehensively sort out missing system library materials (such as libprotobuf), system tool library materials, core process materials, configuration file materials, etc. (main core process scada_core, auxiliary process sca_log_agent, configuration file scada.ini), and configure the missing materials into the corresponding configuration items in the docbuild.ini file, clarifying the storage path and association relationship of various files; the docbuild.ini configuration example is as follows:

[0172]

[0173] The configuration items in docbuild.ini include the defined core processes, auxiliary processes, shared libraries, system libraries, configuration files and their path information. These need to be sorted out and configured into the corresponding options in advance during the program coding or deployment phase. The configuration items in docbuild.ini are shown in Table 1.

[0174] Table 1 docbuild.ini Configuration Items

[0175]

[0176] Write a script module to write the builddoc.sh SHELL script, initialize the script configuration array variable to store the parsed container build standard configuration file (docbuild.ini), including the system missing library list img_notexist_sysso, the configuration file path list img_config_file, the helper process list img_worker_process, and the system tools and their dependency mapping arrays img_sys_tools_exec and img_sys_tools_so, etc.

[0177] The `mkdir -p` command creates the complete directory structure required for image packaging, including `docbuild / user / conf`, `docbuild / user / bin`, `docbuild / user / lib`, `docbuild / system`, and `docbuild / other-file` (used for temporary storage of intermediate materials), ensuring that subsequent file copies have a clear target path;

[0178] The execution script module is used to execute the builddoc.sh script on the host machine, triggering the standard directory pre-creation process. The script automatically creates a directory structure that conforms to the container build specification, including the core business directory and the system dependency directory. The core business directory docbuild / user / {conf,bin,lib} stores the core materials related to SCADA business. conf stores configuration files, bin binary programs, and lib custom dynamic libraries. The system dependency directory docbuild / system creates subdirectories according to the original system path structure (such as / lib64, / usr / lib) to store the system-level materials necessary for container operation, and saves missing system libraries and system tool configuration files according to the original system path structure.

[0179] The script parsing module is used to parse the docbuild.ini file segment by segment according to the [SECTION] tag, mapping the configuration data to corresponding array variables, including:

[0180] Parse the system library names configured under the [img_notexist_sysso] segment and store them in the img_notexist_sysso array;

[0181] Parse the KEY value under the [bool_nusp_conf] segment, and set the configuration file source identifier is_nusp_conf based on the KEY value being true / false;

[0182] Parse the filename and path of the configuration under the [img_config_file] segment, combine it with the is_nusp_conf identifier to concatenate the full path of the configuration file, parse it in the format of "filename=path", and store it in the img_config_file array;

[0183] Parse the tool names and dependent libraries configured under the [img_sys_tools] segment, and store them in img_sys_tools_exec (key: tool name) and img_sys_tools_so (value: dependent library) respectively, using the tool name=path resolution method;

[0184] Parse the KEY values ​​under the [img_sys_tools_conf] segment, split the system tool paths and dependent configuration files in the format KEY=VAL, and store them in the associative array img_sys_tools_conf_map (with tool paths as keys and configuration file lists as values).

[0185] The business logic execution module is used to parse the number of script parameters and execution type, and execute different business logic. It is divided into two main branches: basic construction and incremental addition.

[0186] Based on the parameter type passed to the script, execute the basic build branch logic (the parameter is the path to the scada_core main program), including:

[0187] Determine the source path (~ / conf or current directory conf) based on the is_nusp_conf identifier, iterate through the img_config_file array, copy the core configuration file to the docbuild / user / conf directory, and overwrite if the file already exists;

[0188] Copy the main kernel program binary file (i.e., script parameter $1) to the docbuild / user / bin / directory, and use the chmod command to set executable permissions for all binary programs under docbuild / user / bin (e.g., chmod 755).

[0189] Collect the custom dependency libraries of the main core program, use the `ldd` command to parse the main program's dependency libraries, filter out custom libraries with the `awk` attribute, extract the full path of the library file, and then copy them in batches to `docbuild / user / lib` using `xargs`; for example, by executing `ldd "$1" | awk ' / libscada / {print $3}' | xargs -I {} cp {}docbuild / user / lib / `, filter for custom libraries with the `libscada` attribute; set executable permissions for the dynamic libraries under `docbuild / user / lib` (e.g., `chmod 755`) to ensure that the program and library files in the container can be loaded and run normally;

[0190] The system dependencies of the main core program are collected. The results of `ldd` are used to filter out non-custom shared system libraries. A string matching function is then called to compare these libraries against the `img_notexist_sysso` whitelist. For libraries that meet the criteria, target paths are created according to the principle of path relativity (e.g., ` / lib / libxxx.so` corresponds to `docbuild / system / lib / libxxx.so`). The parent directory is created using `mkdir -p`, and the library files are copied. Alternatively, by executing `ldd "$1" | grep -v "libscada" | awk '{print $3}'`, the missing system libraries array `img_notexist_sysso` is compared and copied; libraries not on the whitelist (defined in `img_notexist_sysso`) are not processed.

[0191] The preprocessing image module is used to preprocess the configuration files and boot environment within the image, including:

[0192] Format conversion: The dos2unix tool is used to remove Windows format line breaks from cshrc (container environment variables), dockerfile (image build instructions), and shell (startup script) in the docbuild / docme directory to ensure cross-system compatibility.

[0193] Version information is generated by executing `. / docbuild / user / bin / scada_core -v` to extract version information to `docbuild / version.info`.

[0194] Copy the script files: By copying the preprocessed startup script (service.sh, which contains program start-up and stop logic and process protection configuration), debugging scripts (such as log viewing and process status detection scripts), and container build files (Dockerfile, Nari_Dockerfile) to the docbuild root directory, a complete container build and runtime support system is formed.

[0195] The module adds a new component. If new auxiliary materials are needed, the `add` command followed by the target parameter (auxiliary process binary path name or `ALL`) in the shell script executes the incremental addition branch logic, such as... Figure 1 As shown, it includes a single auxiliary material addition module and a batch addition module;

[0196] If the parameter is the full path to the auxiliary process binary file, then the logic for adding a single auxiliary process is executed; if the parameter is ALL, then the logic for adding in batches is executed by iterating through img_worker_process (a list of auxiliary processes) and img_sys_tools_exec (a list of system tools).

[0197] The single auxiliary material addition module is used for: copying the binary files of auxiliary processes or system tools to docbuild / user / bin; if the file already exists, it will be overwritten (ensuring the latest version is used), and outputting the copy log; parsing the target binary file with ldd, filtering out custom shared libraries with feature identifiers, and copying them to docbuild / user / lib to supplement the custom dependencies for the program running in the container; based on the ldd parsing results, filtering out non-custom system libraries, comparing them with the list of dependency libraries corresponding to the tool in the img_sys_tools_so array, and copying those that meet the conditions to docbuild / system according to the original path structure; querying the img_sys_tools_conf_map associative array to obtain the list of configuration files corresponding to the current auxiliary material; creating the target path according to the principle of path relativity (e.g., / etc / tools.conf corresponds to docbuild / system / etc / tools.conf), copying the configuration files to the target directory, and ensuring that the configuration is available when the auxiliary material runs;

[0198] The batch add module is used to: repeatedly call the single add logic, process all auxiliary processes in img_worker_process and all system tools in img_sys_tools_exec in sequence, and output a batch add success message after completion.

[0199] For example: If you need to add a new auxiliary process (such as log_agent), execute . / builddoc.sh add / opt / scada / bin / log_agent to trigger the "incremental addition" branch, copy the target binary (script parameter $2) to docbuild / user / bin; add the auxiliary process $2's dependency library to docbuild / user / lib / ; and inject the associated configuration into the docbuild / user / conf directory;

[0200] The command execution module is used to execute container build commands, generate standardized data processing container images, switch to the docbuild directory to execute container build commands, and submit them to the image repository.

[0201] The data acquisition module constructs a multi-dimensional perception indicator system for SCADA services. This system includes container system resource indicators (CPU, memory), host machine resource indicators (CPU, memory, disk I / O, network bandwidth), SCADA service processing indicators (message processing rate, queue backlog depth, processing latency), and channel status indicators (connection count, traffic rate, health status). The above indicator data is collected in real time through a distributed agent client and persistently stored in a time-series database. The specific process includes:

[0202] The database deployment module is used to deploy the real-time database (referred to as the real-time database, which is used to record real-time metric data) and the Prometheus time-series database (referred to as the time-series database, which is used to record metric data in terms of time dimension).

[0203] Create a real-time metric statistics table for containers in the real-time library to record real-time metric data for each container and business process. The table includes the following fields: 1) Container number (set as primary key), 2) Metric name, 3) Measurement type (remote signaling / telemetry), 4) Processing speed per second (number of measurements processed per second), 5) Queue backlog per second (number of measurements backlogged in the current container process per second), 6) Bus channel backlog per second (number of measurements backlogged in the current channel), 7) Container CPU utilization, 8) Container memory utilization, 9) Host to which the container belongs;

[0204] The time-series database uses a triplet structure of tags, timestamps, and values ​​to store various monitoring metrics related to containers. The tags in the triplet structure are as follows: 1) Container number (slice), 2) Metric name, 3) Measurement type (remote signaling / telemetry), 4) Processing speed per second (number of measurements processed per second), 5) Queue backlog per second (number of measurements backlogged in the current container process per second), 6) Bus channel backlog per second (number of measurements backlogged in the current channel), 7) Container CPU utilization, 8) Container memory utilization, and 9) Host of the container. The timestamp attribute value in the triplet structure is written to the time when the distributed agent client publishes data to the time-series database. The value in the triplet structure is obtained by the distributed agent client from the corresponding attribute metrics in the container's real-time metric statistics table.

[0205] The configuration rules module is used to configure the collection rules for container resources and system resources. It adopts a distributed real-time library collection and time-series library aggregation mode to collect four types of core data indicators, forming a closed-loop monitoring link. The collection rules include:

[0206] Collecting container resource metrics includes CPU and memory utilization metrics for teleindication / telemetry processing containers. These metrics are collected by the distributed agent client, which uses regular expressions to match container names and container numbers, and invokes container resource viewing commands to collect the resource utilization of this type of container. The collected data is then published to the time-series database. Figure 2 As shown;

[0207] The collected business queue metrics include the number of items in the circular queue within the core process and processing rate metrics. These metrics are calculated in real time by the metric processing thread in the core process, and the calculated metrics are written to the corresponding attributes (such as queue backlog per second, processing speed per second) in the real-time metric statistics table of the real-time database container. Then, the metric publishing process running on the SCADA host machine collects the corresponding attributes (such as queue backlog per second, processing speed per second) from the real-time metric statistics table in the real-time database in real time and publishes them to the time-series database, such as... Figure 3 As shown;

[0208] The container matching module is used to develop and deploy distributed proxy clients. It matches container names and container numbers using regular expressions, including:

[0209] The collection process of the distributed agent client sca_agent is deployed on the host machine where the SCADA data processing container is located;

[0210] In sca_agent, the container resource collection configuration matches the SCADA data processing container (name prefixed with sca) using regular expressions. The regex pattern(R"(^sca_[a-zA-Z_]+_\d+$)") matches the SCADA data processing container and container number. In this case, the expression match[1] is the container name and match[2] is the container number.

[0211] The container metrics collection module is used to develop and deploy the sca_agent container metrics publishing function. It calls dockerstats to collect container metrics, collects metrics (container CPU utilization, container memory utilization, host I / O busy rate) from the real-time container metrics information table, and publishes them to the time series database. The metrics collection frequency is once every 5 seconds.

[0212] For example, in sca_agent, the system resource collection configuration uses the built-in node_disk_io and node_network_transmit_bytes metrics to collect disk I / O (unit: KB / s) and network bandwidth utilization (unit: %), and publishes them to the time series library. The metric collection frequency is once every 5 seconds.

[0213] The business metrics collection module is used to develop core process metrics processing threads, collect business queue and message bus metrics (such as internal loop queue backlog, data processing throughput, average data processing latency, and message bus channel backlog), and write the collected metrics into the real-time library container metric information table. The collection frequency is once every 5 seconds.

[0214] The dynamic binding module employs a two-level routing mapping and traffic scheduling mechanism. The first level is static configuration, recording a fixed correspondence between data acquisition containers (i.e., FES containers) and channel intervals composed of consecutive channel numbers. This assigns one or more data acquisition units (FES containers) to a specific channel interval, enabling business-level group management that remains unchanged regardless of the scaling up or down of backend physical containers. The second level dynamically maintains the correspondence between channel intervals and SCADA data processing containers. It defines the data traffic within a channel interval and determines which physical SCADA data processing containers should share the data traffic, achieving elastic resource allocation. By dynamically updating the second-level mapping, it achieves lossless traffic migration and load balancing. The migration process uses a double-buffering mechanism to ensure zero data loss. When the system needs to expand, new processing containers are automatically added to this channel interval; when scaling down, the SCADA data processing containers in this channel interval are removed, including:

[0215] The deployment routing table module is used to deploy the routing table storage environment, initialize mapping data, and store the channel routing table in a real-time library. The channel routing table fields include channel ID (channel_id, primary key), target SCADA data processing container ID (dist_dsca_id), routing status (active / inactive), associated FES container ID, and routing version number. Initial data is imported in batches according to the mapping relationship between channel range - FES container - SCADA data processing container, adapting to one-to-one and flexible expansion association modes. The channel routing table is essentially a distributed routing table stored in a high-speed in-memory database (such as Redis) or a real-time SCADA library.

[0216] The initialization channel routing table module is used to start the FES container, initialize and load the channel routing table, form a cache in local memory using a thread-safe hash structure, and start a real-time incremental synchronization mechanism.

[0217] The FES acquisition and distribution module is used by the FES container to collect measurement data (such as remote signaling changes and telemetry values) through TCP ports. It parses the protocol header to extract the channel ID as the route matching key value; it queries the locally cached channel routing table using the channel ID to quickly match the target SCADA data processing container; if the matched route is active, it distributes the measurement data to the target SCADA data processing container; if no matched route is active, it triggers a temporary routing mechanism to send the measurement data to the default buffer container of the SCADA cluster, synchronously records alarms, and initiates route repair.

[0218] The synchronization mechanism module is used to build a route cache synchronization mechanism. By configuring real-time library triggers, when changes such as additions, modifications, or deletions occur in the real-time library channel routing table, a change notification event is immediately triggered, ensuring that a single query response is ≤1ms. The notification is pushed to the corresponding FES container based on the associated FES container ID. Upon receiving the notification, the FES container only updates the corresponding channel routing table information and synchronously increments its local cache version number. Simultaneously, a timed full-scale verification mechanism is configured, performing version alignment every 3 minutes. The FES container compares its local cache version with the global version of the real-time library; if they are inconsistent, it incrementally pulls the differing routes, avoiding the risk of missed incremental synchronization.

[0219] The elastic scaling decision module includes:

[0220] The load assessment module is used to deploy the negative feedback module on the SCADA container host, access the indicator data in the time series library, design a comprehensive load assessment model, and calculate the comprehensive load score using a weighted fusion algorithm. The formula is as follows:

[0221] ;

[0222] Among them, the weighting coefficient It can also be dynamically adjusted according to the container memory situation; each indicator is mapped to the [0,1] range to complete normalization, and the upper limit threshold (such as 0.8) and lower limit threshold (such as 0.3) are preset as the triggering benchmark for elastic scaling action, with a preset 5-second collection cycle;

[0223] The dimensions and meanings of the comprehensive load assessment indicators are shown in Table 2.

[0224] Table 2 Three-dimensional load assessment

[0225]

[0226] A time series forecasting model (such as ARIMA or LSTM) is introduced to predict short-term load trends. This model employs a deployment approach combining online learning and offline training. Initially, a general-purpose model is used, and later, an LSTM model is trained based on collected historical data to achieve scenario adaptation. The predicted values ​​are used to trigger early warning-based pre-scaling and optimize scaling. A collaborative decision-making process, considering prediction confidence and real-time load values, is implemented, enabling a shift from passive response to proactive prediction. Specifically, this includes:

[0227] In the initial stage of system deployment, due to the lack of historical data, a lightweight, general-purpose benchmark model (such as Seasonal ARIMA) was used as the initial predictor. The model parameters were pre-trained on publicly available datasets of typical renewable energy power plants (such as the intraday fluctuation patterns of photovoltaic power plants). Simultaneously, a shadow mode was activated, where the model's predictions were used only for log recording and performance evaluation, not directly driving scaling decisions. Decisions were still primarily based on the current real-time load. This stage was mainly used to rapidly accumulate the system's specific historical data.

[0228] Once the system has accumulated enough historical data (e.g., after running continuously for 2 weeks), it will automatically trigger the scenario-adaptive training process. The training data will use multi-dimensional indicator data collected by the FES container.

[0229] Based on load scoring and trend prediction results, an elastic scaling strategy is generated. The strategy considers multiple factors such as expansion threshold, shrinkage threshold, cooldown time, and scaling range. It is deployed on the SCADA container host and combined with the closed loop of load monitoring system - decision-making - routing update to realize dynamic adjustment of resources and load and elastic scaling of containers.

[0230] The predicted values ​​obtained from the time series forecasting model are used to trigger early warning-based pre-expansion / pre-shrinkage.

[0231] When the predictive model outputs that the load score in the next 5-10 minutes will exceed the expansion threshold, the system will not immediately trigger formal expansion, but will instead enter an expansion preparation state. In this state, the system performs resource pre-checks; these checks whether there are sufficient idle resources in the resource pool. If not, it requests and initializes virtual machines / containers from the resource pool in advance, putting them in a hot standby state. Only when the real-time load also reaches the threshold will the system immediately add the hot standby resources to the service cluster and import traffic through a secondary routing mechanism. Pre-expansion reduces resource preparation time from minutes to seconds, ensuring resources wait for traffic, rather than traffic waiting for resources. Furthermore, a decision arbitrator is set up, whose inputs include the real-time load score (L_real), the predicted load score (L_pred), and the prediction confidence. When the prediction confidence is high (e.g., sunny weather, stable photovoltaic output), the system relies more on the predicted value and is more likely to execute pre-expansion in advance. When the prediction confidence is low (e.g., sudden weather changes), the decision weights shift towards the real-time value, prioritizing a conservative, passive response strategy to ensure system safety.

[0232] In pre-scaling, scaling down decisions are more cautious than scaling up. Even if a future load decrease is predicted, a hard condition must be met that the real-time load remains below the scaling down threshold for several consecutive periods, and a scaling down delay window is introduced. For example, if the system predicts that the load will decrease in 30 minutes, but the real-time load remains high, the system will wait until the real-time load reaches the threshold, and then, based on the predicted trend, confirm that the load will not rebound quickly before executing scaling down to prevent frequent scaling.

[0233] The scaling rules module is used to define the triggering rules for elastic scaling strategies. These triggering rules include expansion triggering and shrinkage triggering. Expansion triggering occurs when Load Score exceeds the upper limit threshold for three consecutive collection periods, or when a single metric exceeds its limit (e.g., queue backlog > 1000 records and continues to grow). Shrinkage triggering occurs when Load Score is less than or equal to the lower limit threshold for ten consecutive collection periods, and the number of associated collection points is less than or equal to 50% of the initial value.

[0234] The scaling adjustment module is used to perform adjustment calculations after triggering the scaling algorithm, determine the number of containers to scale, and link with the container orchestration platform to perform container creation / destruction operations to avoid resource fluctuations. During the execution process, idle resources are prioritized for scheduling, and new resources are automatically requested from the resource pool when there are insufficient resources.

[0235] The expansion capacity is dynamically calculated based on the proportion of load exceeding the threshold. The single container's capacity is calibrated using historical data, and the calculation formula is as follows:

[0236]

[0237] The reduction in capacity is specified as follows: each reduction shall not exceed 20% of the current number of containers, and at least N basic containers shall be retained to avoid single point of failure; N is greater than or equal to 2, and N is 2 in this embodiment;

[0238] The dynamic routing module is used to trigger load balancing and dynamic routing adjustments after scaling operations are completed, so as to achieve coordinated adaptation of routes and resources. When scaling up, the load balancer migrates the collection points associated with high-load containers to new FES containers according to the principle of data type consistency (such as prioritizing the allocation of telemetry data of the same type to the same container), and synchronously updates the target container mapping of the corresponding channel_id in the channel routing table.

[0239] When scaling down, a three-step process is executed: route first → data collection → container destruction. First, the channel routing table is updated to redirect the channel_id associated with the container to be destroyed to other normal containers, triggering the synchronization and caching of the channel routing table of all FES containers. After all FES containers confirm that the route switch is completed, the container to be destroyed is notified to stop receiving new data. The queue backlog in the container is monitored to drop to 0 to ensure that all data has been processed. After a 30-second delay, the container destruction operation is performed.

[0240] The anti-jitter module adds anti-jitter processing to the elastic scaling strategy generation and execution mechanism as a measure to ensure mechanism stability, including:

[0241] Configure anti-jitter processing and set a minimum adjustment interval (e.g., 5 minutes) to avoid frequent scaling in a short period of time and ensure the smoothness of load adjustment and routing adjustment;

[0242] The linked channel routing table updates the routing mapping in real time after the container scales up or down, ensuring continuous data distribution. The scaling-out verification mechanism requires that the number of containers in the secondary verification queue be zero and all FES container channel routing tables be switched before destruction can proceed.

[0243] The elastic scaling strategy generation and execution can also support dynamically adjusting the number of containers based on the number of preceding containers. This includes:

[0244] Start the data processing container elastic scaling task and initialize the channel routing table. The FES subsystem contains all FES containers. Initially map the mapping relationship between the FES subsystem and the SCADA data processing containers to ensure that each data processing container corresponds to a front-end container and processes the measurement data collected by the corresponding FES container.

[0245] When the front-end acquisition is scaled down, the SCADA data processing container is scaled down: When the number of containers in the FES subsystem decreases, the associated data processing container scaling down process is triggered synchronously. The corresponding data processing container is removed according to the one-to-one mapping relationship. At the same time, the dist_dsca_id mapping relationship of the channel_id in the channel routing table of the FES subsystem is updated to ensure continuous data routing.

[0246] When expanding the front-end acquisition capacity, the expansion of the SCADA data processing container is triggered: When a new container is added to the FES subsystem, a corresponding data processing container is automatically added, a new mapping relationship is established in the FES subsystem channel routing table, and a dedicated channel_id range is allocated to realize the independent routing and processing of the new measurement data.

[0247] The new container formation module includes:

[0248] The list module is used to call the container orchestration platform to issue a list of containers to be expanded or shrunk based on elastic scaling decisions, and to update resource quotas synchronously; before triggering, the anti-jitter interval (e.g., 5 minutes) is checked, and the decision is delayed if the interval is not reached.

[0249] Update the container module. Figure 4 This represents the initial state for container data processing. In this initial state, the data processing container environment and the FES container environment are based on standardized container images, deployed and started through a container orchestration platform, and configured with runtime parameters (e.g., 1 core 2GB for a single FES container, 2 cores 4GB for a single SCADA container), dynamic application state, scenario, sub-scenario, number of threads, and other parameters. Routing data is written in batches according to a preset association mode (e.g., one-to-one or flexible expansion). In the one-to-one mode, the FES container, ID-SCADA data processing container, and channel range are bound one-to-one. In the flexible expansion mode, dynamically allocated channel segments are reserved. The initial routing state is uniformly set to "act". The system initializes the routing version number to 1, associates it with the FES container ID, and precisely matches the corresponding data collection node; it triggers the FES container to load the channel routing cache and build a local memory hash cache; it starts the FES route real-time incremental synchronization and 3-minute full verification mechanism; it enables the monitoring load adjustment mechanism, connects the negative feedback module to the monitoring interface of each container, verifies that the initial CPU and memory utilization is ≤30%, and confirms that the container resource status is normal; it sends test measurement data (including each channel identifier) ​​to verify that the entire link of FES container route distribution, SCADA data processing container reception and processing is smooth, and the route matching accuracy is 100%.

[0250] During expansion, such as Figure 5As shown, a new SCADA data processing container is created based on a standardized image by calling the container orchestration API. The load balancer filters the collection points of high-load containers according to data type consistency and updates the target container ID corresponding to the channel_id in the channel routing table. After the FES container cache is synchronized, the new SCADA data processing container starts to receive and process data, and the expansion is completed.

[0251] During volume reduction, such as Figure 6 As shown, by prioritizing the updating of the channel routing table, the channel associated with the container to be destroyed is redirected to other normal containers, and a change notification is pushed to the associated FES subsystem; after receiving the routing switch confirmation from the FES subsystem, the container to be destroyed is notified to stop receiving new data, and its business queue backlog is monitored to drop to 0; after the dual conditions of queue clearing and FES subsystem routing switch completion are met, the container orchestration API is called to destroy the container.

[0252] The calibration module is used to recalibrate the mapping relationship between channel ranges and SCADA data processing containers after the number of SCADA data processing containers has been adjusted, forming a new set of SCADA data processing containers that matches the current load.

[0253] The dynamic execution module includes:

[0254] The data injection module is used to inject real-time multi-dimensional indicator data from the new SCADA data processing container set back into the indicator acquisition and monitoring system.

[0255] The model calibration module is used to calibrate the load assessment model based on the back-injected index data, and update the weight coefficients of the weighted linear fusion algorithm and the short-term trend prediction weights.

[0256] The dynamic scaling module is used to continuously monitor load changes using a calibrated load assessment model, dynamically triggering the expansion or contraction of SCADA data processing containers to maintain a dynamic match between the number of containers and the real-time load.

Claims

1. A dynamic perception-based containerized SCADA data processing elasticity method, characterized in that, The method comprises the following steps: S1, standardizing container image packaging of the SCADA data processing unit to obtain a container image capable of being elastically scheduled; S2, starting the SCADA data processing container by using the container image, collecting multi-dimensional sensing index data of the SCADA data processing container, the host computer, the SCADA service queue and the message bus channel in real time, establishing an index collection and monitoring system, and synchronously writing the index collection and monitoring system into a real-time database and a time sequence database; S3, starting the FES container, generating a path routing table by using a two-level routing mapping mechanism, initializing the path routing table based on the multi-dimensional sensing index data, and dynamically binding the FES container and the SCADA data processing container; S4, inputting the normalized multi-dimensional sensing index data into a load evaluation model to obtain a load score and a short-term load trend prediction value, and outputting an elastic scaling decision in combination with a preset threshold value; S5, expanding or shrinking the SCADA data processing container according to the elastic scaling decision to form a new SCADA data processing container set matched with the current load; S6, feeding back real-time index data of the new SCADA data processing container set to the index collection and monitoring system, iteratively optimizing the load evaluation model, and continuously implementing expansion or shrinkage of the SCADA data processing container.

2. The dynamic perception based containerized SCADA data processing elastic scaling method according to claim 1, wherein, S1 comprises: S1.1, determining a set of materials required for the image according to the running dependence of the SCADA data processing unit; S1.2, writing the set of materials into a standardized configuration; S1.3, generating a directory structure suitable for running the container based on the standardized configuration on the host computer; S1.4, placing various materials in corresponding directories according to rules to form reusable image packaging content; S1.5, processing the permissions and versions of the image packaging content; S1.6, generating a standardized container image from the processed image packaging content.

3. The dynamic perception based containerized SCADA data processing elastic scaling method of claim 1, wherein S2 Comprise: S2.1, starting the SCADA data processing container based on the standardized container image, and deploying a real-time database, a time sequence database and a distributed agent to establish an index collection and monitoring system; S2.2, collecting multi-dimensional index data of the SCADA data processing container resources, the host computer resources, the SCADA service queue and the message bus channel in real time through the distributed agent; S2.3, synchronously writing the collected index data into the real-time database and the time sequence database.

4. The dynamic perception based containerized SCADA data processing elastic scaling method of claim 1, wherein S3 Comprise: S3.1, establishing a two-level path routing table in the real-time database, the first level recording a fixed correspondence between the FES container and a channel interval composed of consecutive channel numbers, and the second level recording a dynamic correspondence between the channel interval and the SCADA data processing container; S3.2, starting the FES container and initializing the path routing table, synchronizing the FES container collection to the real-time database and the time sequence database, and distributing the data to the SCADA data processing container corresponding to the FES container in the path routing table; S3.3, when the path routing table is updated, issuing a routing table change event to each FES container through the message bus, the FES container querying the local cache path routing table by channel number, and re-matching the channel interval and the SCADA data processing container.

5. The dynamic perception based containerized SCADA data processing elastic scaling method according to claim 1, wherein S4 Comprise: S4.1, normalize the multi-dimensional perception index data in the real-time library and the time series library; S4.2, input the normalized multi-dimensional perception index data into the load evaluation model, calculate the current load score by using a weighted linear fusion algorithm, and use an ARIMA or LSTM model to perform short-term trend prediction on the normalized multi-dimensional perception index data to obtain a load trend prediction value in a future preset time; S4.3, according to the preset expansion threshold and the preset contraction threshold, combining the load score and the trend prediction value, output an elastic scaling decision, including the expansion quantity and the contraction quantity.

6. The dynamic perception based containerized SCADA data processing elastic scaling method according to claim 1, wherein S5 Comprise: S5.1, according to the elastic scaling decision, call the container orchestration platform to issue a container list to be expanded or contracted, and synchronously update the resource quota; S5.2, when expanding, create a new SCADA data processing container through a container image, filter FES containers, query the channel interval corresponding to the FES container in the routing table, bind the new SCADA data processing container with the corresponding channel interval, and receive the distributed data of the FES container; When contracting, direct the channel interval bound to the SCADA data processing container to be contracted to other SCADA data processing containers, trigger the routing table of all FES containers to be updated synchronously, and after all FES containers are updated, clear the data in the SCADA data processing container to be contracted, and destroy the SCADA data processing container to be contracted; S5.3, after the number of SCADA data processing containers is adjusted, recalibrate the mapping relationship between the channel interval and the SCADA data processing container to form a new SCADA data processing container set matched with the current load.

7. The dynamic perception based containerized SCADA data processing elastic scaling method according to claim 1, wherein S6 Comprise: S6.1, inject real-time multi-dimensional index data in the new SCADA data processing container set into the index collection and monitoring system; S6.2, calibrate the load evaluation model based on the injected index data, update the weight coefficients of the weighted linear fusion algorithm and the short-term trend prediction weight; S6.3, continuously monitor the load change by using the calibrated load evaluation model, dynamically trigger the expansion or contraction of the SCADA data processing container, and maintain the dynamic matching of the number of containers and the real-time load.

8. A dynamic perception based containerized SCADA data processing elastic scaling system, characterized in that, Comprise: An image generation module for standardizing container image packaging of the SCADA data processing unit to obtain a container image that can be elastically scheduled; A data collection module for starting the SCADA data processing container by using the container image, collecting multi-dimensional perception index data of the SCADA data processing container, host, SCADA service queue and message bus channel in real time, establishing an index collection and monitoring system, and synchronously writing into the real-time library and the time series library; A dynamic binding module for starting the FES container, generating a routing table by using a two-level routing mapping mechanism, initializing the routing table based on multi-dimensional perception index data, and dynamically binding the FES container and the SCADA data processing container; An elastic scaling decision module for inputting the normalized multi-dimensional perception index data into the load evaluation model to obtain a load score and a short-term load trend prediction value, and outputting an elastic scaling decision in combination with a preset threshold; A new container forming module is configured to expand or shrink the SCADA data processing container according to the elastic scaling decision, and form a new SCADA data processing container set matched with the current load; A dynamic execution module is configured to feed back real-time index data of the new SCADA data processing container set to an index collection monitoring system, iteratively optimize a load evaluation model, and continuously implement SCADA data processing container expansion or shrinkage.

9. A computer device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor executes the computer program to implement the steps of the method of any one of claims 1 to 7.

10. A computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement the steps of the method of any one of claims 1 to 7.

Images