Real-time monitoring method and system for satellite data processing software

Abstract

The application relates to the technical field of software running state monitoring, and provides a real-time monitoring method and system for satellite data processing software, which comprises the following steps: acquiring the consumption delay and consumption offset of an input message queue and the production offset of an output message queue; calculating the difference between the consumption offset and the production offset to obtain a processing success amount and a processing rate; acquiring the cumulative received data quantity in a counting storage module, performing time sliding window statistics, and obtaining the total amount of received data in the window; and based on the consumption delay, the processing success amount, the processing rate and the total amount of received data in the window, a software state is obtained through combined judgment rules. The false positives and false negatives are greatly reduced, the software state can be accurately distinguished, consumption slow, processing card, writing error and upstream breakage are distinguished, and the monitoring is more accurate.

Description

Technical Field

[0001] This invention belongs to the field of software operation status monitoring technology, and particularly relates to a real-time monitoring method and system for satellite data processing software. Background Technology

[0002] The statements in this section are merely background information related to the present invention and do not necessarily constitute prior art.

[0003] Current satellite data processing systems generally adopt a distributed architecture, using message queues (such as the distributed publish-subscribe messaging system Kafka) for data transmission and decoupling. Data is received upstream, processed by the processing software, and then written to the downstream message queue or storage system. To ensure the stable operation of the satellite data processing system, it is necessary to monitor the consumption status, processing speed, and data integrity of the data processing software.

[0004] Existing monitoring solutions mainly fall into the following two categories: It only monitors a single metric of the message queue, such as consumption latency (ConsumerLag) and consumption offset (ConsumerOffset), and triggers an alarm when ConsumerLag increases. The system counts the number of input and output messages, but only performs simple counting and comparison; it does not perform trend prediction, rate comparison, or multi-dimensional joint judgment.

[0005] Existing technologies collect ConsumerLag data and issue alarms when it exceeds a fixed threshold. They also count the number of records processed over a period of time to determine if there is data inflow. However, relying on a single indicator makes it impossible to distinguish between different root causes of failures, such as slow consumption, processing freezes, write failures, upstream disconnections, and logical errors. This results in slow fault location, many false alarms, and an inability to predict the risk of data backlog in advance. Summary of the Invention

[0006] To address the technical problems mentioned above, this invention provides a real-time monitoring method and system for satellite data processing software. It proposes a combined consumption status dimension, processing success rate / volume dimension, and data reception sliding window dimension to monitor the satellite data processing software status: quantifying the processing success rate by the offset difference between the input and output message queues; judging performance insufficiency, freezing, and logical errors by rate comparison; and implementing data reception interruption detection based on the time sliding window of the counting storage module. This upgrade from a single indicator to a three-indicator joint judgment significantly reduces false alarms and missed alarms, accurately distinguishing between slow consumption, processing freezes, write errors, and upstream interruptions, making monitoring more accurate.

[0007] To achieve the above objectives, the present invention adopts the following technical solution: The first aspect of this invention provides a real-time monitoring method for satellite data processing software, comprising: Get the consumption delay and consumption offset of the input message queue and the production offset of the output message queue; Calculate the difference between the consumption offset and the production offset to obtain the processing success rate and processing speed; Obtain the cumulative number of received data entries in the counting storage module, perform time sliding window statistics, and obtain the total amount of received data within the window; The software state is obtained by combining judgment rules based on consumption latency, number of successful processing, processing rate, and total amount of data received within the window.

[0008] Furthermore, it also includes: based on the consumption latency of the input message queue, through smoothing calculation and trend prediction, to obtain the predicted value for future time moments; if the predicted value for future time moments exceeds the limit, an early warning of data accumulation risk is issued.

[0009] Furthermore, when the data processing software consumes each piece of satellite data from the input message queue, the counting storage module writes an incremental record.

[0010] Furthermore, the consumption offset is the total amount of data successfully consumed by the data processing software from the input queue.

[0011] Furthermore, the production offset is the total amount of processing results successfully written by the data calculation and processing software to the output queue.

[0012] A second aspect of the present invention provides a real-time monitoring system for satellite data processing software, comprising: The status acquisition module is configured to acquire the consumption delay and consumption offset of the input message queue and the production offset of the output message queue. The difference calculation module is configured to calculate the difference between the consumption offset and the production offset to obtain the processing success rate and the processing speed. The sliding window calculation module is configured to: obtain the cumulative number of received data entries in the counting storage module, perform time sliding window statistics, and obtain the total amount of received data within the window; The joint decision module is configured to obtain the software state based on consumption delay, number of successful processing, processing rate, and total amount of data received within the window by combining decision rules.

[0013] Furthermore, it also includes a trend prediction module, which is configured to: based on the consumption delay of the input message queue, obtain the predicted value for the future time through smoothing calculation and trend prediction; if the predicted value for the future time exceeds the limit, it will issue a warning of data accumulation risk.

[0014] Furthermore, when the data processing software consumes each piece of satellite data from the input message queue, the counting storage module writes an incremental record.

[0015] Furthermore, the consumption offset is the total amount of data successfully consumed by the data processing software from the input queue.

[0016] Furthermore, the production offset is the total amount of processing results successfully written by the data calculation and processing software to the output queue.

[0017] Compared with the prior art, the beneficial effects of the present invention are: This invention proposes a combined consumption status dimension, a processing success rate dimension, and a data reception sliding window dimension to monitor the status of satellite data processing software: quantifying the processing success rate by the offset difference between the input and output message queues; judging performance insufficiency, freezing, and logical errors by rate comparison; and realizing data reception interruption detection based on the time sliding window of the counting storage module. Upgrading from a single indicator to a three-indicator joint judgment significantly reduces false alarms and missed alarms, and can accurately distinguish between slow consumption, processing freezes, write errors, and upstream interruptions, making monitoring more accurate. Attached Figure Description

[0018] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an improper limitation of the invention.

[0019] Figure 1 This is a flowchart of a real-time monitoring method for satellite data processing software according to Embodiment 1 of the present invention; Figure 2 This is a schematic diagram of a real-time monitoring method for satellite data processing software according to Embodiment 1 of the present invention. Detailed Implementation

[0020] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings.

[0021] It should be noted that the following detailed description is illustrative and intended to provide further explanation of the invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0022] Terminology Explanation: ConsumerLag: Consumption latency of data processing software. This value will appear when the data processing software reads and processes data with a lag. Ideally, ConsumerLag will be 1 or 0.

[0023] ConsumerOffset: The consumer offset of the data processing software.

[0024] Redis: A high-speed in-memory database with high-speed storage and retrieval capabilities, capable of storing time-series data.

[0025] Example 1 This embodiment provides a real-time monitoring method for satellite data processing software.

[0026] This embodiment provides a real-time monitoring method for satellite data processing software, which is mainly applied to high-throughput, high-reliability distributed processing systems for satellite remote sensing data and satellite observation data.

[0027] In practical satellite data processing scenarios, monitoring a single indicator is highly prone to false alarms and missed alarms: ConsumerLag is functioning normally, but the processing program is actually stuck. ConsumerLag rises slowly but does not exceed the threshold, making it impossible to issue an early warning. Data reception is normal, but processing results are lost, making it impossible to quickly locate the problem.

[0028] Existing technologies cannot predict consumption delay trends, can only issue passive alerts, and cannot intervene in advance; they cannot jointly determine the health status of the processing link by input Topic (message queue topic) and output Topic offset; they cannot perform sliding window statistics on data reception, making it difficult to quickly determine whether the upstream is disconnected; and they lack a joint judgment mechanism among multiple indicators, making it impossible to automatically give fault conclusions and handling suggestions.

[0029] Therefore, this embodiment performs joint monitoring and intelligent judgment from three dimensions: consumption status, processing capacity, and data access, to achieve accurate positioning, early warning, and automatic diagnosis.

[0030] This embodiment provides a real-time monitoring method for satellite data processing software, which achieves full-process status perception and anomaly diagnosis through a three-layer monitoring mechanism of consumption status monitoring, processing efficiency evaluation, and data access verification.

[0031] Satellite data processing software involves modules such as Figure 2 As shown, it includes: Satellite transceiver module: Responsible for receiving raw satellite data and writing it into a topic (Topic A) in the input message queue as the data source for data processing; Data processing software: Reads and processes data from a topic (Topic A) in the input message queue, and simultaneously writes the number of received data entries to the Redis counting storage module / sliding window calculation module in real time; after processing, it writes the result data to a topic (Topic B) in the output message queue. The output message queue contains Topic B, which stores processed satellite data for use by downstream systems.

[0032] This embodiment provides a real-time monitoring method for satellite data processing software, such as... Figure 1 As shown, it includes the following steps: Step 1: Start the satellite data processing software to monitor the process in real time.

[0033] Step 2: The satellite data processing software collects status data and calculates indicator parameters to provide a basis for the next step of judgment.

[0034] Step 201: Input a topic (Topic A) in the message queue to provide the current maximum offset and the amount of unconsumed data to the consumption status acquisition module. Based on this, the consumption status acquisition module calculates and outputs the ConsumerLag (consumption delay) and the ConsumerOffset (consumption offset) of the input queue.

[0035] Topic A is where the decrypted raw binary data transmitted from the satellite is stored (using a middleware-message queue). The purpose of the data processing software is to convert this binary data into predefined physical quantities, allowing ground operators to view the satellite's specific status and operation.

[0036] This example uses a Kafka message queue to monitor metrics based on topics within the message queue: The Topic includes two native basic metrics that can be directly collected: LogEndOffset (LEO, the current maximum offset of the partition) and Current Offset (consumer position). The ConsumerOffset and ProducerOffset mentioned in the article are essentially the same parameter for this consumer position. The only difference between the two is the naming based on the target Topic being read: the consumer position that has been submitted by the consumer group is denoted as ConsumerOffset, which also supports direct collection. The formula for calculating data lag, ConsumerLag, is: ConsumerLag = LogEndOffset - CurrentOffset. This can be understood through a simple analogy: a topic is like a book, LogEndOffset represents the total number of pages (100), and CurrentOffset represents the 80th page currently being read. The difference of 20 is the number of unread pages, corresponding to the metric ConsumerLag, which directly reflects the amount of data lag.

[0037] Satellite equipment continuously writes data to Topic A in the input message queue, while the data processing software continuously pulls data from Topic A to complete its calculations. However, the satellite's data write rate fluctuates, and the data processing software's read efficiency is constrained by the hardware operating environment. If the software's processing speed cannot keep up with the satellite's data write rate for an extended period, data backlog and lag will occur. If this lag cannot be detected in real time, the accumulated backlog can easily lead to data processing program crashes, loss of delayed data, and other malfunctions.

[0038] In this embodiment, the message queue can be Kafka (Distributed Streaming Platform), RabbitMQ (Rabbit Message Queue), RocketMQ (Rocket Message Queue), Pulsar, etc.

[0039] The data processing software receives the ConsumerLag (consumption delay) and ConsumerOffset (consumption offset) from the producer's Topic, processes them, and outputs the ProducerOffset (production offset) from the consumer's Topic. The specific judgment rule is as follows: ConsumerLag=0: Consumption is real-time; ConsumerLag continues to rise: Consumers are unable to process information or are stuck. ConsumerLag remains unchanged, ConsumerOffset remains unchanged: Consumer abnormally stops.

[0040] ProducerOffset is used in the next step. It is also a parameter value that is read directly. It reads the offset of a message queue topic (Topic A) that the data processing software distributes to the outside after parsing the satellite data.

[0041] Step 202: The ConsumerLag in the input queue is sent to the EWMA trend prediction module for smoothing and trend prediction, which is used to provide early warning of data accumulation risks.

[0042] This embodiment uses EWMA (Exponentially Weighted Moving Average) to smooth ConsumerLag and predict its trend. The trend prediction formula is as follows: ; Where: t represents time, in minutes; EWMA This represents the predicted ConsumerLag value after t minutes; α represents the weighting coefficient, which can be set to 0.2. This represents the actual value of the current ConsumerLag in the input queue.

[0043] Preferably, The flow fluctuation coefficient F = mean flow rate over the time-series statistical period / standard deviation of flow rate over the time-series statistical period. The time-series statistical period is a fixed duration used to assess satellite flow fluctuations. Multiple sets of instantaneous flow rate samples are continuously collected within this period to calculate the flow fluctuation coefficient. The single set of instantaneous flow rate within the period is the number of new data entries added to Topic A within a single sampling interval. The fault risk coefficient = current ConsumerLag / static alarm threshold Lag alarm S is the early warning tolerance coefficient, a manually configured parameter with a value of (0,1); α base The weighting coefficient is used as the baseline. The dynamic weighting coefficient calculation formula integrates the time-series statistical periodic traffic fluctuation coefficient, fault risk coefficient, and manually configured early warning fault tolerance coefficient. It can automatically adjust the EWMA weighting coefficient in real time based on the stability of satellite downlink traffic, the urgency of current data accumulation, and the alarm tolerance strategy preset by the operation and maintenance. Compared with the traditional fixed α value, it can automatically reduce the coefficient and enhance the smoothing effect to filter instantaneous lag spikes and reduce false alarms when the satellite pulses sudden traffic. At the same time, it can automatically increase the coefficient and improve the trend response sensitivity when the data accumulation is close to the alarm threshold, so as to realize the early prediction of high-risk scenarios. At the same time, it can adapt to the operation and maintenance management requirements of different levels of satellite missions through the fault tolerance coefficient, and constrain α within a reasonable range throughout the process to avoid prediction failure. There is no need to manually modify static parameters repeatedly. It takes into account the prediction accuracy, fault early warning timeliness, and operation and maintenance adaptation flexibility, and perfectly adapts to the business monitoring scenario of intermittent and highly fluctuating satellite data.

[0044] An alarm is triggered if the current value exceeds the configured threshold, and if the future value is predicted. If the limit is exceeded, an early warning will be issued. Specifically, if the current ConsumerLag exceeds the configured alarm value, an alarm will be issued directly; when... If the predicted value of ConsumerLag after t minutes is greater than the configured warning value, then a warning is issued.

[0045] The configuration thresholds vary greatly depending on the data consumption frequency of different satellite missions.

[0046] In this embodiment, EWMA can be replaced by trend prediction methods such as simple linear regression, moving average, and first-order filtering.

[0047] Step 203: Simultaneously, the output message queue (Topic B) provides the current maximum offset data to the processing status acquisition module, and the processing status acquisition module outputs the ProducerOffset of the output queue based on this.

[0048] Step 204: Calculate the difference between the collected input offset (i.e., ConsumerOffset) and output offset (i.e., ProducerOffset) to obtain the processing success rate and processing speed, which are used to evaluate processing performance.

[0049] Normally, data processing software should output the same number of data points it receives, meaning the input offset equals the output offset. If the output offset is greater than the input offset, it means the data processing software received the data but discarded a portion before completing the processing.

[0050] By calculating the difference between the input offset and the output offset, a quantitative evaluation of data processing performance can be achieved. Compared with the existing technology that only uses simple counting and comparison, it has higher accuracy and fault differentiation.

[0051] Calculation method: Unprocessed data volume ≈ Input offset - Output offset; Wherein: Input offset: represents the total amount of data successfully consumed by the data processing software from the input queue (Topic A), which is collected in real time by the consumption status acquisition module; Output offset: represents the total amount of processing results successfully written by the data processing software to the output queue (Topic B), which is collected in real time by the processing status acquisition module.

[0052] Judgment rules: ① Input Topic increment > 0, output Topic increment = 0; This indicates that data is continuously flowing in, but no processing results are produced. Existing technologies can only determine "no data output". This embodiment can accurately locate the consumer suspension or the processing logic deadlock, clearly distinguishing between the two completely different fault scenarios of "no data inflow" and "data inflow but processing failure". This embodiment can distinguish them by comparing ProducerOffset and ConsumerOffset. Under normal circumstances, the data processing software generates one piece of data when it receives one piece of data. ProducerOffset and ConsumerOffset can be used to check the values ​​of the middleware to determine whether the data processing software has received data and whether the processing was successful. ② The growth rate of the output topic is significantly lower than that of the input topic; at this time, the speed at which processing results are generated is far behind the speed at which data flows in. Existing technologies often categorize this as "insufficient performance." This embodiment, through rate comparison, can further distinguish whether it is a bottleneck in processing performance (such as algorithm complexity or insufficient resources) or partial data loss (such as abnormal data being discarded or retry mechanism failure), providing a clear basis for expansion, optimization, or data recovery. In this embodiment, the rate at which ConsumerOffset increases is much greater than the rate at which ProducerOffset increases, meaning that the data processing software's data receiving speed is far behind the speed of the data processing business. Therefore, distributed expansion needs to be considered, and this provides a basis for expansion. ③ Output Topic increment > Input Topic increment; at this time, the number of results generated is greater than the number of data consumed. Existing technology is difficult to detect this kind of logical error. This embodiment can accurately identify it as a write logic error or program abnormality (such as duplicate writing, result generation logic error), effectively preventing data redundancy or downstream system abnormalities caused by program bugs (errors); in this embodiment, when the rate of increase of ConsumerOffset is much smaller than the rate of increase of ProducerOffset, the data processing software has no data input, but there is output, indicating that unknown data is flowing into the output Topic, which is not allowed and requires the operation and maintenance personnel to investigate.

[0053] Step 205: Based on the number of received data entries stored in Redis, perform time-sliding window statistics, output the received data volume in the sliding window statistics, determine the data access status, and identify whether there is a data reception interruption.

[0054] This embodiment uses a time-sliding window statistical mechanism to continuously and accurately monitor the data reception status, effectively identifying anomalies such as upstream data interruptions. The specific implementation is as follows: The data processing software counts and increments the number of received data entries in real time as it consumes each satellite data entry from the input message queue (Topic A). For each data item successfully consumed, an incremental record is written to the Redis counting storage module. The record contains a timestamp and the current cumulative number of data items received. Redis uses sorted sets or time-series structures for storage, sorted by timestamps, which facilitates subsequent window statistics.

[0055] This embodiment performs statistics based on a time-sliding window, using a fixed time window (e.g., 1 minute, 5 minutes, configurable), with the window scrolling forward over time. During each statistical analysis, all received records within the current window (e.g., [now-window_size,now], where now is the current time and window_size represents the fixed window length) are filtered from Redis. The total amount of received data within the window is calculated (i.e., the cumulative number of records at the end of the window minus the cumulative number of records at the beginning of the window). If the total amount of received data within the window is 0, it is determined that data reception has been interrupted.

[0056] In this embodiment, the Redis sliding window can be replaced by open-source time-series databases such as InfluxDB (time-series database), Prometheus, and TDengine (time-series engine).

[0057] Step 3: Joint Decision and Output.

[0058] It receives key indicators from three dimensions, including EWMA trend prediction results, input queue offset (ConsumerOffset), output queue offset (ProducerOffset), and sliding window statistics; it makes a comprehensive judgment based on preset multi-state combination rules, and outputs monitoring conclusions and anomaly levels; the final results are displayed or alarms are triggered through the monitoring result output / alarm module, realizing real-time perception and anomaly diagnosis of the entire process status of satellite data processing software.

[0059] Step 301: Comprehensively match the three types of indicators: consumption status, processing efficiency, and data access, and output the monitoring conclusion and anomaly level according to the preset multi-state combination rules.

[0060] The status is mainly classified as normal, requires attention, abnormal, or severe abnormal, and automatic handling suggestions are provided.

[0061] As shown in Table 1, the rules for determining different operating states are specifically divided into 5 scenarios: Scenario 1: When ConsumerLag continues to rise, the sliding window receive volume > 0, and the output increment lags significantly behind the input, it is judged as a serious anomaly, indicating a problem of rapid data accumulation and approaching the backlog collapse threshold; Scenario 2: When ConsumerLag remains constant and ConsumerOffset does not advance, the sliding window's received amount is greater than 0, the input increment is greater than 0, and the output increment is 0, it is judged as a serious anomaly, resulting in the consumer thread being stuck or the business processing being suspended. Scenario 3: When ConsumerLag is normal and stable, the sliding window receive volume > 0 and the output increment (output offset) > input increment (input offset), it is judged as a general anomaly, indicating that there is a problem with the result writing logic, duplicate generation or duplicate distribution. Scenario 4: When ConsumerLag is normal and stable, the sliding window receives >0, the input increment >0, and the output increment =0, it is judged as a general anomaly. The consumer can pull messages normally but there are no processing results output. Scenario 5: When the sliding window receive amount = 0, the input increment = 0, and the output increment approaches 0 in any ConsumerLag state, it is judged as a serious anomaly, resulting in interruption of upstream satellite data reception and link disconnection.

[0062] Table 1. Typical Combination Judgment Rules (Core)

[0063] Step 302: Whether the limit is exceeded or the predicted limit is exceeded: Determine whether the current ConsumerLag exceeds the configured threshold, or whether the EWMA trend prediction result shows that the limit will be exceeded in the future; if yes, then execute an alarm or warning; if no, then return directly to step 2 and enter the next round of monitoring cycle.

[0064] Step 303: When it is determined that the limit has been exceeded or the limit is predicted to be exceeded, the corresponding alarm or early warning mechanism is triggered to notify the operation and maintenance personnel to handle it.

[0065] Step 304: Complete this round of monitoring process and return to Step 2 to continue real-time monitoring.

[0066] This embodiment provides a real-time monitoring method for satellite data processing software, which enables accurate monitoring of the entire link status of satellite data processing software.

[0067] This embodiment provides a real-time monitoring method for satellite data processing software, which supports trend prediction and enables early warning of data accumulation and loss.

[0068] This embodiment provides a real-time monitoring method for satellite data processing software, proposing a three-dimensional joint monitoring method for the status of satellite data processing software: consumption status dimension + processing success volume / rate dimension + data reception sliding window dimension. It upgrades from a single indicator to a joint judgment of three indicators, which greatly reduces false alarms and missed alarms, and can accurately distinguish between slow consumption, processing lag, write errors, and upstream disconnection, making the monitoring more accurate.

[0069] This embodiment provides a real-time monitoring method for satellite data processing software, which achieves accurate monitoring of the entire data processing chain based on a combination of ConsumerLag, ConsumerOffset, and ProducerOffset judgment.

[0070] This embodiment provides a real-time monitoring method for satellite data processing software. Based on EWMA, it performs trend prediction and graded early warning of consumption delay. By introducing EWMA trend prediction, it can provide early warning of accumulation risks rather than passive alarms, and has predictive capabilities.

[0071] This embodiment provides a real-time monitoring method for satellite data processing software, which has built-in combined judgment rules, automatically locates faulty links in seconds and provides handling suggestions, eliminating the need for manual troubleshooting and making fault location faster.

[0072] This embodiment provides a real-time monitoring method for satellite data processing software. It quantifies the processing success rate by measuring the offset difference between the input Topic and the output Topic; it judges performance insufficiency, freezing, and logical errors by comparing the rate; it implements data reception interruption detection based on Redis time sliding window; and it proposes an intelligent diagnostic mechanism that automatically outputs fault conclusions and processing suggestions by combining multiple states.

[0073] This embodiment provides a real-time monitoring method for satellite data processing software, which is designed for high-throughput, high-reliability satellite processing systems that cannot lose data. It significantly improves stability and availability and is more suitable for satellite data scenarios.

[0074] This embodiment provides a real-time monitoring method for satellite data processing software. It monitors the middleware based on message queue offset and Redis sliding window, without requiring intrusion or modification of business code, resulting in low deployment costs.

[0075] Example 2 This embodiment provides a real-time monitoring system for satellite data processing software, including: The status acquisition module is configured to acquire the consumption delay and consumption offset of the input message queue and the production offset of the output message queue. The difference calculation module is configured to calculate the difference between the consumption offset and the production offset to obtain the processing success rate and the processing speed. The sliding window calculation module is configured to: obtain the cumulative number of received data entries in the counting storage module, perform time sliding window statistics, and obtain the total amount of received data within the window; The joint decision module is configured to obtain the software state based on consumption delay, number of successful processing, processing rate, and total amount of data received within the window by combining decision rules.

[0076] Furthermore, it also includes a trend prediction module, which is configured to: based on the consumption delay of the input message queue, obtain the predicted value for the future time through smoothing calculation and trend prediction; if the predicted value for the future time exceeds the limit, it will issue a warning of data accumulation risk.

[0077] Furthermore, when the data processing software consumes each piece of satellite data from the input message queue, the counting storage module writes an incremental record.

[0078] Furthermore, the consumption offset is the total amount of data successfully consumed by the data processing software from the input queue.

[0079] Furthermore, the production offset is the total amount of processing results successfully written by the data calculation and processing software to the output queue.

[0080] Furthermore, the rules for determining different operating states are specifically divided into 6 scenarios: Scenario 1: When ConsumerLag continues to rise, the sliding window receive volume > 0, and the output increment lags significantly behind the input, it is judged as a serious anomaly, indicating a problem of rapid data accumulation and approaching the backlog collapse threshold; Scenario 2: When ConsumerLag remains constant and ConsumerOffset does not advance, the sliding window's received amount is greater than 0, the input increment is greater than 0, and the output increment is 0, it is judged as a serious anomaly, resulting in the consumer thread being stuck or the business processing being suspended. Scenario 3: When ConsumerLag is normal and stable, the sliding window receive volume > 0 and the output increment > input increment, it is judged as a general anomaly, indicating that there is a problem with the result writing logic, duplicate generation or duplicate distribution. Scenario 4: When ConsumerLag is normal and stable, the sliding window receives >0, the input increment >0, and the output increment =0, it is judged as a general anomaly. The consumer can pull messages normally but there is no processing result output. Scenario 5: When the sliding window receive amount = 0, the input increment = 0, and the output increment approaches 0 in any ConsumerLag state, it is judged as a serious anomaly, resulting in interruption of upstream satellite data reception and link disconnection.

[0081] This embodiment provides a real-time monitoring system for satellite data processing software, proposing a joint dimension of consumption status, processing success rate, and data reception sliding window to monitor the status of the satellite data processing software: quantifying the processing success rate by the offset difference between the input message queue and the output message queue; judging performance insufficiency, freezing, and logical errors by rate comparison; realizing data reception interruption detection based on the time sliding window of the counting storage module; upgrading from a single indicator to a joint judgment of three indicators, significantly reducing false alarms and missed alarms, and accurately distinguishing between slow consumption, processing freezes, write errors, and upstream interruptions, making monitoring more accurate.

[0082] It should be noted that each module in this embodiment corresponds one-to-one with each step in Embodiment 1, and their specific implementation processes are the same, so they will not be repeated here.

[0083] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A real-time monitoring method for satellite data processing software, characterized in that, include: Get the consumption delay and consumption offset of the input message queue and the production offset of the output message queue; Calculate the difference between the consumption offset and the production offset to obtain the processing success rate and processing speed; Obtain the cumulative number of received data entries in the counting storage module, perform time sliding window statistics, and obtain the total amount of received data within the window; Based on consumption latency, successful processing volume, processing rate, and total amount of data received within the window, the software state is determined by combining judgment rules. Specifically, the judgment rules for different operating states are divided into five scenarios: Scenario 1: When the consumption latency continues to rise, the sliding window reception volume is greater than 0, and the output increment lags significantly behind the input, it is judged as a serious anomaly, indicating that there is a problem of rapid data accumulation and approaching the backlog collapse threshold; the number of consumers should be expanded immediately and non-core computing tasks should be temporarily downgraded to prevent data loss and process crashes. Scenario 2: When the consumption delay remains constant and the consumption offset does not advance, the sliding window receive volume > 0, the input increment > 0, and the output increment = 0, it is judged as a serious anomaly, resulting in the consumer thread being stuck or the business processing being suspended; the consumer instance should be restarted, deadlock / blocking code should be investigated, message submission anomalies should be checked, and the difference between data flowing in but not being processed at all should be distinguished. Scenario 3: When the consumption delay is normal and stable, the sliding window receive volume is >0 and the output increment is > the input increment, it is judged as a general anomaly, indicating that there is a problem with the result writing logic, duplicate generation or duplicate distribution; the business branch logic, deduplication mechanism, and bug of duplicate writing of result into the database should be checked to avoid downstream data redundancy and disorder. Scenario 4: When the consumption delay is normal and stable, the sliding window receive volume > 0, the input increment > 0, and the output increment = 0, it is judged as a general anomaly. The consumer can pull messages normally but there is no processing result output. The following issues should be investigated: a topic logic written to the output message queue after processing, abnormal result storage, and data discarding in the intermediate process. Scenario 5: When the sliding window receive volume = 0, the input increment = 0, and the output increment approaches 0 under any consumption delay state, it is judged as a serious anomaly, indicating that the upstream satellite data reception is interrupted or the link is disconnected; the satellite data source, network link, and message queue production end status should be investigated to quickly restore data access; Based on the consumption latency of the input message queue, a predicted value for future time moments is obtained through smoothing calculation and trend prediction. If the predicted value for future time moments exceeds the limit, a warning of data backlog risk is issued. An exponentially weighted moving average is used to perform smoothing calculation and trend prediction on the consumption latency. The trend prediction formula is as follows: ； Where: t represents time, in minutes; EWMA This represents the predicted consumption delay value after t minutes; α represents the weighting coefficient. This represents the actual value of the current consumption delay in the input queue; Wherein, the flow fluctuation coefficient F = the average flow rate of the time-series statistical period / the standard deviation of the flow rate of the time-series statistical period, and the fault risk coefficient. =Current data lag (ConsumerLag) / Static alarm threshold (Lag) alarm S is the early warning fault tolerance coefficient, α base The base weighting coefficient.

2. The real-time monitoring method for satellite data processing software as described in claim 1, characterized in that, When the data processing software consumes each piece of satellite data from the input message queue, the counting storage module writes an incremental record.

3. The real-time monitoring method for satellite data processing software as described in claim 1, characterized in that, The consumption offset is the total amount of data successfully consumed by the data processing software from the input queue.

4. The real-time monitoring method for satellite data processing software as described in claim 1, characterized in that, The production offset is the total amount of processing results successfully written by the data processing software to the output queue.

5. A real-time monitoring system for satellite data processing software, characterized in that, A method for implementing real-time monitoring of satellite data processing software as described in any one of claims 1-4 includes: The status acquisition module is configured to acquire the consumption delay and consumption offset of the input message queue and the production offset of the output message queue. The difference calculation module is configured to calculate the difference between the consumption offset and the production offset to obtain the processing success rate and the processing speed. The sliding window calculation module is configured to: obtain the cumulative number of received data entries in the counting storage module, perform time sliding window statistics, and obtain the total amount of received data within the window; The joint decision module is configured to obtain the software state based on consumption delay, number of successful processing, processing rate, and total amount of data received within the window by combining decision rules.

6. The real-time monitoring system for satellite data processing software as described in claim 5, characterized in that, When the data processing software consumes each piece of satellite data from the input message queue, the counting storage module writes an incremental record.

7. The real-time monitoring system for satellite data processing software as described in claim 5, characterized in that, The consumption offset is the total amount of data successfully consumed by the data processing software from the input queue.

8. The real-time monitoring system for satellite data processing software as described in claim 5, characterized in that, The production offset is the total amount of processing results successfully written by the data processing software to the output queue.

Images