Distributed satellite observation data processing task progress index real-time method

Abstract

The invention discloses a distributed satellite observation data processing task progress index real-time method. The index real-time method comprises the following steps: performing target code instrumentation on a distributed satellite observation data processing task based on OpenTelemetry framework multi-language characteristics; real-time receiving and persistent storage of Trace type telemetry data sequences are realized at a service monitoring rear end; and performing task completion time consumption prediction based on a task data volume and task time consumption relation model, and performing dynamic estimation on a real-time progress index of the task after the task completion time consumption prediction is combined with the received telemetry data. Through task progress index estimation vividly expressed by percentage, an index value represents a deviation degree between actual task execution and a task time consumption estimation value, and through combination with a Trace telemetry data sequence receiving condition, a health state of a task can be judged, and operation and maintenance personnel can take corresponding measures conveniently.

Description

Technical Field

[0001] The present invention relates to the field of distributed satellite observation data processing tasks, and in particular to a real-time method for distributed satellite observation data processing task progress indicators. Background Art

[0002] Observability comes from control theory. Observability refers to the ability to measure the internal state of a system by checking its outputs. The stronger the ability of these outputs to reflect the internal system state, the better the observability. In recent years, observability has been introduced into the IT field and has gradually become a necessary capability and strategic technology trend in the cloud-native era. Observability has gradually evolved from an operation and maintenance troubleshooting tool to a productivity tool, providing monitoring, analysis and system diagnosis capabilities for key links such as IT infrastructure, system applications, business and user feedback.

[0003] Observability uses three types of telemetry data, including metrics, logs, and traces, to provide in-depth visibility into distributed systems. By analyzing application metrics, logs, links, and other data, a complete observation model is built. OpenTelemetry is a mainstream open source observability framework in the industry. By providing a series of tools, APIs, and SDKs, it realizes the unified collection of metrics, logs, and traces, making it easier for developers to build observable modern applications.

[0004] Distributed satellite observation data processing tasks mainly refer to one-time operations of satellite observation data business processing in a distributed cluster computing environment. Usually, satellite observation data contains different observation payloads (such as optical imaging, infrared imaging, SAR imaging, etc.), which cannot be segmented by unified data slicing. It is necessary to extract data from different observation payloads and then perform distributed observation data processing. The amount of observation data that needs to be processed by the task varies greatly due to different payloads and observation modes. The task monitoring implemented by existing technologies is mainly collected, stored and displayed through distributed task logs, but the log content is only available to professionals and cannot intuitively reflect the real-time progress of the task.

[0005] Under the constraints of technical architecture and deployment architecture, the execution time of distributed satellite observation data processing tasks is related to the amount of processed data (such as the size of the input data file) and the complexity of the processing algorithm. When the algorithm is complex and relatively time-consuming, task monitoring requires understanding the real-time execution progress of the task / algorithm and how long it is expected to complete the task. The real-time progress indicator of the task is difficult to obtain solely relying on the observability of the application itself. Summary of the invention

[0006] In view of the above problems, the present invention is proposed to provide a real-time method for distributed satellite observation data processing task progress indicators that overcomes the above problems or at least partially solves the above problems.

[0007] According to one aspect of the present invention, a distributed satellite observation data processing task progress indicator real-time method is provided, and the indicator real-time method comprises:

[0008] Perform target code instrumentation for distributed satellite observation data processing tasks based on the multi-language features of the OpenTelemetry framework;

[0009] Implement real-time reception and persistent storage of Trace type telemetry data sequences in the service monitoring backend;

[0010] The task completion time is predicted based on the relationship model between task data volume and task duration. After being combined with the received telemetry data, the real-time progress indicators of the task are dynamically estimated.

[0011] Optionally, the method for establishing the task data volume and task time consumption relationship model includes:

[0012] Through historical data analysis, test data is analyzed and a linear regression relationship model between task data volume and task processing time is established.

[0013] Optionally, the target code stub specifically includes:

[0014] Based on the OpenTelemetry framework, the observation target code segment of the distributed satellite observation data processing task is tracked and instrumented. Different programming languages ​​correspond to different APIs:

[0015] The location of the code stubs is arranged according to the business needs of satellite payload observation data processing. It is required to have clear business meanings, realize the measurement of the entire task in one span, realize span stubs for important processing methods, and realize multi-span measurement and span nested measurement according to business needs;

[0016] The telemetry data of the distributed satellite observation data processing task is uniformly collected through the OTLP collector;

[0017] Task monitoring backend connection.

[0018] Optionally, the task monitoring backend connection specifically includes:

[0019] The task monitoring backend implements OTLP collector docking, and pulls the telemetry data collected by OTLP collector in real time through an API that supports HTTP / gRPC, or pushes it to the task monitoring backend in real time through OTLP exporter.

[0020] Optionally, the task running status processing specifically includes:

[0021] Obtain telemetry data;

[0022] Tracking data storage;

[0023] Prediction of time required to complete a task;

[0024] Real-time task progress estimation.

[0025] Optionally, the acquiring of telemetry data specifically includes:

[0026] When the task is running, the task monitoring backend continuously obtains the measurement span information of the trace through the OTLP exporter. The duration between the start and end of telemetry in the span information is the actual time consumption of the task.

[0027] The Span information received at the intermediate time interval can actually reflect the health status of the task.

[0028] Optionally, the tracking data storage specifically includes:

[0029] The trace span information obtained through telemetry data is formatted and persistently stored in the task monitoring backend. The trace data items that need to be persistently stored include: trace name trace_name, trace identifier trace_id, span name span_name, span ID span_id, parent span ID span_id, span start time start, span end time end, and status status;

[0030] For each tracking measurement, a span only needs to persist one piece of information, and the telemetry sequence information received at the intermediate time interval is used as the basis for judging the health status of the task.

[0031] Optionally, the task completion time prediction specifically includes:

[0032] After the task is started, the task calculation time is predicted based on the data volume of the task and the linear regression relationship model between the task data volume and the task time is established through historical data.

[0033] Optionally, the real-time progress estimation of the task specifically includes: estimating the real-time progress indicator of the task through a predicted value of the task execution time / task completion time.

[0034] Optionally, the adaptive iteration of the task data volume and task time consumption relationship model specifically includes:

[0035] By accumulating actual operation data, the relationship model is iterated according to the actual operation data volume and task duration to ensure the accuracy of task duration estimation;

[0036] The linear regression equation between the actual running data volume and the task duration is updated at a fixed period to calculate the predicted value of the task completion time.

[0037] The present invention provides a real-time method for distributed satellite observation data processing task progress indicators, which includes: target code plugging for distributed satellite observation data processing tasks based on the multi-language characteristics of the OpenTelemetry framework; real-time reception and persistent storage of Trace type telemetry data sequences in the service monitoring backend; task completion time prediction based on the task data volume and task time-consuming relationship model, and after combining with the received telemetry data, dynamic estimation of the task's real-time progress indicators. The task progress indicator estimation is expressed in percentage, and the indicator value represents the degree of deviation between the actual task execution and the estimated task time-consuming value. Combined with the reception of the Trace telemetry data sequence, it can judge the health status of the task, which is convenient for operation and maintenance personnel to take corresponding measures.

[0038] The above description is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the present invention, it can be implemented according to the contents of the specification. In order to make the above and other purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0040] Figure 1 A flowchart of a real-time method for distributed satellite observation data processing task progress indicators provided by an embodiment of the present invention;

[0041] Figure 2 A schematic diagram of a linear regression relationship model between task data volume and task processing time based on historical data / test data provided by an embodiment of the present invention;

[0042] Figure 3 A schematic diagram of a task code insertion position mode arranged according to business requirements provided by an embodiment of the present invention;

[0043] Figure 4A schematic diagram of Trace telemetry data collection and reception based on the OpenTelemetry framework provided in an embodiment of the present invention;

[0044] Figure 5 A schematic diagram of updating a linear regression relationship model between task data volume and task processing time consumption based on task running data provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0045] The exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided to enable a more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art.

[0046] The terms "comprises" and "having" and any variations thereof in the description embodiments, claims and drawings of the present invention are intended to cover non-exclusive inclusions, for example, including a series of steps or units.

[0047] The technical solution of the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments.

[0048] like Figure 1 As shown, the present invention provides a method for realizing real-time progress indicators of distributed satellite observation data processing tasks based on observability technology, performs target code plugging on distributed satellite observation data processing tasks based on the multi-language characteristics of the OpenTelemetry framework, realizes real-time reception and persistent storage of Trace type telemetry data sequences in the service monitoring backend, and predicts the time consumption for task completion based on the relationship model between task data volume and task time consumption, and after combining with the received telemetry data, realizes dynamic estimation of the real-time progress indicators of the task.

[0049] The content of the present invention includes:

[0050] 1. Establish a model for the relationship between task data volume and task time consumption.

[0051] Through historical data analysis, and also through test data analysis when there is no historical data, a linear regression relationship model between the task data volume (data file size) and the task processing time is established.

[0052] The linear regression relationship model between the task data volume and task processing time based on historical data / test data, such as Figure 2 shown.

[0053] For example, the regression trend line equation established based on test data is: y=0.3448x+330.54, where the x variable is the task data volume (unit: megabytes) and the y variable is the task duration (unit: seconds). The task duration for any amount of data input can be estimated based on the linear regression equation.

[0054] For example, when the data volume is 1550M, the model predicts that the task completion time is 864.98 seconds.

[0055] 2. Task code instrumentation.

[0056] 1) Based on the OpenTelemetry framework, the observation target code segment of the distributed satellite observation data processing task is traced and instrumented. Different programming languages ​​correspond to different APIs. Take C++ as an example:

[0057] auto provider=opentelemetry::trace::Provider::GetTracerProvider();

[0058] auto tracer = provider->GetTracer ("target task", "1.0.0");

[0059] trace_id = opentelemetry::tractrace.TraceId ("target task business unique identifier");

[0060] auto span = tracer->StartSpan ("target task measurement span");

[0061] The trace_id is assigned a unique identifier with business meaning, such as "satellite model + observation data ID + payload ID", and does not use automatically generated identifiers, which facilitates association analysis with other business information.

[0062] The location of code stubs can be flexibly handled and laid out according to the business needs of satellite payload observation data processing. It is required to have clear business meanings and achieve measurement of the entire task in one span. It can also implement span stubs for important processing methods (algorithms) and realize multi-Span measurement and Span nesting (parent-child relationship) measurement according to business needs.

[0063] like Figure 3 As shown in the figure, a schematic diagram of the task code insertion location pattern arranged according to business requirements.

[0064] The telemetry data of distributed satellite observation data processing tasks are uniformly collected through OTLP collector.

[0065] 2) Task monitoring backend connection

[0066] The task monitoring backend implements OTLP collector docking, and pulls the telemetry data collected by OTLP collector in real time through an API that supports HTTP / gRPC, or pushes it to the task monitoring backend in real time through OTLP exporter.

[0067] 2. Task running state processing

[0068] 1) Telemetry data acquisition

[0069] When the task is running, the task monitoring backend continuously obtains the measurement span information of the trace through the OTLP exporter. The time between the start and end of telemetry in the span information is the actual time consumption of the task. The span information received at the intermediate time interval can actually reflect the health status of the task.

[0070] like Figure 4 As shown in the figure, a schematic diagram of Trace telemetry data collection and reception based on the OpenTelemetry framework.

[0071] 2) Tracking data storage

[0072] The Trace Span information (Json format) obtained through telemetry data is formatted and persistently stored in the task monitoring backend. The trace data items that need to be persistently stored include: trace name (trace_name), trace identifier (trace_id), span name (span_name), span ID (span_id), parent span ID (span_id), span start time (start), span end time (end), and status (status). For each trace measurement, a span only needs to persist one piece of information. The telemetry sequence information received at the intermediate time interval can be used as a basis for judging the health status of the task (for example, if the measurement is performed once every 3 minutes and no measurement data is received for 3 consecutive times, it may indicate an abnormal state of task execution and the operation and maintenance personnel should pay close attention to it).

[0073] 3) Task completion time prediction

[0074] After the task is started, the predicted value of the time required to complete the task is calculated based on the data volume of the task and the linear regression relationship model between the task data volume and the task duration established through historical data.

[0075] 4) Real-time task progress estimation

[0076] The real-time progress indicator of the task is estimated by "Task execution time / Task completion time prediction value". If the real-time progress percentage of the task is within 100, it means that the task is still within the predicted completion time range. If it exceeds 100, it means that the task is delayed than the predicted completion time, and the task log situation needs to be paid special attention.

[0077] For example, the data volume of this task is 1550M. According to the regression trend line formula, the predicted completion time is 864.98 seconds. Under normal reception of telemetry data, at 600 seconds from the start of measurement, the real-time progress percentage of the task is: (600 / 864.98*100)%=69.37%; when the measurement starts 950 seconds later, it indicates that the task is delayed than the predicted completion time, and the real-time progress percentage of the task is: (950 / 831.89*100)%=109.83%.

[0078] 3. Adaptive iteration of the relationship model between task data volume and task time consumption, including:

[0079] Through the accumulation of actual operation data, the relationship model is iterated according to the actual operation data volume and task duration to ensure the accuracy of task duration estimation. The linear regression equation of actual operation data volume and task duration can be updated at a fixed period (such as every week) to calculate the predicted value of task completion time.

[0080] like Figure 5 As shown, a schematic diagram of updating the linear regression relationship model between task data volume and task processing time based on task running data.

[0081] For example, the regression trend line equation established based on actual task operation data is: y=0.2703x+558.03, where the x variable is the task data volume (unit: megabytes) and the y variable is the task duration (unit: seconds). Compared with the regression trend line based on test data, the slope is lower, indicating that the actual operation time is not as ideal as the test.

[0082] Beneficial effects:

[0083] The observability technology of distributed tasks is based on the OpenTelemetry framework and Otel protocol, ensuring multi-language support and standard protocol support for tasks;

[0084] Only trace data is telemetered for distributed tasks, which is relatively less invasive to task codes;

[0085] The task code instrumentation is laid out according to business needs and has clear business meanings. It can achieve the measurement of the entire task and key methods in one span, and also supports multi-span measurement and span nesting (parent-child relationship) measurement.

[0086] The actual task time prediction based on the relationship model between task data volume and task processing time can calculate the predicted value of task completion time after the task is started and telemetry is performed;

[0087] Real-time task progress indicator estimation: by combining telemetry trace data and the predicted value of task completion time based on the model, the real-time task progress indicator of the task is estimated;

[0088] The unique TraceID constraint can realize the correlation analysis with other business information in a distributed computing environment, providing more accurate problem and fault location.

[0089] As actual operation data accumulates, the task data volume and task duration relationship model is regularly iterated, so that the task duration estimation result can be closer to the actual duration;

[0090] The task progress indicator is estimated by expressing it in percentage. The indicator value represents the degree of deviation between the actual task execution and the estimated task time. Combined with the reception of Trace telemetry data sequence, it can judge the health status of the task and facilitate operation and maintenance personnel to take corresponding measures.

[0091] The above specific implementation methods further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above are only specific implementation methods of the present invention and are not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A real-time method for distributed satellite observation data processing task progress indicators, characterized in that: The indicator real-time method includes: Perform target code instrumentation for distributed satellite observation data processing tasks based on the multi-language features of the OpenTelemetry framework; Implement real-time reception and persistent storage of Trace type telemetry data sequences in the service monitoring backend; The task completion time is predicted based on the relationship model between task data volume and task duration. After being combined with the received telemetry data, the real-time progress indicators of the task are dynamically estimated.

2. A distributed satellite observation data processing task progress indicator real-time method according to claim 1, characterized in that: The method for establishing the task data volume and task time consumption relationship model includes: Through historical data analysis, test data is analyzed and a linear regression relationship model between task data volume and task processing time is established.

3. A distributed satellite observation data processing task progress indicator real-time method according to claim 1, characterized in that: The target code plugging specifically includes: Based on the OpenTelemetry framework, the observation target code segment of the distributed satellite observation data processing task is tracked and instrumented. Different programming languages ​​correspond to different APIs: The location of the code stubs is arranged according to the business needs of satellite payload observation data processing. It is required to have clear business meanings, realize the measurement of the entire task in one span, realize span stubs for important processing methods, and realize multi-span measurement and span nested measurement according to business needs; The telemetry data of the distributed satellite observation data processing task is uniformly collected through the OTLP collector; Task monitoring backend connection.

4. A distributed satellite observation data processing task progress indicator real-time method according to claim 3, characterized in that: The task monitoring backend docking specifically includes: The task monitoring backend implements OTLP collector docking, and pulls the telemetry data collected by OTLP collector in real time through an API that supports HTTP / gRPC, or pushes it to the task monitoring backend in real time through OTLP exporter.

5. The distributed satellite observation data processing task progress indicator real-time method according to claim 1, characterized in that: The task running status processing specifically includes: Obtain telemetry data; Tracking data storage; Prediction of time required to complete a task; Real-time task progress estimation.

6. A distributed satellite observation data processing task progress indicator real-time method according to claim 5, characterized in that: The acquisition of telemetry data specifically includes: When the task is running, the task monitoring backend continuously obtains the measurement span information of the trace through the OTLP exporter. The duration between the start and end of telemetry in the span information is the actual duration of the task. The Span information received at the intermediate time interval can actually reflect the health status of the task.

7. A distributed satellite observation data processing task progress indicator real-time method according to claim 5, characterized in that: The tracking data storage specifically includes: The trace span information obtained through telemetry data is formatted and persistently stored in the task monitoring backend. The trace data items that need to be persistently stored include: trace name trace_name, trace identifier trace_id, span name span_name, span ID span_id, parent span ID span_id, span start time start, span end time end, and status status; For each tracking measurement, a span only needs to persist one piece of information, and the telemetry sequence information received at the intermediate time interval is used as the basis for judging the health status of the task.

8. A distributed satellite observation data processing task progress indicator real-time method according to claim 5, characterized in that: The task completion time prediction specifically includes: After the task is started, the task calculation time is predicted based on the data volume of the task and the linear regression relationship model between the task data volume and the task time is established through historical data.

9. A distributed satellite observation data processing task progress indicator real-time method according to claim 5, characterized in that: The real-time task progress estimation specifically includes: estimating the real-time task progress indicator through the predicted value of task execution time / task completion time.

10. The distributed satellite observation data processing task progress indicator real-time method according to claim 1, characterized in that: The adaptive iteration of the task data volume and task time-consuming relationship model specifically includes: By accumulating actual operation data, the relationship model is iterated according to the actual operation data volume and task duration to ensure the accuracy of task duration estimation; The linear regression equation between the actual running data volume and the task duration is updated at a fixed period to calculate the predicted value of the task completion time.

Images