A distributed satellite observation data processing task progress index real-time method

Abstract

The application discloses a kind of distributed satellite observation data processing task progress index real-time methods, the index real-time method includes: based on the target code insertion of distributed satellite observation data processing task is staked to OpenTelemetry framework multilingual characteristic;Real-time receiving and persistent storage of Trace type telemetry data sequence are realized in service monitoring back end;Task completion time consumption prediction is carried out based on the relationship model of task data volume and task time consumption, and after being combined with the received telemetry data, the real-time progress index of task is dynamically estimated.Through the task progress index estimation of percentage image expression, the deviation degree between the actual task execution and the estimated value of task time consumption is represented by index value, combined with the reception condition of Trace telemetry data sequence, the health status of task can be judged, and it is convenient for operation and maintenance personnel to take corresponding measures.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of distributed satellite observation data processing tasks, and in particular to a distributed satellite observation data processing task progress index real-time method. BACKGROUND

[0002] Observerbility comes from control theory, which refers to the ability to measure the internal state of a system by checking its output. The stronger the ability of these outputs to reflect the internal system state, the better the observerbility. In recent years, observerbility has been introduced into the IT field and has gradually become a necessary ability and strategic technology trend in the cloud-native era. Observerbility has gradually evolved from a problem troubleshooting tool to a productivity tool, providing monitoring, analysis, and system diagnosis capabilities for key links such as IT infrastructure, system applications, business, and user feedback.

[0003] Observerbility uses three types of telemetry data, including metrics, logs, and traces, to provide deep visibility into distributed systems. By analyzing application metrics, logs, and link data, a complete observation model is constructed. OpenTelemetry is a leading open-source observerbility framework in the industry. By providing a series of tools, APIs, and SDKs, it has achieved the unified collection of metrics, logs, and traces, providing convenience for developers to build modern observerbility applications.

[0004] Distributed satellite observation data processing tasks refer to one-time jobs for satellite observation data processing in a distributed cluster computing environment. Satellite observation data often contains different observation payloads (such as optical imaging, infrared imaging, SAR imaging, etc.), which cannot be divided by uniform data slicing. Different observation payloads need to be extracted for distributed observation data processing. The amount of observation data that needs to be processed varies greatly depending on the payload and observation mode. Existing task monitoring technologies mainly collect, store, and display distributed task logs. However, log content is only understood by professionals and cannot directly reflect the real-time progress of the task.

[0005] Under the constraints of technical architecture and deployment architecture, the execution time of distributed satellite observation data processing tasks is related to the size of the processed data (such as the size of the input data file) and the complexity of the processing algorithm. In the case of complex algorithms and relatively long execution time, task monitoring needs to understand the real-time execution progress of the task / algorithm and estimate how long it will take to complete the task. Real-time task progress indicators are difficult to obtain solely relying on the observerbility of the application itself. SUMMARY

[0006] In view of the above problems, the present application is proposed in order to provide a distributed satellite observation data processing task progress index real-time method to overcome the above problems or at least partially solve the above problems.

[0007] According to one aspect of the present application, a distributed satellite observation data processing task progress index real-time method is provided, the index real-time method comprising:

[0008] Target code instrumentation is performed on the distributed satellite observation data processing task based on the OpenTelemetry framework multilingual feature;

[0009] Real-time reception and persistent storage of Trace type telemetry data sequences are implemented in the service monitoring backend;

[0010] Based on the task data volume and task time consumption relationship model, the task completion time consumption is predicted, and after combining with the received telemetry data, the real-time progress index of the task is dynamically estimated.

[0011] Optionally, the method for establishing the task data volume and task time consumption relationship model comprises:

[0012] Through historical data analysis, test data is analyzed, and a linear regression relationship model between task data volume and task processing time consumption is established.

[0013] Optionally, the target code instrumentation specifically comprises:

[0014] Based on the OpenTelemetry framework, the observation target code segment of the distributed satellite observation data processing task is instrumented with tracking code, and different programming languages correspond to different APIs:

[0015] The location of code instrumentation is arranged according to the needs of satellite payload observation data processing business, requires clear business meaning, realizes measurement of an entire task with one span, realizes span instrumentation of important processing methods, and can also realize multi-span measurement and span nested measurement according to business needs;

[0016] The telemetry data of the distributed satellite observation data processing task is collected through the OTLP collector;

[0017] The task monitoring backend is connected.

[0018] Optionally, the task monitoring backend connection specifically comprises:

[0019] The task monitoring backend realizes the connection of the OTLP collector, and through the API supporting HTTP / gRPC, the telemetry data collected by the OTLP collector is pulled in real time, or through the OTLP exporter, the telemetry data is pushed to the task monitoring backend in real time.

[0020] Optionally, the running state processing of the task specifically includes:

[0021] acquiring telemetry data;

[0022] tracking data storage;

[0023] task completion time consumption prediction;

[0024] task real-time progress estimation.

[0025] Optionally, the acquiring telemetry data specifically includes:

[0026] In the task running state, the task monitoring backend continuously acquires measurement span information of the trace through the OTLP exporter, and the time length between the telemetry start and the telemetry end in the span information is the actual time consumption of the task;

[0027] The span information received in the intermediate time interval can actually reflect the health status of the task.

[0028] Optionally, the tracking data storage specifically includes:

[0029] The trace span information acquired through the telemetry data is format-parsed and persistently stored in the task monitoring backend, and the tracking data items that need to be persistently stored include: trace name trace_name, trace identification trace_id, span name span_name, span ID span_id, parent span ID span_id, span start time start, span end time end, and status status.

[0030] Each time the trace is measured, only one piece of information needs to be persistently saved for a span, and the telemetry sequence information received in the intermediate time interval is used as a basis for judging the health status of the task.

[0031] Optionally, the task completion time consumption prediction specifically includes:

[0032] After the task is started, the task data volume and the task time consumption linear regression relationship model established through historical data are used to perform task calculation time consumption prediction value according to the data volume of the task.

[0033] Optionally, the task real-time progress estimation specifically includes:

[0034] Optionally, the self-adaptive iteration of the task data volume and the task time consumption relationship model specifically includes:

[0035] Through accumulation of actual operation data, the relationship model is iterated according to the actual operation data quantity and the task time consumption, so as to ensure the accuracy of the task time consumption estimation.

[0036] The actual operation data quantity and the task time consumption linear regression equation are updated through a fixed cycle to calculate the predicted value of the task completion time consumption.

[0037] The application provides a distributed satellite observation data processing task progress index real-time method, the index real-time method comprises the following steps: target code insertion is performed on the distributed satellite observation data processing task based on the multilingual characteristics of the OpenTelemetry framework; real-time receiving and persistent storage of the Trace type telemetry data sequence are realized in the service monitoring backend; the task completion time consumption is predicted based on the task data quantity and the task time consumption relationship model, and the real-time progress index of the task is dynamically estimated after being combined with the received telemetry data. The task progress index estimation is expressed in percentage, the index value represents the deviation degree between the actual task execution and the task time consumption estimation value, and the health state of the task can be judged in combination with the receiving condition of the Trace telemetry data sequence, so that the operation and maintenance personnel can take corresponding measures.

[0038] The above description is only a summary of the technical scheme of the application, in order to more clearly understand the technical means of the application, the application can be implemented according to the content of the specification, and in order to make the above and other purposes, characteristics and advantages of the application more obvious and easy to understand, the following specific embodiments of the application are described. BRIEF DESCRIPTION OF DRAWINGS

[0039] In order to more clearly illustrate the technical scheme of the embodiments of the application, the drawings needed in the embodiment description will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the application, and those skilled in the art can obtain other drawings according to these drawings without creating any creative labor.

[0040] Figure 1 A flow chart of a distributed satellite observation data processing task progress index real-time method provided by the embodiment of the application is provided.

[0041] Figure 2 A task data quantity and task processing time consumption linear regression relationship model based on historical data / test data provided by the embodiment of the application is provided.

[0042] Figure 3 A task code insertion position mode schematic diagram according to the business requirement layout provided by the embodiment of the application is provided.

[0043] Figure 4A Trace telemetry data collection and receiving schematic diagram based on an OpenTelemetry framework is provided for an embodiment of the present application.

[0044] Figure 5 A task data volume and task processing time consumption linear regression relationship model updating schematic diagram based on task running data is provided for an embodiment of the present application. DETAILED DESCRIPTION

[0045] Exemplary embodiments of the present disclosure will be described in detail with reference to the accompanying drawings. Although exemplary embodiments of the present disclosure are shown in the drawings, it is understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. Rather, these embodiments are provided so that the present disclosure can be more thoroughly understood and the scope of the present disclosure can be accurately conveyed to those skilled in the art.

[0046] The terms "include" and "have" and any variations thereof in the specification, claims and drawings of the present application are intended to cover the non-exclusive inclusion, for example, the inclusion of a series of steps or units.

[0047] The technical solutions of the present application will be described in further detail below in conjunction with the drawings and embodiments.

[0048] As Figure 1 shown, the present application provides a distributed satellite observation data processing task real-time progress indicator implementation method based on observability technology, target code instrumentation is performed on distributed satellite observation data processing tasks based on the multi-language characteristics of the OpenTelemetry framework, real-time reception and persistent storage of Trace type telemetry data sequences are implemented in the service monitoring backend, and task completion time consumption prediction is performed based on the task data volume and task time consumption relationship model. After combining with the received telemetry data, the real-time progress indicator of the task is dynamically estimated.

[0049] The content of the present application includes:

[0050] 1. Task data volume and task time consumption relationship model establishment.

[0051] Through historical data analysis, in the absence of historical data, test data is also analyzed to establish a linear regression relationship model between the task data volume (data file size) and the task processing time consumption.

[0052] The task data volume and task processing time consumption linear regression relationship model based on historical data / test data is as shown in Figure 2 .

[0053] For example, the regression trend line equation established based on test data is: y = 0.3448x + 330.54, where x is the amount of task data (in megabytes) and y is the task time (in seconds). The task time for any amount of data input can be estimated based on the linear regression equation.

[0054] For example, when the data volume is 1550M, the time taken for the model to complete the predicted task is 864.98 seconds.

[0055] 2. Task code instrumentation.

[0056] 1) Based on the OpenTelemetry framework, code instrumentation is performed on the observation target code segments of distributed satellite observation data processing tasks. Different programming languages ​​correspond to different APIs; for example, in C++:

[0057] auto provider=opentelemetry::trace::Provider::GetTracerProvider();

[0058] auto tracer = provider->GetTracer("target task", "1.0.0");

[0059] trace_id = opentelemetry::tractrace.TraceId(“Target Task Business Unique Identifier”);

[0060] auto span = tracer->StartSpan("Target task measurement span");

[0061] The trace_id is assigned a unique identifier with business significance, such as "satellite model + observation data ID + payload ID", instead of using automatically generated identifiers, which facilitates correlation analysis with other business information.

[0062] The placement of code instrumentation can be flexibly handled, and the layout can be arranged according to the needs of satellite payload observation data processing business. It is required to have clear business meaning, realize the measurement of the entire task in one span, and also realize span instrumentation for important processing methods (algorithms). It can also realize multi-span measurement and Span nesting (parent-child relationship) measurement according to business needs.

[0063] like Figure 3 As shown, this is a schematic diagram of the task code instrumentation pattern arranged according to business requirements.

[0064] Telemetry data for distributed satellite observation data processing tasks are collected uniformly through the OTLP collector.

[0065] 2) Task monitoring backend interface

[0066] The task monitoring backend implements the OTLP collector interface, which pulls the telemetry data collected by the OTLP collector in real time through the HTTP / gRPC API, or pushes it to the task monitoring backend in real time through the OTLP exporter.

[0067] 2、Task running state processing

[0068] 1) Telemetry data acquisition

[0069] In the task running state, the task monitoring backend continuously acquires the measurement span information of the trace through the OTLP exporter. The time interval between the start of telemetry and the end of telemetry in the span information is the actual time consumed by the task. The span information received in the middle time interval can actually reflect the health status of the task.

[0070] As shown in Figure 4 , a schematic diagram of Trace telemetry data collection and reception based on the OpenTelemetry framework.

[0071] 2) Trace data storage

[0072] The Trace Span information (in Json format) acquired through telemetry data is parsed and stored in the task monitoring backend. The trace data items that need to be stored include: trace name (trace_name), trace identifier (trace_id), span name (span_name), span ID (span_id), parent span ID (span_id), span start time (start), span end time (end), and status (status). Each time the trace is measured, only one piece of information needs to be stored for a span. The telemetry sequence information received in the middle time interval can be used as a basis for judging the health status of the task (e.g., if the measurement data is not received for 3 consecutive measurements, it may indicate an abnormal state of the task execution, and the operation and maintenance personnel should pay close attention to it).

[0073] 3) Task completion time prediction

[0074] After the task is started, the task completion time prediction value is calculated based on the data volume of the task and the linear regression relationship model between the task data volume and the task completion time established through historical data.

[0075] 4) Real-time progress estimation of the task

[0076] The real-time progress metric for a task is estimated using the ratio of "task execution time / predicted task completion time". A real-time progress percentage below 100% indicates that the task is still within the predicted completion time, while a percentage above 100% indicates that the task is running later than the predicted completion time, requiring close monitoring of the task logs.

[0077] For example, the data volume of this task is 1550M. According to the regression trend line formula, the predicted completion time is 864.98 seconds. Under normal telemetry data reception, 600 seconds after the start of measurement, the real-time progress percentage of the task is: (600 / 864.98*100)% = 69.37%; when the measurement starts 950 seconds later, it indicates that the task is delayed compared to the predicted completion time, and the real-time progress percentage of the task is: (950 / 831.89*100)% = 109.83%.

[0078] 3. Adaptive iterative model of the relationship between task data volume and task duration, including:

[0079] By accumulating actual operational data, the relationship model is iterated based on the actual data volume and task duration to ensure the accuracy of task duration estimation. The linear regression equation between actual operational data volume and task duration can be updated at fixed intervals (e.g., weekly) to calculate the predicted task completion time.

[0080] like Figure 5 As shown, this is a schematic diagram illustrating the update of the linear regression model of the relationship between task data volume and task processing time based on task execution data.

[0081] For example, the regression trend line equation established based on actual task operation data is: y = 0.2703x + 558.03, where x is the task data volume (unit: megabytes) and y is the task time (unit: seconds). Compared with the regression trend line based on test data, the slope is lower, indicating that the actual operation time is not as ideal as the test.

[0082] Beneficial effects:

[0083] The observability technology for distributed tasks is based on the OpenTelemetry framework and the Otel protocol, ensuring multi-language support and standard protocol support for tasks;

[0084] Telemetry is performed only on trace data of distributed tasks, which has relatively low intrusion into the task code.

[0085] Task code instrumentation is laid out according to business needs, with clear business meaning. It can realize the measurement of the entire task and key methods by a single span, and also supports multi-span measurement and span nesting (parent-child relationship) measurement.

[0086] Based on the actual task time consumption prediction of the task data volume and task processing time consumption relationship model, the predicted value of the task completion time consumption can be calculated after the task start telemetry;

[0087] Real-time task progress index estimation, by combining the Trace data of telemetry and the predicted value of the task completion time consumption based on the model, the real-time task progress index estimation of the task is performed;

[0088] TraceID constraint with business uniqueness, in a distributed computing environment, the correlation analysis with other business information can be realized, and more accurate problem and fault positioning can be provided;

[0089] With the accumulation of actual operation data, the task data volume and task time consumption relationship model is periodically iterated, and the task time consumption estimation result can be closer to the actual time consumption;

[0090] The task progress index estimation expressed by percentage, the index value represents the deviation degree between the actual task execution and the task time consumption estimation value, combined with the Trace telemetry data sequence receiving condition, the health status of the task can be judged, and the corresponding measures can be taken by the operation and maintenance personnel.

[0091] The above specific embodiments further illustrate the purpose, technical scheme and beneficial effects of the present application. It should be understood that the above is only a specific embodiment of the present application, and is not used to limit the protection scope of the present application. Any modification, equivalent replacement, improvement, etc. within the spirit and principles of the present application should be included in the protection scope of the present application.

Claims

1. A distributed satellite observation data processing task progress indicator real-time method, characterized in that, The index real-time method comprises: Based on the OpenTelemetry framework multilingual characteristics, target code instrumentation is performed on the distributed satellite observation data processing task; The target code instrumentation specifically comprises: Based on the OpenTelemetry framework, tracking code instrumentation is performed on the observation target code segment of the distributed satellite observation data processing task, and different programming languages correspond to different APIs: The position of the code instrumentation is arranged according to the satellite payload observation data processing business needs, has a clear business meaning, realizes measurement of the entire task in one span, realizes span instrumentation of important processing methods, and can also realize multi-span measurement and span nested measurement according to the business needs; Telemetry data of the distributed satellite observation data processing task is collected through the OTLP collector; Task monitoring backend connection; Real-time reception and persistent storage of Trace type telemetry data sequences are realized in the service monitoring backend; Based on the task data volume and task time consumption relationship model, the task completion time consumption is predicted, and the real-time progress index of the task is dynamically estimated in combination with the received telemetry data.

2. The method of claim 1, wherein, The method for establishing the task data volume and task time consumption relationship model comprises: Through historical data analysis, test data is analyzed, and a linear regression relationship model between the task data volume and the task processing time consumption is established.

3. The method of claim 1, wherein, The task monitoring backend connection specifically comprises: The task monitoring backend realizes the OTLP collector connection, and realizes real-time pulling of the telemetry data collected by the OTLP collector through the HTTP / gRPC API or real-time pushing to the task monitoring backend through the OTLP exporter.

4. The method of claim 1, wherein, The task running state processing specifically comprises: Obtaining telemetry data; Tracking data storage; Task completion time consumption prediction; Task real-time progress estimation.

5. The method of claim 4, wherein, The obtaining telemetry data specifically comprises: In the task running state, the task monitoring backend continuously obtains the measurement span information of the trace Trace through the OTLP exporter, and the time length between the telemetry start and the telemetry end in the span information is the actual time consumption of the task; The span information received in the intermediate time interval can actually reflect the health status of the task.

6. The method of claim 4, wherein, The tracking data storage specifically comprises: The Trace Span information obtained through the telemetry data is format-analyzed and persistently stored in the task monitoring backend, and the tracking data items that need to be persistently stored include: tracking name trace_name, tracking identifier trace_id, span name span_name, span ID span_id, parent span ID span_id, span start time start, span end time end, and status status; Each time the tracking measurement is performed, only one piece of information needs to be persistently saved for a span Span, and the telemetry sequence information received in the intermediate time interval is used as the basis for judging the health status of the task.

7. The method of claim 4, wherein the method further comprises: The task completion time consumption prediction specifically comprises: After the task is started, according to the data amount of the task, the linear regression relation model of the data amount of the task and the time consumption of the task established by the historical data is used to perform the time consumption prediction value of the task.

8. The method of claim 4, wherein the method further comprises: The real-time progress estimation of the task specifically includes: the real-time progress index estimation of the task is performed by the time consumption prediction value of the task execution and the time consumption prediction value of the task completion.

9. The method of claim 1, wherein, The self-adaptive iteration of the relation model of the data amount of the task and the time consumption of the task specifically includes: According to the actual running data amount and the time consumption of the task, the iteration of the relation model is performed by the accumulation of the actual running data, so as to ensure the accuracy of the time consumption estimation of the task; The linear regression equation of the actual running data amount and the time consumption of the task is updated by a fixed period, and the time consumption prediction value of the task completion is calculated.

Images