A production data detection and analysis system and method based on scheduling tasks

Abstract

The present invention relates to the field of data processing technology, and in particular to a production data detection and analysis system and method based on scheduling tasks. This system includes: acquiring production data based on scheduling tasks, and performing composite intelligent processing on the production data using a bidirectional vector anomaly recognition mechanism and a cross-trajectory consistency calibration algorithm to obtain anomaly data and intelligently processed production data; and adaptively analyzing the intelligently processed production data using a heterogeneous state reconstruction and anomaly factor analysis algorithm to obtain a state anomaly index. This system addresses the technical issues of untimely production data anomaly identification, incomplete anomaly processing, and the lack of causal relationships between multi-task states during the execution of scheduling tasks.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a production data detection and analysis system and method based on scheduling tasks. Background Art

[0002] In modern industrial production systems, with the continuous advancement of equipment networking, process digitization, and task scheduling automation, the management and operational status monitoring of scheduling tasks during production have become core components for ensuring production efficiency and quality. Existing scheduling task monitoring systems typically use methods based on static rules or empirical thresholds to identify anomalies in scheduling data. These methods rely on manually set fixed thresholds to determine upper and lower limits for a single indicator, and any deviation from the preset range is considered an anomaly. However, in real-world production environments, due to the complexity of task types, changing execution environments, and frequent data fluctuations, fixed threshold methods have significant limitations. First, they cannot adapt to the dynamic distribution of task data, resulting in frequent false positives and false negatives. Second, these methods cannot handle the causal logical connections and state coordination relationships between tasks, making them unable to identify hidden anomalies caused by scheduling strategy adjustments or changes in task dependencies. Third, after anomaly identification, they often lack the ability to analyze and repair the outlier data. Anomalous data is directly eliminated or roughly filled in, without a structured anomaly repair mechanism. This leads to structural defects in the input data for subsequent data analysis, scheduling optimization, or prediction models, impacting overall system performance.

[0003] In summary, the above technologies have technical problems such as untimely identification of production data anomalies, incomplete anomaly handling, and lack of causal relationships between multiple task states during the execution of scheduling tasks. Summary of the Invention

[0004] The present invention provides a production data detection and analysis system and method based on scheduling tasks to solve the technical problems of untimely production data anomaly identification, incomplete anomaly processing, and lack of causal association between multiple task states during the execution of scheduling tasks.

[0005] The present invention provides a production data detection and analysis system and method based on scheduling tasks, which specifically includes the following technical solutions:

[0006] A production data detection and analysis method based on scheduling tasks includes the following steps:

[0007] S1. Acquire production data based on scheduling tasks and perform composite intelligent processing on the production data using a bidirectional vector anomaly recognition mechanism and a cross-trajectory consistency calibration algorithm to obtain abnormal data and intelligently processed production data.

[0008] S2. Through heterogeneous state reconstruction and abnormal factor analysis algorithms, adaptive analysis is performed on the production data after intelligent processing to obtain the state abnormality index.

[0009] Preferably, the S1 specifically includes:

[0010] In the implementation process of the bidirectional vector anomaly recognition mechanism and the cross-trajectory consistency calibration algorithm, bidirectional modeling is performed by combining the historical trends and future change trends of production data. A forward sliding window is set to calculate the mean of the state indicator vectors of the production data before the current moment to generate a forward expectation vector. A backward window is then set to calculate the mean of the state indicator vectors of the production data after the current moment to generate a backward calibration vector.

[0011] Preferably, the S1 specifically includes:

[0012] Based on the forward expected vector and the backward calibration vector, the abnormal deviation degree of the trend direction is quantified to obtain the comprehensive trend deviation angle.

[0013] Preferably, the S1 specifically includes:

[0014] Based on the forward expectation vector, the state residual between the current moment and the previous moment is calculated, and the rate of change of the state residual is obtained.

[0015] Preferably, the S1 specifically includes:

[0016] Based on the comprehensive trend deviation angle and the state residual change rate, the residual sensitivity coefficient is introduced to calculate the anomaly score. The anomaly score threshold is introduced, and the anomaly score is compared with the anomaly score threshold. When the anomaly score is greater than the anomaly score threshold, it indicates that the production data is abnormal data, and the abnormal data is processed to obtain the production data after abnormal data processing.

[0017] Preferably, the S1 specifically includes:

[0018] Cross-trajectory consistency calibration is performed on the production data after abnormal data processing. The production data corresponding to tasks with causal relationships are collected. The physical state deviation and execution time difference of the production data are calculated, and the state and time coupling deviation are obtained through integration.

[0019] Preferably, the S1 specifically includes:

[0020] Based on the state and time coupling deviation, the optimal alignment path is obtained through the dynamic time warping algorithm. Based on the optimal alignment path, the production data whose state and time coupling deviation exceeds the preset deviation threshold is calibrated to obtain the calibrated production data. The production data after abnormal data processing and cross-track consistency calibration is used as the production data after intelligent processing.

[0021] Preferably, the S2 specifically includes:

[0022] The state indicator vector of the intelligently processed production data is represented as a task state matrix, and a tensor data structure is constructed in chronological order. Based on the tensor data structure, the average task state matrix is ​​introduced to construct the state disturbance tensor.

[0023] Preferably, the S2 specifically includes:

[0024] Based on the state disturbance tensor, combined with the number of tasks and the number of monitoring indicators, the state anomaly index is calculated; the state anomaly index is compared with the preset monitoring indicator threshold to obtain the alarm result; based on the alarm result, the alarm is triggered, the alarm signal is recorded in the task monitoring table and the alarm information is sent to the personnel terminal; based on the state anomaly index and abnormal data, reports and visualization charts for daily monitoring are created.

[0025] A production data detection and analysis system based on scheduling tasks, including the following parts:

[0026] Production data acquisition module, intelligent data processing module, adaptive analysis module, visualization and reporting module, and alarm and automatic notification module;

[0027] The production data acquisition module acquires production data based on the scheduling task and sends the acquired production data to the intelligent data processing module;

[0028] The intelligent data processing module intelligently processes production data through a bidirectional vector anomaly recognition mechanism and a cross-trajectory consistency calibration algorithm to obtain intelligently processed production data. The intelligently processed production data is sent to the adaptive analysis module, and the abnormal data obtained from the intelligent processing is sent to the visualization and reporting module and the alarm and automated notification module.

[0029] The adaptive analysis module uses heterogeneous state reconstruction and anomaly factor analysis algorithms to adaptively analyze intelligently processed production data to obtain a state anomaly index. This index is then sent to the visualization and reporting module and the alarm and automated notification module.

[0030] The visualization and reporting module creates reports and visualization charts for daily monitoring based on the status anomaly index and abnormal data;

[0031] The alarm and automatic notification module obtains alarm results based on the status anomaly index and abnormal data, combined with the monitoring indicator threshold; triggers the alarm based on the alarm result and sends the alarm information to the staff, and sends the abnormal data to the staff at the same time.

[0032] The beneficial effects of the technical solution of the present invention are:

[0033] 1. By introducing a bidirectional vector anomaly identification mechanism, a forward expected vector and a backward calibration vector are constructed. The angle between the forward expected vector and the backward calibration vector is used to quantify the degree of abnormal deviation in the trend direction. The state residual growth rate is also introduced to form an anomaly identification indicator with both trend and burst dimensions. Compared with traditional one-way sliding window methods, the bidirectional vector anomaly identification mechanism can identify potential abnormal trends earlier and more accurately, improving the response lead time and accuracy of the production data detection and analysis system.

[0034] 2. After anomaly detection is complete, the cross-trajectory consistency calibration algorithm performs vector alignment on the causally dependent scheduling task trajectories. A dynamic time warping algorithm is used to find the optimal alignment path and dynamically correct severely drifted data points. This cross-trajectory consistency calibration algorithm effectively resolves data deviation issues caused by execution delays, drift, or synchronization failures in multi-task scheduling systems, fundamentally improving the accuracy of subsequent anomaly detection and scheduling optimization. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 This is a structural diagram of a production data detection and analysis system based on scheduling tasks according to the present invention;

[0036] Figure 2 This is a flow chart of a production data detection and analysis method based on scheduling tasks described in the present invention. DETAILED DESCRIPTION

[0037] In order to further illustrate the technical means and effects adopted by the present invention to achieve the predetermined purpose of the invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present invention.

[0038] Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs.

[0039] The following describes in detail a specific solution of a production data detection and analysis system and method based on scheduling tasks provided by the present invention with reference to the accompanying drawings.

[0040] Refer to the attached Figure 1 , which shows a structural diagram of a production data detection and analysis system based on scheduling tasks provided by an embodiment of the present invention. The system includes the following parts:

[0041] Production data acquisition module, intelligent data processing module, adaptive analysis module, visualization and reporting module, and alarm and automatic notification module;

[0042] The production data acquisition module acquires production data based on the scheduling task and sends the acquired production data to the intelligent data processing module;

[0043] The intelligent data processing module intelligently processes production data through a bidirectional vector anomaly recognition mechanism and a cross-trajectory consistency calibration algorithm to obtain intelligently processed production data. The intelligently processed production data is sent to the adaptive analysis module, and the abnormal data obtained during the intelligent processing is sent to the visualization and reporting module and the alarm and automatic notification module.

[0044] The adaptive analysis module uses heterogeneous state reconstruction and anomaly factor analysis algorithms to adaptively analyze intelligently processed production data to obtain a state anomaly index. This index is then sent to the visualization and reporting module and the alarm and automated notification module.

[0045] The visualization and reporting module uses data visualization tools or the reporting function of the scheduling platform to create reports and visualization charts for daily monitoring based on the status anomaly index and abnormal data;

[0046] The alarm and automated notification module obtains alarm results based on the status anomaly index and abnormal data, combined with the monitoring indicator threshold; triggers an alarm based on the alarm result and automatically sends it to the staff, and automatically sends the abnormal data to the staff at the same time.

[0047] Refer to the attached Figure 2 , which shows a flow chart of a production data detection and analysis method based on scheduling tasks provided by an embodiment of the present invention, the method comprising the following steps:

[0048] S1. Acquire production data based on scheduling tasks and perform composite intelligent processing on the production data using a bidirectional vector anomaly recognition mechanism and a cross-trajectory consistency calibration algorithm to obtain abnormal data and intelligently processed production data.

[0049] First, we obtain the running process data of each task node in the existing scheduling system, namely the production data, including structured data such as task start timestamp, end timestamp, CPU usage, memory load rate, IO waiting time, task input and output data size and rate; construct the production data into a state vector sequence in the form of a time series, which is recorded as: ,in for The collection of time includes dimensional state indicator vector of production data, is the total number of sampling times in the scheduling period;

[0050] Furthermore, a bidirectional vector anomaly recognition mechanism and a cross-trajectory consistency calibration algorithm are introduced to analyze production data. Perform composite intelligent processing to obtain intelligently processed production data; the bidirectional vector anomaly recognition mechanism and cross-track consistency calibration algorithm, starting from the time evolution trend of multi-dimensional state data, gradually achieve high-accuracy anomaly recognition and data consistency repair of production data through space vector modeling, time series residual analysis and track alignment optimization. The specific implementation process is as follows:

[0051] In order to identify abnormal behaviors in scheduling tasks, it is necessary to combine the historical trend and future trend of production data for bidirectional modeling. First, according to the expert experience method, a length of The forward sliding window is used to calculate the mean of the state indicator vector of the production data before the current moment and generate the forward expectation vector at the current moment , , the forward expectation vector Indicates the state that the production data should reach at the current moment under normal execution conditions; yes The collection of time includes The technical purpose is to construct a time prediction reference point based on historical information to detect the degree of deviation of the status indicator vector of the current production data;

[0052] Then, according to the expert experience method, a length of The backward window of Time has come The state indicator vector of the production data at the moment is averaged to generate the backward calibration vector , , the backward calibration vector It is used to retrospectively evaluate the causal impact of the state indicator vector of current production data on future states under the condition that the production data is known; yes The collection of time includes The production data of the dimension status indicator vector (when it is an offline analysis system, is a known quantity, which is directly obtained from the existing database; when it is an online detection system, It is the value of the future moment predicted by a lightweight prediction model (such as a local linear model or exponential weighted average). The angle between the forward expectation vector and the backward calibration vector is introduced, that is, the comprehensive trend offset angle. To measure the degree of abnormal deviation in trend direction:

[0053]

[0054] in, yes The comprehensive trend deviation angle at the moment indicates the production data corresponding to the scheduling task at the current moment. The angle between the forward desired vector and the backward calibration vector; Represents vector dot product; Indicates the L2 norm of the vector, that is, the Euclidean length of the vector. Comprehensive trend deviation angle This reflects the degree of difference between the forward expected trend and the backward corrected trend. A larger angle indicates a more dramatic trend change. This technique aims to establish a quantitative measure of abnormal deviations based on changes in trend direction.

[0055] In order to avoid considering only the trend direction, the rate of change of state residuals is introduced:

[0056]

[0057] in, Indicates The rate of change of the state residual at the moment; Indicates The state residual at the moment; Indicates The state residual at the moment; the numerator is the difference between the state residual at the current moment and the state residual at the previous moment, reflecting the state residual increase; the denominator is the state residual at the previous moment plus a minimum value (used to prevent the denominator from being zero), Possible values ​​are , which overall expresses the relative rate of change of the state residual increase.

[0058] Based on the comprehensive trend deviation angle and the rate of change of state residuals, the anomaly score calculation formula is defined to obtain the anomaly score: ,in, Indicates Abnormality score at the moment, The residual sensitivity coefficient is used to control the weight ratio of the comprehensive trend deviation angle and the state residual change rate in the anomaly score. It is set according to the expert experience method, and the recommended value range is between 0.5 and 2.0. The anomaly score threshold is generated by MAD (median absolute deviation) or distribution adaptive method. .like , then determine The production data obtained at time is abnormal data, and its corresponding scheduling task is The abnormal state is at the moment, and the abnormal data is deleted or processed using the existing abnormal data processing method to obtain the production data after the abnormal data is processed.

[0059] Furthermore, after the abnormal data processing is completed, the cross-trajectory consistency calibration is performed on the production data after the abnormal data processing. That is, the consistency analysis of the causal relationship between multiple task trajectories is performed to correct the trajectory deviation caused by factors such as delay and drift between multiple tasks. Assume that there is a logical dependency relationship between tasks A and B in the existing scheduling system (for example, the output of A is the input of B), and then their trajectory point sets are collected separately: ;in 、 Task A and Task B are 、 The production data corresponding to each time point, 、 The number of sampling time points of the production data corresponding to task A and task B respectively. The state and time coupling deviation between the tasks corresponding to the production data are defined by Euclidean distance and time-sensitive weighting. :

[0060]

[0061] The first term in the above formula Indicates that task A is in The state indicator vector of the production data corresponding to the time point is the same as that of task B at the The Euclidean distance between the state indicator vectors of the production data corresponding to the time points represents the physical state deviation; the second item is the execution time difference Multiply by the time sensitivity factor , used to emphasize the contribution of time to the state and time coupling deviation; the time sensitivity coefficient , set according to expert experience, value The purpose is to integrate the physical state deviation and execution time difference into a unified evaluation index; based on the state and time coupling deviation, the optimal alignment path is found through the existing dynamic time warping algorithm. When the optimal alignment path is found, the deviation between the state and time coupling deviation exceeds the deviation threshold set according to the expert experience method. The specific calibration formula is as follows:

[0062]

[0063] in, Indicates the calibrated Production data at a point in time; is the calibration intensity coefficient, which is used to control the calibration amplitude and has a value range of ,The closer it is to 1, the stronger the calibration strength is,determined according to the expert experience method.,The goal of this technical step is to maintain the consistency of task trajectories,correct inconsistent states caused by time delay or data loss across tasks, thereby ensuring the,accuracy of subsequent scheduling and analysis;

[0064] The production data after abnormal data processing and cross-trajectory consistency calibration is used as the production data after intelligent processing, providing a high-confidence data basis for the next step of analysis or optimization.

[0065] S2. Through heterogeneous state reconstruction and abnormal factor analysis algorithms, adaptive analysis is performed on the production data after intelligent processing to obtain the state abnormality index.

[0066] The state anomaly index is obtained by adaptively analyzing the intelligently processed production data through heterogeneous state reconstruction and abnormal factor analysis algorithms. The specific process is as follows:

[0067] First, the state indicator vector of the intelligently processed production data at each time point is expressed as a task state matrix , and build tensor data structures in chronological order ; The tensor data structure As the basic data structure for analysis, its physical meaning is to describe the length before the current time point as The evolution trajectory of all scheduling tasks in all indicator dimensions within the time window constitutes the spatiotemporal coupling representation of the scheduling task status.

[0068] Furthermore, based on the traditional time series difference method and residual analysis method, a state disturbance tensor is constructed to quantify the significant deviation of the current task execution state relative to its steady-state trajectory, that is, to calculate the difference between each task state matrix and the average task state matrix in the time window. Tensor perturbation of :

[0069]

[0070] in, It is the composite tensor perturbation on the task dimension and the indicator dimension, that is, the state perturbation tensor; It is The task status matrix collected at each time point; is the average task state matrix within the time window, which is used to describe the steady state of the task; is a very small positive number used to prevent the denominator from being zero or undefined behavior in logarithmic operations. Possible values ​​are ; Represents the Hadamard product, which is used to represent element-wise multiplication. This structure acts on the state difference matrix by introducing square root and logarithmic nonlinear transformation at the same time , achieving multi-scale response modeling to data perturbations, thereby enhancing the sensitivity to small nonlinear perturbations.

[0071] Based on the above state disturbance tensor, the state anomaly index is defined based on the existing tensor norm calculation method and the scale transformation (such as logarithmic transformation) technology in statistics. , used to quantitatively measure the overall deviation of the current production data detection and analysis system, where represents the number of tasks and the number of monitoring indicators, respectively, representing the scale factor. The inclusion of a logarithmic term in this formula compresses the scale factor into a quasi-linear growth, improving the scale invariance of the state anomaly index. The state anomaly index represents the cumulative disturbance intensity of the production data detection and analysis system under historical stable conditions and serves as the basis for subsequent determination of whether the system has reached alarm conditions.

[0072] After the calculation of the abnormal state index is completed, the abnormal state index is compared with the monitoring index threshold set according to the expert experience method. 、 For comparison, if , no alarm occurs, if , a yellow warning is triggered. , a red alert is triggered; the corresponding alarm signal will be recorded in the task monitoring table and automatically sent to the personnel terminal (such as the monitoring screen, SMS channel or email system) through the configured system channel;

[0073] At the same time, based on the status anomaly index and abnormal data, use the data visualization tool or the reporting function of the scheduling platform to create reports and visual charts for daily monitoring.

[0074] In summary, a production data detection and analysis system and method based on scheduling tasks have been completed.

[0075] The order in which the embodiments of the invention are presented is for illustrative purposes only and does not necessarily represent the superiority or inferiority of the embodiments. The processes depicted in the accompanying drawings do not necessarily require the specific order or sequential order shown to achieve the desired results. In certain embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0076] The various embodiments in this specification are described in a progressive manner, and the same or similar parts between the various embodiments can be referred to each other. Each embodiment focuses on the differences from other embodiments.

[0077] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the above embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included in the scope of protection of the present invention.

Claims

1. A production data detection and analysis method based on scheduling tasks, characterized in that: The following steps are involved: S1. Based on the scheduling task, production data is acquired, a bidirectional vector anomaly recognition mechanism and a cross-trajectory consistency calibration algorithm are introduced, and bidirectional modeling is performed based on the historical and future trends of production data. Set a forward sliding window, calculate the mean of the state indicator vector of the production data before the current moment, and generate a forward expectation vector; Then, a backward window is set to calculate the mean of the state indicator vector of the production data after the current moment to generate a backward calibration vector. Based on the forward expected vector and the backward calibration vector, the degree of abnormal deviation in the trend direction is quantified to obtain the comprehensive trend deviation angle. Based on the forward expected vector, the rate of change of the state residual is calculated. The specific formula is: in, Indicates The rate of change of the state residual at the moment; and They are Moment and Production data at all times; and Respectively Moment and The forward expectation vector at time t; and Respectively expressed in Moment and The state residual at the moment; Used to prevent the denominator from being zero; Based on the comprehensive trend deviation angle and the state residual change rate, a residual sensitivity coefficient is introduced to calculate the anomaly score; an anomaly score threshold is introduced, and the anomaly score is compared with the anomaly score threshold. When the anomaly score is greater than the anomaly score threshold, it is indicated that the production data is abnormal data, and the abnormal data is processed to obtain production data after abnormal data processing; Perform cross-trajectory consistency calibration on the production data after abnormal data processing, collect production data corresponding to tasks with causal relationships, calculate the physical state deviation and execution time difference of the production data, and obtain the state and time coupling deviation through integration. The specific formula is: in, Indicates the state and time coupling deviation; Indicates that task A is in The state indicator vector of the production data corresponding to the time point With Task B in The state indicator vector of the production data corresponding to the time point The Euclidean distance between them represents the physical state deviation; is the time sensitivity coefficient; Based on the state-time coupling deviation, the dynamic time warping algorithm is used to obtain the optimal alignment path. Based on the optimal alignment path, the production data whose state-time coupling deviation exceeds the preset deviation threshold is calibrated to obtain the calibrated production data. The production data after abnormal data processing and cross-trajectory consistency calibration is used as the intelligently processed production data. S2. Through heterogeneous state reconstruction and abnormal factor analysis algorithms, adaptive analysis is performed on the production data after intelligent processing to obtain the state abnormality index.

2. The method for detecting and analyzing production data based on scheduling tasks according to claim 1, characterized in that: Said S2 specifically includes: The state indicator vector of the intelligently processed production data is represented as a task state matrix, and a tensor data structure is constructed in chronological order. Based on the tensor data structure, the average task state matrix is ​​introduced to construct the state disturbance tensor.

3. The method for detecting and analyzing production data based on scheduling tasks according to claim 2, characterized in that: Said S2 specifically includes: Based on the state disturbance tensor, combined with the number of tasks and the number of monitoring indicators, the state anomaly index is calculated; the state anomaly index is compared with the preset monitoring indicator threshold to obtain the alarm result; based on the alarm result, the alarm is triggered, the alarm signal is recorded in the task monitoring table and the alarm information is sent to the personnel terminal; based on the state anomaly index and abnormal data, reports and visualization charts for daily monitoring are created.

4. A production data detection and analysis system based on scheduling tasks, applied to the production data detection and analysis method based on scheduling tasks as claimed in claim 1, characterized in that: Includes the following sections: Production data acquisition module, intelligent data processing module, adaptive analysis module, visualization and reporting module, and alarm and automatic notification module; The production data acquisition module acquires production data based on the scheduling task and sends the acquired production data to the intelligent data processing module; The intelligent data processing module intelligently processes production data through a bidirectional vector anomaly recognition mechanism and a cross-trajectory consistency calibration algorithm to obtain intelligently processed production data. The intelligently processed production data is sent to the adaptive analysis module, and the abnormal data obtained from the intelligent processing is sent to the visualization and reporting module and the alarm and automated notification module. The adaptive analysis module uses heterogeneous state reconstruction and abnormal factor analysis algorithms to adaptively analyze intelligently processed production data and obtain a state abnormality index. Send status anomaly index to the visualization and reporting module and the alarm and automated notification module; The visualization and reporting module creates reports and visualization charts for daily monitoring based on the status anomaly index and abnormal data; The alarm and automated notification module generates alarm results based on the status anomaly index and abnormal data, combined with the monitoring indicator threshold; Based on the alarm results, an alarm is triggered and an alarm message is sent to the staff, and the abnormal data is also sent to the staff.

Images