Target detection task execution time prediction method based on big data

Abstract

The invention relates to a target detection task execution time prediction method based on big data. The method comprises the steps of collecting multi-source data related to a target detection task; performing quality optimization processing on the multi-source data to form normalized data; constructing and maintaining a dynamic feature library based on the standardized data, wherein the dynamic feature library comprises static features and dynamic features extracted from the time series data; a hierarchical integration prediction model is used for processing, the model comprises a time sequence sub-model and a static sub-model which are arranged in parallel, the time sequence sub-model processes dynamic features, the static sub-model processes static features, and outputs of the two sub-models are integrated through a fusion layer to obtain a final task execution time prediction value; and dynamically updating the dynamic feature library and the layered integrated prediction model based on the feedback of the predicted value and the actual value. According to the method, the problems of poor data quality, insufficient dynamic feature capture, model updating lagging and the like in the prior art can be solved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of scheduling task prediction, and particularly relates to a target detection task execution time prediction method based on big data. BACKGROUND

[0002] As one of the core applications in the field of computer vision, target detection tasks are widely used in scenarios such as autonomous driving, intelligent monitoring, and industrial quality inspection. With the continuous improvement of task complexity and data scale, how to accurately predict the execution time of target detection tasks has become a key problem for optimizing resource scheduling and improving system efficiency. The accuracy of execution time prediction directly affects the rationality of task scheduling and resource utilization. Inaccurate prediction may lead to task delays, resource waste, or system overload.

[0003] Existing task execution time prediction methods are mostly based on historical averages, expert experience, or simple statistical models, which are difficult to cope with dynamic changes in target detection tasks. For example, the Chinese patent with publication number CN109981749A proposes a cloud workflow task execution time prediction method based on extreme gradient boosting. This method complements missing values through random forests and establishes a prediction model using the XGBoost algorithm. However, this method does not fully consider the real-time changes of dynamic features such as hardware load and network status, and lacks an optimization mechanism for multi-source data quality, resulting in large prediction errors in dynamic environments.

[0004] The Chinese patent with publication number CN112200208A proposes a cloud workflow task execution time prediction method based on multi-dimensional feature fusion, which improves prediction accuracy through columnar extraction and fusion of categorical and numerical features. However, this method does not involve the construction and real-time updating of a dynamic feature library, making it difficult to adapt to frequent fluctuations in hardware load and network status in target detection tasks.

[0005] In addition, existing technologies such as CN112286990A and US20060143538A1 are related to platform job execution time prediction and static timing analysis, respectively. However, the former lacks a mechanism for data quality optimization and dynamic feature weight allocation, and the latter is only applicable to embedded systems and cannot be extended to target detection tasks in a big data environment.

[0006] In summary, the existing technologies have the following defects:

[0007] Data quality is difficult to guarantee, and there is a lack of systematic processing of noise, missing values, and inconsistencies in multi-source data;

[0008] Dynamic feature capture is insufficient, and cannot reflect real-time changes in hardware load, network status, and other environmental changes;

[0009] Feature engineering and model structure are too simple, and the complementarity between temporal features and static features is not fully utilized;

[0010] The model update mechanism is imperfect and cannot adapt to environmental changes during long-term operation.

[0011] Therefore, there is an urgent need for a method to predict the execution time of target detection tasks that can comprehensively optimize data quality, dynamically capture feature changes, and have adaptive update capabilities. Summary of the Invention

[0012] To address the technical problems existing in the prior art, the present invention aims to provide a method for predicting the execution time of target detection tasks based on big data. This method can solve problems such as poor data quality, insufficient dynamic feature capture, and lagging model updates in the prior art, comprehensively optimize data quality, dynamically capture feature changes, and has adaptive update capabilities.

[0013] To achieve the above-mentioned objectives, this invention provides a method for predicting the execution time of target detection tasks based on big data, comprising the following steps:

[0014] Step S1: Collect multi-source data related to the target detection task;

[0015] Step S2: Perform quality optimization processing on the multi-source data to form standardized data;

[0016] Step S3: Construct and maintain a dynamic feature library based on the normalized data. The dynamic feature library contains static features and dynamic features extracted from time-series data.

[0017] Step S4: Process the data using a hierarchical integrated prediction model. The model includes a time-series sub-model and a static sub-model set in parallel. The time-series sub-model processes the dynamic features, and the static sub-model processes the static features. The outputs of the two sub-models are integrated through a fusion layer to obtain the final predicted task execution time.

[0018] Step S5: Based on the feedback between the predicted and actual values, dynamically update the dynamic feature library and the hierarchical integrated prediction model.

[0019] According to one technical solution of the present invention, step S1 includes:

[0020] Step S11: Build a distributed data acquisition framework and collect historical log data of the target detection task through monitoring and acquisition tools;

[0021] Step S12: Use the monitoring and acquisition tool to collect dynamic data on hardware load and network status in real time;

[0022] Step S13: The collected multi-source data is uniformly stored in the distributed file system of the data processing tool, and preliminarily integrated by the data processing tool to form the original data set.

[0023] According to one technical solution of the present invention, step S2 includes:

[0024] Step S21: Employ a dual mechanism of rule validation and statistical anomaly detection, and set reasonable thresholds based on domain knowledge to filter erroneous data;

[0025] Step S22: Use the Isolation Forest algorithm to identify abnormal noise in the data, and use the K Nearest Neighbors algorithm to correct outliers and replace them with reasonable predicted values.

[0026] Step S23: For missing values, use a differentiated strategy based on the data type to complete them;

[0027] Step S24: Standardize the data format and units, and use the Z-score standardization method to normalize continuous features, and use one-hot encoding for categorical features.

[0028] According to one technical solution of the present invention, step S23 includes:

[0029] For numerical data, a time-series-based linear interpolation method is used to supplement it;

[0030] For categorical data, the mode is used to fill the data;

[0031] For samples where the proportion of missing key features exceeds a preset threshold, a data generation model is constructed using a generative adversarial network to complete the missing features.

[0032] According to one technical solution of the present invention, step S3 includes:

[0033] Step S31: Extract the basic parameters of the task and the inherent attributes of the hardware from the normalized data as static features;

[0034] Step S32: Extract hardware load timing features and network state timing features as dynamic features using a sliding time window;

[0035] Step S33: Calculate the task complexity index and hardware resource matching index as derived features;

[0036] Step S34: Construct a stream processing pipeline based on monitoring and acquisition tools to monitor and update the dynamic data of hardware load and network status in real time.

[0037] Step S35: Use an attention mechanism to assign weights to dynamic features;

[0038] Step S36: Use Pearson correlation coefficient to screen features that are highly correlated with execution time, and perform dimensionality reduction on high-dimensional features through principal component analysis.

[0039] According to one technical solution of the present invention, in step S33, the task complexity index includes the product of resolution and target quantity; the hardware resource matching index includes the ratio of model memory requirement to actual available memory.

[0040] According to one technical solution of the present invention, step S4 includes:

[0041] Step S41: Divide the processed dataset into a training set and a validation set according to a preset ratio;

[0042] Step S42: Train the time-series sub-model and the static sub-model using cross-validation, and optimize the model hyperparameters using grid search.

[0043] Step S43: In the fusion layer, the prediction results of the two sub-models are fused using a weighted voting method, wherein the weights are dynamically adjusted according to the prediction accuracy of each sub-model on the validation set.

[0044] According to one technical solution of the present invention, the preset ratio of the dataset is 7:3.

[0045] According to one technical solution of the present invention, step S5 includes:

[0046] Step S51: Deploy the trained ensemble model to an edge computing node or cloud server, receive feature data for new object detection tasks through the API interface, call the latest features in the feature library in real time, input them into the ensemble model and output the predicted values.

[0047] Step S52: Establish a prediction result feedback mechanism, compare the actual execution time of the new task with the predicted value, calculate the prediction error, and periodically use new historical data to incrementally train the model.

[0048] Step S53: Use mean absolute error, root mean square error, and coefficient of determination as evaluation indicators to periodically evaluate the prediction accuracy of the model.

[0049] Step S54: When the prediction error exceeds the preset threshold, the feature library update and model retraining process is automatically triggered.

[0050] According to one technical solution of the present invention, in step S54, the environmental changes that trigger model retraining include hardware upgrades, network topology adjustments, and task type updates.

[0051] Compared with the prior art, the present invention has the following advantages:

[0052] This invention proposes a method for predicting the execution time of target detection tasks based on big data. It combines rule verification and machine learning algorithms to double-filter erroneous data and noise, ensuring data accuracy. It adopts personalized repair strategies such as interpolation and generative models for different types of data to avoid model bias caused by missing values. It unifies data format, units and dimensions to eliminate feature interference caused by data inconsistency, providing a high-quality data foundation for model learning.

[0053] This invention separates dynamic features from static features, captures the temporal evolution of dynamic factors through an LSTM-Transformer model to prevent dynamic information from being masked by static features, and achieves real-time monitoring and updating of dynamic features based on Flink to ensure that the model can obtain the latest environmental status in a timely manner and adapt to dynamic changes in hardware load and network status. The influence of key dynamic factors is enhanced through an attention mechanism, and the prediction results of time series and static features are fused by an integrated model to comprehensively cover the combined impact of dynamic and static factors on execution time and reduce prediction errors. Attached Figure Description

[0054] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the embodiments will be briefly described below. Obviously, the drawings described below are merely some embodiments of the present invention, and those skilled in the art can obtain other drawings based on these drawings without creative effort.

[0055] Figure 1 This is a flowchart illustrating the steps of the target detection task execution time prediction method based on big data in an embodiment of the present invention. Detailed Implementation

[0056] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0057] like Figure 1 As shown, the present invention provides a method for predicting the execution time of a target detection task based on big data, comprising the following steps:

[0058] Step S1: Collect multi-source data related to the target detection task;

[0059] Step S2: Perform quality optimization processing on the multi-source data to form standardized data;

[0060] Step S3: Construct and maintain a dynamic feature library based on the normalized data. The dynamic feature library contains static features and dynamic features extracted from time-series data.

[0061] Step S4: Process the data using a hierarchical integrated prediction model. The model includes a time-series sub-model and a static sub-model set in parallel. The time-series sub-model processes the dynamic features, and the static sub-model processes the static features. The outputs of the two sub-models are integrated through a fusion layer to obtain the final predicted task execution time.

[0062] Step S5: Based on the feedback between the predicted and actual values, dynamically update the dynamic feature library and the hierarchical integrated prediction model.

[0063] By employing multi-source data acquisition, quality optimization, dynamic feature library construction, hierarchical integrated modeling, and dynamic updates, high-precision prediction of target detection task execution time was achieved. This comprehensively covers the entire process of data preprocessing, feature engineering, model building, and optimization, ensuring the accuracy and adaptability of the prediction model in complex and dynamic environments. The combination of a dynamic feature library and a hierarchical model effectively captures the combined impact of temporal changes and static attributes, avoiding the limitations of a single model when processing multi-source heterogeneous data. Furthermore, the feedback mechanism and dynamic update strategy enable the model to continuously learn from environmental changes, maintaining predictive performance over the long term and providing a reliable basis for task scheduling and resource allocation.

[0064] In some embodiments of the present invention, step S1 includes:

[0065] Step S11: Build a distributed data acquisition framework and collect historical log data of the target detection task through monitoring and acquisition tools;

[0066] Step S12: Use the monitoring and acquisition tool to collect dynamic data on hardware load and network status in real time;

[0067] Step S13: The collected multi-source data is uniformly stored in the distributed file system of the data processing tool, and preliminarily integrated by the data processing tool to form the original data set.

[0068] In step S11, the distributed data acquisition framework is built on Apache Flink or Apache Kafka, supporting high concurrency and low latency data inflow. Monitoring and acquisition tools include Prometheus, Grafana, etc., used to collect historical log data such as task parameters (e.g., resolution, algorithm type), execution time, and hardware parameters (e.g., CPU model, GPU model).

[0069] During data acquisition, it is necessary to analyze all factors affecting the execution time of the target detection task in advance to ensure that the collected data covers all dimensions, including core task parameters, hardware status data, network transmission data, and environmental auxiliary data. Dynamic data such as hardware load and network status are highly time-sensitive, and reasonable acquisition intervals must be set based on the frequency of data changes. To avoid the acquisition tool itself consuming excessive system resources, a lightweight acquisition agent should be selected to prevent the acquisition process from affecting the normal execution of the target detection task and causing distortion of execution time records. For wireless transmission or remote acquisition scenarios, encrypted transmission protocols should be used to prevent packet loss and tampering during data transmission, ensuring the accuracy of the original data.

[0070] During the data integration process, the collected data may have different formats, so it is important to define a unified data format in advance; perform preliminary analysis on unstructured data, extract key information and transform it into structured data to facilitate subsequent integration and processing.

[0071] The distributed acquisition framework ensures comprehensive and real-time data collection, covering multi-dimensional information such as task parameters, hardware status, and network conditions, providing a rich data foundation for subsequent feature engineering. Real-time data stream processing avoids the lag of historical data, enabling the feature library to reflect the current system status in a timely manner. Preliminary data integration reduces redundant storage and improves data query and processing efficiency.

[0072] In some embodiments of the present invention, step S2 includes:

[0073] Step S21: Employ a dual mechanism of rule validation and statistical anomaly detection, and set reasonable thresholds based on domain knowledge to filter erroneous data;

[0074] Step S22: Use the Isolation Forest algorithm to identify abnormal noise in the data, and use the K Nearest Neighbors algorithm to correct outliers and replace them with reasonable predicted values.

[0075] Step S23: For missing values, use a differentiated strategy based on the data type to complete them;

[0076] Step S24: Standardize the data format and units, and use the Z-score standardization method to normalize continuous features, and use one-hot encoding for categorical features.

[0077] In step S21, rule validation sets thresholds based on domain knowledge, including that execution time cannot be negative and GPU load will not exceed 100%. Statistical anomaly detection uses the Z-score method to remove data exceeding the 3σ range. In step S22, the Isolation Forest algorithm isolates outliers by constructing random trees to identify anomalous noise; the K-Nearest Neighbors algorithm (K=5) corrects outliers based on the mean of similar samples. In step S23, numerical data (such as memory usage) is supplemented using time-series-based linear interpolation; categorical data (such as algorithm type) uses the mode() function to calculate and fill in the most frequently occurring values ​​in the dataset; for samples with a missing key feature ratio exceeding 30%, a generative adversarial network (GAN) is used to construct a data generation model to complete the missing features. In step S24, execution time is uniformly converted to seconds, and network bandwidth is uniformly converted to Mbps; continuous features are processed using the Z-score normalization method, and categorical features are encoded using one-hot encoding to ensure data consistency in the feature space.

[0078] The Z-score standardization method is a way to transform raw data into a standard normal distribution by using the standard deviation and mean. Its core principle is to adjust the data to a distribution with a mean μ=0 and a standard deviation σ=1. The calculation principle of Z-score is to achieve data transformation through the following formula: z=(x-μ) / σ; where: x is a single sample value in the original data; μ is the mean of all samples; and σ is the standard deviation of all samples.

[0079] The dual anomaly detection and repair mechanism significantly improves data quality and reduces the interference of noise and erroneous data on the model. The differential missing value handling strategy avoids the bias caused by a single method, especially with GAN-assisted key feature completion, improving sample validity. Data standardization and encoding eliminate differences in units and formats, ensuring consistency in the feature space and providing high-quality input for model training.

[0080] In some embodiments of the present invention, step S23 includes:

[0081] For numerical data, a time-series-based linear interpolation method is used to supplement it;

[0082] For categorical data, the mode is used to fill the data;

[0083] For samples where the proportion of missing key features exceeds a preset threshold, a data generation model is constructed using a generative adversarial network to complete the missing features.

[0084] Linear interpolation for numerical data is based on time series, assuming the data changes linearly over a short period. For example, for consecutively missing GPU load values, interpolation is calculated based on data from different time points. Mode imputation for categorical data is based on the most frequently occurring category in the dataset. For example, if "YOLOv5" appears most frequently among algorithm types, then that value is used for imputation. For features with a missing proportion higher than a preset threshold (e.g., 30%) (such as GPU memory usage), a GAN model is used to learn the feature distribution of similar tasks and generate data samples that conform to the true distribution.

[0085] By employing a differentiated imputation strategy, both the characteristics of the data type and the severity of missing data are taken into account, avoiding the distribution bias caused by traditional methods (such as mean imputation). GAN imputation of key features can generate high-quality synthetic data, improving the completeness of samples and the stability of model training. Linear interpolation preserves the continuity of time-series data, while mode imputation maintains the distribution characteristics of categorical data, thus improving the overall reliability of the dataset and the generalization ability of the model.

[0086] In some embodiments of the present invention, step S3 includes:

[0087] Step S31: Extract the basic parameters of the task and the inherent attributes of the hardware from the normalized data as static features;

[0088] Step S32: Extract hardware load timing features and network state timing features as dynamic features using a sliding time window;

[0089] Step S33: Calculate the task complexity index and hardware resource matching index as derived features;

[0090] Step S34: Construct a stream processing pipeline based on monitoring and acquisition tools to monitor and update the dynamic data of hardware load and network status in real time.

[0091] Step S35: Use an attention mechanism to assign weights to dynamic features;

[0092] Step S36: Use Pearson correlation coefficient to screen features highly correlated with execution time, and perform dimensionality reduction on high-dimensional features using principal component analysis. For example, Pearson correlation coefficient can be used to screen features with a correlation higher than 0.7 with execution time. At the same time, PCA dimensionality reduction retains 95% of the variance information, reducing the computational complexity of subsequent modeling.

[0093] In step S31, static features include task resolution, algorithm type, CPU model, and GPU model. In step S32, the sliding time window size is 5 minutes (the window size is adaptively adjusted according to the task execution cycle), extracting hardware load time-series features (such as average load and load fluctuation variance within 5 minutes) and network state time-series features (such as bandwidth change rate and latency peak). In step S33, the task complexity metric is the product of resolution and the number of targets, and the hardware resource matching metric is the ratio of model memory requirements to available memory. In step S34, the stream processing pipeline is implemented based on Flink, monitoring data in real time and updating the feature library. In step S35, the attention mechanism automatically identifies dynamic factors that significantly affect execution time (such as hardware load during peak hours and transmission latency during network congestion), improving the targeting of features.

[0094] By separating and extracting static and dynamic features, the influence of task attributes and environmental states is comprehensively captured. Sliding window temporal feature extraction preserves dynamic change patterns, and derived metrics strengthen the correlation between features and execution time. Real-time stream processing ensures the timeliness of the feature library, and the attention mechanism enhances the contribution of key features. Feature selection and dimensionality reduction reduce redundancy and computational complexity, improving model training efficiency and prediction accuracy.

[0095] In some embodiments of the present invention, in step S33, the task complexity index includes the product of resolution and target quantity; the hardware resource matching index includes the ratio of model memory requirements to actual available memory.

[0096] The task complexity metric (Resolution × ObjectCount) reflects the computational burden of image processing; higher resolution and more objects result in longer execution time. The hardware resource matching metric (Model VRAM Demand / Available VRAM) measures the degree of matching between task requirements and hardware capabilities; a higher ratio indicates more scarce resources, which may prolong execution time.

[0097] Overquantifying task complexity and resource matching transforms abstract task attributes into computable feature metrics, enhancing the interpretability and practicality of feature engineering. These metrics are directly related to execution time, improving the model's understanding of the task's nature and resource status, thereby increasing prediction accuracy and robustness.

[0098] In some embodiments of the present invention, step S4 includes:

[0099] Step S41: Divide the processed dataset into a training set and a validation set according to a preset ratio;

[0100] Step S42: Train the time-series sub-model and the static sub-model using cross-validation, and optimize the model hyperparameters using grid search.

[0101] Step S43: In the fusion layer, the prediction results of the two sub-models are fused using a weighted voting method, wherein the weights are dynamically adjusted according to the prediction accuracy of each sub-model on the validation set.

[0102] In step S41, the dataset is divided into training and validation sets according to a preset ratio. In step S42, the temporal sub-model adopts an LSTM-Transformer hybrid structure, inputting dynamic temporal features (including hardware load time series and network state time series) to capture the temporal evolution of dynamic factors; the static sub-model adopts XGBoost or Random Forest, inputting static and derived features to learn the mapping relationship between task-specific attributes and execution time. Cross-validation methods can use 5-fold cross-validation, and grid search can be used to optimize hyperparameters (such as the number of hidden layer units for temporal feature modeling and the learning rate of the ensemble learning model). In step S43, the weighted voting weights are dynamically adjusted based on the sub-model's prediction accuracy on the validation set; for example, the weights of the temporal sub-model are increased in dynamic environments, and the weights of the static sub-model are increased in stable environments.

[0103] By employing a hierarchical ensemble model, the ability of a time-series model to capture dynamic changes is combined with the ability of a static model to learn inherent attributes, thus improving the comprehensiveness and accuracy of predictions. Cross-validation and grid search optimize model parameters, avoiding overfitting and underfitting. Dynamic weight adjustment enables the model to adapt to changes in feature importance under different environments, further reducing prediction errors.

[0104] In some embodiments of the present invention, the preset ratio of the dataset is 7:3. This ratio, based on experimental verification, ensures sufficient training data while providing enough validation samples to evaluate the model's generalization ability. The training set is used for model learning, and the validation set is used for parameter tuning and early stopping to prevent overfitting. The 7:3 split achieves a balance between model training and validation, making full use of data resources while ensuring the reliability of model evaluation. This ratio is suitable for most object detection task datasets, improving the stability of model training and generalization performance.

[0105] In some embodiments of the present invention, step S5 includes:

[0106] Step S51: Deploy the trained ensemble model to an edge computing node or cloud server, receive feature data for new object detection tasks through the API interface, call the latest features in the feature library in real time, input them into the ensemble model and output the predicted values.

[0107] Step S52: Establish a prediction result feedback mechanism, compare the actual execution time of the new task with the predicted value, calculate the prediction error, and periodically use new historical data to incrementally train the model.

[0108] Step S53: Use mean absolute error, root mean square error, and coefficient of determination as evaluation indicators to periodically evaluate the prediction accuracy of the model.

[0109] Step S54: When the prediction error exceeds the preset threshold, the feature library update and model retraining process is automatically triggered.

[0110] In step S51, model deployment utilizes Docker containerization technology, and the API interface is designed based on RESTful principles. In step S52, a feedback mechanism records the error between the actual execution time and the predicted value; incremental training is triggered after a certain amount of new data has accumulated. In step S53, MAE, RMSE, and R² are evaluated monthly. In step S54, the preset threshold MAE can be set to 5%; exceeding this threshold triggers retraining. Environmental changes include hardware upgrades, network topology adjustments, and task type updates.

[0111] Through model deployment and real-time prediction, the method's practical application value is realized, supporting edge computing and cloud collaboration. Feedback mechanisms and incremental training enable the model to continuously learn from new data and adapt to environmental changes. Multi-metric evaluation comprehensively monitors model performance, and an automatic triggering mechanism ensures the system's adaptability, maintaining high prediction accuracy over the long term.

[0112] In some embodiments of the present invention, the environmental changes that trigger model retraining in step S54 include hardware upgrades, network topology adjustments, and task type updates.

[0113] Hardware upgrades (such as GPU replacements) can alter computing power, network topology adjustments (such as increased bandwidth) can affect data transmission, and task type updates (such as the introduction of new algorithms) can change execution patterns. These changes can lead to feature distribution shifts, triggering retraining to recalibrate the model.

[0114] By identifying key environmental change events and triggering timely model updates, performance degradation caused by environmental changes is avoided. Coverage of hardware, network, and task types ensures the model's adaptability in various changing scenarios, improving the method's practicality and reliability.

[0115] According to one aspect of the present invention, an electronic device is provided, comprising: one or more processors, one or more memories, and one or more computer programs; wherein the processor is connected to the memory, and the one or more computer programs are stored in the memory; when the electronic device is running, the processor executes the one or more computer programs stored in the memory to cause the electronic device to perform a big data-based target detection task execution time prediction method as described in any of the above technical solutions.

[0116] The processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor.

[0117] The memory can be an internal storage unit of the terminal device, such as a hard drive or RAM. Alternatively, it can be an external storage device, such as a plug-in hard drive, Smart Media Card (SMC), Secure Digital (SD) card, or Flash Card. Furthermore, the memory can include both internal and external storage units. The memory is used to store the computer program and other programs and data required by the terminal device. It can also be used to temporarily store data that has been output or will be output.

[0118] According to one aspect of the present invention, a computer-readable storage medium is provided for storing computer instructions, which, when executed by a processor, implement a method for predicting the execution time of a target detection task based on big data, as described in any of the above technical solutions.

[0119] For example, computer-readable storage media can be read-only memory (ROM), random access memory (RAM), read-only optical disc (CD-ROM), magnetic tape, floppy disk, and optical data storage devices. They can be implemented using computer-executable program code, thus allowing them to be stored in a storage device for execution by a computing device, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. Therefore, this invention is not limited to any particular hardware and software combination.

[0120] This invention presents a method for predicting the execution time of target detection tasks based on big data. Through multi-source data acquisition, quality optimization, dynamic feature library construction, hierarchical integrated modeling, and dynamic updates, it achieves high-precision prediction of the execution time of target detection tasks. This method significantly improves data quality, feature engineering, model structure, and adaptability, and is suitable for task scheduling and resource optimization in complex dynamic environments.

[0121] In this invention, the terms "comprising," "including," or any other variations thereof are intended to cover a non-exclusive inclusion, such that a process, method, article, or terminal device that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or terminal device. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or terminal device that includes said element.

[0122] Finally, it should be noted that the above description represents a preferred embodiment of the present invention. It should be pointed out that although preferred embodiments have been described, those skilled in the art, once they understand the basic inventive concept of the present invention, can make various improvements and modifications without departing from the principles described herein. These improvements and modifications should also be considered within the scope of protection of the present invention. Therefore, the appended claims are intended to be interpreted as including both the preferred embodiments and all changes and modifications falling within the scope of the embodiments of the present invention.

Claims

1. A method for predicting the execution time of a target detection task based on big data, characterized in that, Includes the following steps: Step S1: Collect multi-source data related to the target detection task; Step S2: Perform quality optimization processing on the multi-source data to form standardized data; Step S3: Construct and maintain a dynamic feature library based on the normalized data. The dynamic feature library contains static features and dynamic features extracted from time-series data. Step S4: Process the data using a hierarchical integrated prediction model. The model includes a time-series sub-model and a static sub-model set in parallel. The time-series sub-model processes the dynamic features, and the static sub-model processes the static features. The outputs of the two sub-models are integrated through a fusion layer to obtain the final predicted task execution time. Step S5: Based on the feedback between the predicted and actual values, dynamically update the dynamic feature library and the hierarchical integrated prediction model.

2. The method according to claim 1, characterized in that, Step S1 includes: Step S11: Build a distributed data acquisition framework and collect historical log data of the target detection task through monitoring and acquisition tools; Step S12: Use the monitoring and acquisition tool to collect dynamic data on hardware load and network status in real time; Step S13: The collected multi-source data is uniformly stored in the distributed file system of the data processing tool, and preliminarily integrated by the data processing tool to form the original data set.

3. The method according to claim 1, characterized in that, Step S2 includes: Step S21: Employ a dual mechanism of rule validation and statistical anomaly detection, and set reasonable thresholds based on domain knowledge to filter erroneous data; Step S22: Use the Isolation Forest algorithm to identify abnormal noise in the data, and use the K Nearest Neighbors algorithm to correct outliers and replace them with reasonable predicted values. Step S23: For missing values, use a differentiated strategy based on the data type to complete them; Step S24: Standardize the data format and units, and use the Z-score standardization method to normalize continuous features, and use one-hot encoding for categorical features.

4. The method according to claim 3, characterized in that, Step S23 includes: For numerical data, a time-series-based linear interpolation method is used to supplement it; For categorical data, the mode is used to fill the data; For samples where the proportion of missing key features exceeds a preset threshold, a data generation model is constructed using a generative adversarial network to complete the missing features.

5. The method according to claim 1, characterized in that, Step S3 includes: Step S31: Extract the basic parameters of the task and the inherent attributes of the hardware from the normalized data as static features; Step S32: Extract hardware load timing features and network state timing features as dynamic features using a sliding time window; Step S33: Calculate the task complexity index and hardware resource matching index as derived features; Step S34: Construct a stream processing pipeline based on monitoring and acquisition tools to monitor and update the dynamic data of hardware load and network status in real time. Step S35: Use an attention mechanism to assign weights to dynamic features; Step S36: Use Pearson correlation coefficient to screen features that are highly correlated with execution time, and perform dimensionality reduction on high-dimensional features through principal component analysis.

6. The method according to claim 5, characterized in that, In step S33, the task complexity index includes the product of resolution and the number of targets; the hardware resource matching index includes the ratio of model memory requirements to actual available memory.

7. The method according to claim 1, characterized in that, Step S4 includes: Step S41: Divide the processed dataset into a training set and a validation set according to a preset ratio; Step S42: Train the time-series sub-model and the static sub-model using cross-validation, and optimize the model hyperparameters using grid search. Step S43: In the fusion layer, the prediction results of the two sub-models are fused using a weighted voting method, wherein the weights are dynamically adjusted according to the prediction accuracy of each sub-model on the validation set.

8. The method according to claim 7, characterized in that, The preset ratio of the dataset is 7:

3.

9. The method according to claim 1, characterized in that, Step S5 includes: Step S51: Deploy the trained ensemble model to an edge computing node or cloud server, receive feature data for new object detection tasks through the API interface, call the latest features in the feature library in real time, input them into the ensemble model and output the predicted values. Step S52: Establish a prediction result feedback mechanism, compare the actual execution time of the new task with the predicted value, calculate the prediction error, and periodically use new historical data to incrementally train the model. Step S53: Use mean absolute error, root mean square error, and coefficient of determination as evaluation indicators to periodically evaluate the prediction accuracy of the model. Step S54: When the prediction error exceeds the preset threshold, the feature library update and model retraining process is automatically triggered.

10. The method according to claim 9, characterized in that, In step S54, the environmental changes that trigger model retraining include hardware upgrades, network topology adjustments, and task type updates.

Images