Time series data acquisition method, system and equipment based on large model and medium

Abstract

The invention provides a time series data acquisition method, system and device based on a large model and a medium, and belongs to the technical field of data acquisition. The method comprises the following steps: collecting time sequence data from a plurality of data sources, and sending the time sequence data to an edge computing node for data filtering, protocol conversion and time synchronization processing to generate structured time sequence data; preprocessing the structured time sequence data, and generating standardized time sequence data after time sequence segmentation processing; reading the standardized time series data, respectively extracting signal features and deep features in the standardized time series data, fusing the signal features and the deep features, and screening feature vectors from the signal features and the deep features; selecting a neural network model according to the task type, performing distributed training of the model by using the feature vector, optimizing and adjusting hyper-parameters of the model, and generating a target model; and deploying the target model in a production environment, collecting time sequence data in real time, preprocessing the time sequence data, inputting the preprocessed time sequence data into the target model for reasoning, and optimizing the target model according to a reasoning result.

Description

Technical Field

[0001] The present invention belongs to the field of data acquisition technology, and more specifically relates to a time series data acquisition method, system, device and medium based on a large model. Background Art

[0002] Currently, time series data has been widely used in industrial monitoring, financial forecasting, smart cities and other fields. Its collection and processing efficiency directly affects the real-time and reliability of decision-making systems. Traditional time series data collection methods usually rely on a single data source or centralized processing architecture, and there are significant efficiency bottlenecks. For example, in the Internet of Things scenario, massive amounts of sensor data need to be transmitted to the cloud through multiple levels of aggregation nodes, resulting in high network latency and large bandwidth usage; at the same time, the private protocols used by different equipment manufacturers vary significantly, and data analysis needs to rely on customized interfaces, further increasing the complexity of integration. In addition, traditional methods often use batch processing mode for time synchronization and anomaly filtering, which is difficult to meet the low-latency requirements in high-frequency data stream scenarios, resulting in the loss of key information or processing delays.

[0003] In terms of data accuracy, traditional time series data processing technology faces multi-dimensional challenges. On the one hand, multi-source heterogeneous data are susceptible to noise interference, equipment failure or communication interruption during the collection process, resulting in outliers or missing fragments. Traditional methods rely on simple threshold filtering or fixed interpolation algorithms, which make it difficult to distinguish between real anomalies and instantaneous fluctuations, causing the repaired data to deviate from the physical meaning. On the other hand, the non-stationary nature of time series data requires dynamic normalization methods, while traditional normalization algorithms do not consider local distribution characteristics and are prone to introducing distribution offset errors. In addition, clock synchronization across devices and systems relies on manual calibration or coarse-grained protocols, and timestamp errors can reach seconds or even higher, seriously restricting the accuracy of event correlation analysis and prediction.

[0004] The lack of intelligent processing capabilities is another core flaw of traditional technologies. Feature engineering relies heavily on manual experience, such as manually designing statistical features or frequency domain indicators, which makes it difficult to capture complex nonlinear patterns and long-range dependencies. Traditional machine learning models have limited modeling capabilities for high-dimensional time series data and require frequent retraining to adapt to distribution drift. During the model deployment phase, the lack of a dynamic feedback mechanism prevents data drift in the production environment from being captured in real time, and model performance significantly decreases with extended runtime. At the same time, the lack of distributed training and online learning technologies makes model optimization inefficient in large-scale time series data scenarios, making it difficult to meet industrial-grade real-time requirements. Summary of the Invention

[0005] In response to the above problems, the purpose of the present invention is to provide a time series data acquisition method, system, equipment and medium based on a large model, which models the time series data through a deep learning model, thereby optimizing the data acquisition strategy, improving data quality, and making time series data acquisition, processing and analysis more efficient and intelligent.

[0006] To achieve the above-mentioned purpose, the present invention is implemented through the following technical solutions: In a first aspect, an embodiment of the present application provides a time series data collection method based on a large model, comprising: Collect time series data from various data sources and send it to edge computing nodes for data filtering, protocol conversion, and time synchronization to generate structured time series data; Preprocess the structured time series data, perform time series segmentation processing to generate standardized time series data, and save it; Read the standardized time series data, extract the signal features and deep features respectively, fuse the signal features and deep features, and then filter out the feature vectors; Select a neural network model based on the task type, use feature vectors to perform distributed training on the model, optimize and adjust the model's hyperparameters, and generate the target model; Deploy the target model to the production environment, collect time series data in real time, pre-process it and input it into the target model for inference, store the inference results, and optimize the target model based on the inference results.

[0007] In an optional embodiment, the time series data is collected from multiple data sources and sent to the edge computing node for data filtering, protocol conversion and time synchronization processing to generate structured time series data, including: Collect sensor data from sensors through IoT protocols, extract timestamps, device IDs, and measurement values ​​from the sensor data, and generate time series data; Use log parsing tools to extract log data from log files, extract the time series fields, and generate time series data; Send the time series data to the edge computing node, and use threshold filtering to remove invalid values ​​in the time series data; Convert the time series data format to JSON format; Calibrate edge node clocks using the NTP protocol to reduce timestamp errors and generate structured time series data. Structured time series data is transferred in batches to the preset cloud data center for storage through the Kafka message queue or HDFS protocol.

[0008] In an optional embodiment, the preprocessing of the structured time series data, performing time series segmentation processing to generate standardized time series data, and saving the data includes: Read structured time series data through Kafka message queues or HDFS protocols; For normally distributed structured time series data, the Z-score method is used to identify outliers and then repair them by linear interpolation of the previous and next valid values. For non-normally distributed structured time series data, the IQR method is used to identify outliers and fill them with the median; Standardize or normalize structured time series data to unify the data scale; Split structured time series data by business cycle or sliding window to generate standardized time series data and store it in Hive tables or Parquet files.

[0009] In an optional embodiment, the reading of the standardized time series data and extracting the signal features and deep features therein respectively include: Read standardized time series data, calculate the mean, variance, maximum, and minimum values ​​within the time window, and generate time series features; Calculate the power spectral density of the standardized time series data through Fourier transform to generate frequency domain features; Through wavelet transform, the signal of standardized time series data is decomposed into sub-bands of different frequencies, the energy proportion of each sub-band is extracted, and the time-frequency domain features are generated; Use deep learning models to automatically extract complex patterns and long-term dependencies from standardized time series data and generate deep features.

[0010] In an optional embodiment, the fusing of the signal features and the deep features to select a feature vector therefrom includes: Concatenate frequency domain features, time-frequency domain features, time series features, and deep features to generate feature vectors; The importance of the eigenvectors is calculated using the SHAP value, and the eigenvectors whose cumulative contribution is greater than the preset value are screened out and stored in the data table.

[0011] In an optional embodiment, the process of selecting a neural network model according to the task type, performing distributed training of the model using feature vectors, optimizing and adjusting the hyperparameters of the model, and generating a target model includes: Load the feature vectors in the data table; Identify the current task type; if the task type is a prediction task, select the LSTM model; if the task type is a classification task, select the CNN-RNN model; Based on the selected neural network model, PyTorch DDP is used to distribute training tasks on a multi-GPU cluster, feature vectors are used for distributed training of the model, training is accelerated through gradient synchronization and distributed sampling, and the Optuna tool is used to tune hyperparameters. After training is completed, the target model is generated and stored in the model warehouse.

[0012] In an optional embodiment, the target model is deployed in a production environment, time series data is collected in real time, pre-processed and then input into the target model for inference, the inference results are stored, and the target model is optimized according to the inference results, including: Deploy the target model as a RESTful API service to receive real-time time series data collected from multiple data sources in the production environment; After standardizing and windowing the real-time time series data, it is input into the target model for inference within the chassis. The inference results are then stored in the database for use by the business system. Generate training samples from real-time time series data and corresponding inference results; Regularly obtain a preset number of training samples, use the training samples to perform distributed training of the target model, and optimize and adjust the hyperparameters of the target model.

[0013] In a second aspect, an embodiment of the present application further provides a time series data acquisition system based on a large model, comprising: The data acquisition module is used to collect time series data from various data sources and send it to the edge computing node for data filtering, protocol conversion, and time synchronization processing to generate structured time series data; The data preprocessing module is used to preprocess the structured time series data, generate standardized time series data after time series segmentation, and save it; The feature engineering module is used to read standardized time series data, extract signal features and deep features, fuse the signal features and deep features, and filter out feature vectors from them; The model selection and training module is used to select a neural network model based on the task type, perform distributed training of the model using feature vectors, optimize and adjust the model's hyperparameters, and generate the target model; The application deployment module is used to deploy the target model into the production environment, collect time series data in real time, input the data into the target model for inference after preprocessing, store the inference results, and optimize the target model based on the inference results.

[0014] In a third aspect, an embodiment of the present application further provides an electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein when the processor executes the program, the steps of the large model-based time series data acquisition method as described in any one of the above items are implemented.

[0015] In a fourth aspect, an embodiment of the present application further provides a storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of the time series data acquisition method based on a large model as described in any one of the above items are implemented.

[0016] It can be seen from the above technical solutions that the present invention has the following advantages: The large-model-based time series data acquisition method provided in this application efficiently collects time series data from multiple data sources, generates structured time series data through edge computing node processing, and then preprocesses, time-series segmentes, extracts and fuses the data to screen out key feature vectors. Based on the task type, the appropriate neural network model is selected for distributed training and hyperparameter optimization to generate the target model. Finally, the target model is deployed to the production environment for real-time inference, and the model is continuously optimized based on the inference results. This method effectively improves the efficiency of time series data collection, processing, and utilization, enhances the accuracy and adaptability of the model, and provides strong technical support for time series data analysis applications.

[0017] This application integrates multi-source data such as IoT sensors and log files through edge computing nodes, and combines protocol conversion and time synchronization technology to achieve data cleaning, format unification and error calibration, significantly improving the integrity and consistency of time series data.

[0018] This application combines statistical signal features (such as frequency domain and time-frequency domain analysis) with deep patterns extracted by deep learning to form a multi-dimensional feature vector, and screens key features through SHAP values ​​to effectively capture complex time series patterns and improve the model's interpretability and generalization capabilities.

[0019] This application dynamically selects LSTM or CNN-RNN models for prediction / classification tasks, uses a distributed training framework (PyTorch DDP) and a hyperparameter tuning tool (Optuna) to accelerate model convergence, and significantly optimizes computational efficiency and prediction accuracy.

[0020] This application deploys the model as a RESTful API service, supports millisecond-level real-time data stream processing, and forms incremental training samples through online inference result feedback to achieve continuous iterative optimization of the model and ensure long-term prediction stability.

[0021] This application realizes the automation of the entire process from collection, preprocessing, storage to model update. Combining Kafka / HDFS streaming and Hive / Parquet structured storage, it builds a scalable time series data middle platform to provide end-to-end solutions for scenarios such as the Industrial Internet of Things. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the technical solution of the present invention, the following is a brief introduction to the drawings required for the description. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0023] Figure 1 A flowchart of the large model-based time series data acquisition method provided in this application.

[0024] Figure 2 A schematic diagram of the structure of the time series data acquisition system based on a large model provided in this application.

[0025] Figure 3 This is a schematic diagram of the structure of the electronic device provided in this application. DETAILED DESCRIPTION

[0026] In the following detailed description of the specific steps of the time series data acquisition method based on the large model, various embodiments of the present disclosure will be described in more detail. The present disclosure may have various embodiments, and adjustments and changes may be made therein. However, it should be understood that there is no intention to limit the various embodiments of the present disclosure to the specific embodiments disclosed herein, but rather the present disclosure should be understood to cover all adjustments, equivalents and / or alternatives that fall within the spirit and scope of the various embodiments of the present disclosure.

[0027] Hereinafter, the terms "include" or "may include" as used in various embodiments of the present disclosure indicate the presence of disclosed functions, operations, or elements, and do not limit the addition of one or more functions, operations, or elements. In addition, as used in various embodiments of the present disclosure, the terms "include," "have," and their cognates are intended only to indicate specific features, numbers, steps, operations, elements, components, or combinations of the foregoing, and should not be understood as excluding the presence of one or more other features, numbers, steps, operations, elements, components, or combinations of the foregoing, or the possibility of adding one or more features, numbers, steps, operations, elements, components, or combinations of the foregoing.

[0028] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0029] See also Figure 1 FIG. 1 is a flow chart of a method for collecting time series data based on a large model in a specific embodiment, the method comprising: S1: Collect time series data from multiple data sources and send it to edge computing nodes for data filtering, protocol conversion, and time synchronization to generate structured time series data.

[0030] In specific implementations, this step aims to collect time-series data in real time or periodically from various sources (e.g., sensors, log files, web services, etc.). This step often requires addressing issues such as inconsistent data formats and transmission delays. Furthermore, in some cases, preliminary data processing and filtering can be performed close to the data source to reduce data transmission volume and improve response speed.

[0031] Specifically, sensor data is collected from sensors through the Internet of Things protocol, and timestamps, device IDs, and measurement values ​​are extracted from the sensor data to generate time series data. At the same time, log data is extracted from log files through log parsing tools, and the time series fields are extracted to generate time series data.

[0032] Then, the time series data is sent to the edge computing node, and threshold filtering is used to eliminate invalid values ​​in the time series data; the format of the time series data is uniformly converted to JSON format; the edge node clock is calibrated through the NTP protocol to reduce timestamp errors and generate structured time series data.

[0033] Finally, the structured time series data is transferred in batches to the preset cloud data center for storage through the Kafka message queue or HDFS protocol.

[0034] For example, time series data can be collected in real time or periodically from multiple sources such as sensors, log files, and network services through protocols or tools such as MQTT and Flume. To address the problem of inconsistent data formats, protocol conversion is performed at edge nodes (such as gateways), and the data is unified into standard formats such as JSON. The timestamp is calibrated through the NTP protocol to ensure accuracy. At the same time, lightweight preprocessing is implemented, such as filtering invalid data based on thresholds and aggregating statistics by time windows to reduce transmission volume and improve response speed. Finally, the data is transmitted to the cloud or data center through channels such as Kafka message queues or FTP.

[0035] S2: Preprocess the structured time series data, perform time series segmentation processing to generate standardized time series data, and save it.

[0036] In a specific implementation, this step aims to implement data preprocessing, which specifically includes the following aspects: Cleaning: Remove noise and outliers, fill in missing values, and ensure data quality.

[0037] Standardization / normalization: Converting data to the same scale to facilitate subsequent analysis.

[0038] Segmentation: Segment continuous time series data reasonably according to business needs, such as by day, hour, etc.

[0039] First, read structured time series data through the Kafka message queue or HDFS protocol.

[0040] For normally distributed structured time series data, the Z-score method is used to identify outliers and then repair them by linear interpolation of the previous and next valid values. The algorithms used include:

[0041] Where x is the current data point, μ is the sample mean, and σ is the standard deviation. If |Z| > 3, it is considered an outlier and is repaired using linear interpolation of the previous and next valid values.

[0042] For structured time series data with non-normal distribution, the IQR method is used to identify outliers and fill them with the median. Specifically, the IQR method is used to identify data points that exceed the upper and lower bounds, mark them as outliers, and fill them with the median.

[0043] Standardize or normalize structured time series data to unify the data scale. Specifically, select a common standardization or normalization algorithm based on the distribution of the time series data to unify the scale.

[0044] Finally, the structured time series data is segmented by business cycle or sliding window to generate standardized time series data, which is then stored in a Hive table or Parquet file. For example, when segmenting by fixed window, daily data is split into a sample (24 hours x 60 minutes = 1440 time points), labeled with the prediction target for the same time period the next day. When segmenting by sliding window, for structured time series data (such as equipment vibration data), a window size of 100 and a step size of 10 are used to generate overlapping samples (for example, sample 1 contains points 1-100, and sample 2 contains points 11-110).

[0045] S3: Read the standardized time series data, extract the signal features and deep features respectively, fuse the signal features and deep features, and filter out the feature vectors.

[0046] In a specific embodiment, this step aims to extract useful features using statistical methods (e.g., mean, variance), frequency domain transforms (e.g., Fourier transform), and wavelet transforms. Furthermore, it uses deep learning models (e.g., LSTM, CNN) to automatically learn high-level features from the raw data. The two types of features are then fused to create a feature vector.

[0047] Specifically, first extract signal features, including: Read standardized time series data, calculate the mean, variance, maximum, and minimum values ​​within the time window, and generate time series features. At the same time, use linear regression to fit the data within the window, and the slope k represents the trend (if k>0, it means an upward trend).

[0048] The power spectral density of the standardized time series data is calculated by Fourier transform to generate frequency domain features. Among them, the first three frequency components with the highest energy in the spectrum can be extracted as frequency domain features.

[0049] Through wavelet transform, the signal of standardized time series data is decomposed into sub-bands of different frequencies, the energy proportion of each sub-band is extracted, and the time-frequency domain features are generated.

[0050] Then, deep features are extracted, including: Deep learning models are used to automatically extract complex patterns and long-term dependencies from standardized time series data, generating deep features. For example, a CNN-LSTM network is used to automatically extract features. In the CNN layer, a one-dimensional convolution kernel (e.g., kernel_size=5) is used to capture local patterns (e.g., short-term vibration fluctuations). In the LSTM layer, the feature sequence output by the CNN is processed to capture long-term dependencies (e.g., periodic anomalies in device vibration).

[0051] Furthermore, the frequency domain features, time-frequency domain features, time series features and deep features are concatenated to generate a feature vector.

[0052] Finally, the SHAP value is used to calculate the importance of the feature vector, and the feature vectors whose cumulative contribution is greater than the preset value are screened out and stored in the data table (HBase / memory table) for model training.

[0053] S4: Select a neural network model based on the task type, use feature vectors to perform distributed training of the model, optimize and adjust the model's hyperparameters, and generate the target model.

[0054] In a specific implementation, first, the feature vectors in the data table are loaded. Then, the current task type is identified; if the task type is a prediction task, the LSTM model is selected; if the task type is a classification task, the CNN-RNN model is selected; Finally, based on the selected neural network model, PyTorch DDP is used to distribute training tasks on a multi-GPU cluster, feature vectors are used for distributed training of the model, training is accelerated through gradient synchronization and distributed sampling, and the Optuna tool is used to tune hyperparameters. After training is completed, the target model is generated and stored in the model warehouse.

[0055] For example, load feature data from HBase or a memory table, select a model based on the task type (such as LSTM for prediction and CNN-RNN for classification), tune hyperparameters (such as the number of hidden layer units and learning rate) using tools such as Optuna, perform distributed training on a multi-GPU cluster using frameworks such as PyTorchDDP, and optimize the model using the validation set mean square error (MSE) / area under curve (AUC) as the objective function. After training, serialize the optimal model and store it in S3 or a model warehouse to provide an inference engine for application deployment.

[0056] S5: Deploy the target model to the production environment, collect time series data in real time, preprocess it, and then input it into the target model for inference. Store the inference results and optimize the target model based on the inference results.

[0057] In a specific implementation, the purpose of this step is to: deploy the trained model to a production environment, receive new data in real time and make predictions; establish an effective feedback mechanism to continuously update the model to adapt to changes in data distribution.

[0058] Specifically, the target model is first deployed as a RESTful API service to receive real-time time series data collected from multiple data sources in the production environment; the real-time time series data is then standardized and windowed, and then input into the target model for in-chassis reasoning. The output inference results are stored in the database for business system calls; and training samples are generated using the real-time time series data and the corresponding inference results.

[0059] Finally, a preset number of training samples are obtained regularly, and the training samples are used to perform distributed training of the target model to optimize and adjust the hyperparameters of the target model.

[0060] For example, a trained model is deployed as a RESTful API service (e.g., via TorchServe). This service receives real-time production data (e.g., real-time sensor values). After normalization (using the μ and σ values ​​from training) and windowing, it is fed into the model for inference. The output (e.g., failure probability) is stored in Redis / MySQL for use by business systems. Simultaneously, new data is fed back to the training process via Kafka. Flink implements incremental learning (SGD parameter updates). MMD / KL divergence is used to monitor data drift and trigger full retraining, forming a closed loop of "deployment-feedback-optimization."

[0061] In this embodiment, by building a full-process processing system for multi-source heterogeneous time series data, efficient collection and standardized processing are completed at the edge to ensure data consistency, and statistical features and deep learning features are integrated to form a multi-dimensional expression to improve pattern recognition capabilities. Combined with task-driven neural network model selection and distributed training to optimize modeling efficiency, it is finally deployed to form a real-time reasoning closed loop and support online updates, achieving a comprehensive improvement in time series analysis accuracy, system response speed and model adaptability, effectively reducing data governance costs and enabling real-time business decision-making.

[0062] like Figure 2 As shown, the following is an embodiment of the time series data acquisition system based on a large model provided by the embodiment of the present disclosure. This system and the time series data acquisition method based on a large model in the above-mentioned embodiments belong to the same inventive concept. For details not fully described in the embodiment of the time series data acquisition system based on a large model, please refer to the embodiment of the time series data acquisition method based on a large model.

[0063] A time series data acquisition system based on a large model includes: a data acquisition module, a data preprocessing module, a feature engineering module, a model selection and training module, and an application deployment module.

[0064] The data acquisition module is used to collect time series data from multiple data sources and send it to the edge computing node for data filtering, protocol conversion and time synchronization processing to generate structured time series data.

[0065] The data preprocessing module is used to preprocess the structured time series data, generate standardized time series data after time series segmentation, and save it.

[0066] The feature engineering module is used to read standardized time series data, extract the signal features and deep features respectively, fuse the signal features and deep features, and filter out feature vectors from them.

[0067] The model selection and training module is used to select a neural network model based on the task type, use feature vectors to perform distributed training of the model, optimize and adjust the model's hyperparameters, and generate the target model.

[0068] The application deployment module is used to deploy the target model into the production environment, collect time series data in real time, input the data into the target model for inference after preprocessing, store the inference results, and optimize the target model based on the inference results.

[0069] The large-model-based time series data acquisition system provided in this embodiment builds an efficient and adaptive time series analysis system through multi-source heterogeneous data fusion, edge intelligent preprocessing, deep extraction of hybrid features, task-driven dynamic modeling and production closed-loop optimization. It significantly improves the integrity of data acquisition, the richness of feature expression, the efficiency of model training and the accuracy of prediction, while supporting real-time reasoning and online iterative updates, providing an end-to-end full-link intelligent solution for complex time series scenarios.

[0070] Figure 3 A schematic diagram of the hardware structure of an electronic device for implementing various embodiments of the present invention.

[0071] The time series data acquisition method based on a large model provided in the embodiment of the present application can be applied to electronic devices. Those skilled in the art will understand that the electronic device structure involved in the embodiment of the present invention does not constitute a limitation on the electronic device, and the electronic device may include more or fewer components than shown in the figure, or combine certain components, or arrange components differently. In the embodiment of the present invention, the electronic device includes but is not limited to a laptop computer, a desktop computer, a workbench, a personal digital assistant, a server, a blade server, a mainframe computer, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the embodiments of the present application described and / or required herein.

[0072] The electronic device may include a processor, an external memory interface, an internal memory, a universal serial bus (USB) interface, a charging management module, a power management module, a battery, a wireless communication module, an audio module, a speaker, a microphone, a sensor module, a button, a camera, a display, and a SIM card interface, etc.

[0073] A processor may include one or more processing units, such as a central processing unit (CPU), an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU). Different processing units may be independent devices or integrated into one or more processors.

[0074] The processor can be the nerve center and command center of the electronic device. The controller can generate operation control signals based on the instruction opcode and timing signal to complete the control of instruction fetching and execution.

[0075] The processor may also include a memory for storing instructions and data. In some embodiments, the memory in the processor is a cache memory. This memory can store instructions or data that the processor has just used or is reusing. If the processor needs to use the instruction or data again, it can directly call it from the memory. This avoids repeated accesses, reduces processor latency, and thus improves system efficiency.

[0076] The external memory interface can be used to connect an external memory card, such as a MicroSD card, to expand the storage capacity of an electronic device. The external memory card communicates with the processor through the external memory interface, enabling data storage. For example, files such as music and videos can be stored on the external memory card.

[0077] Internal memory can be used to store computer-executable program code, which includes instructions. The processor executes the instructions stored in the internal memory to perform various functional applications and data processing of the electronic device. The internal memory can include a program storage area and a data storage area. The internal memory can include high-speed random access memory and non-volatile memory, such as at least one disk storage device, flash memory device, universal flash storage (UFS), etc.

[0078] The wireless communication function of an electronic device can be implemented through an antenna, a wireless communication module, a modem processor, and a baseband processor.

[0079] Wireless communication modules can provide wireless communication solutions for electronic devices, including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared technology (IR), etc.

[0080] Electronic devices can implement audio functions through audio modules, speakers, receivers, microphones, headphone jacks, and application processors.

[0081] Electronic devices can achieve shooting functions through ISP, camera, video codec, GPU, display and application processor.

[0082] Electronic devices can achieve display functions through GPU, display screen and application processor.

[0083] A GPU is a microprocessor for image processing that connects the display screen to the application processor. The GPU performs mathematical and geometric calculations for graphics rendering. A processor may include one or more GPUs, which execute program instructions to generate or modify display information.

[0084] The display screen is used to display images, videos, etc. The display screen includes a display panel.

[0085] The above-mentioned electronic device realizes the time series data acquisition method based on a large model of the present application through edge intelligent preprocessing of multi-source heterogeneous time series data, deep extraction of hybrid features, task-driven dynamic modeling and continuous optimization of the production closed loop, achieving the beneficial effects of significantly improving the accuracy of time series analysis, reducing data governance costs, enhancing model generalization capabilities and supporting real-time decision-making applications.

[0086] The storage medium provided in this application stores a program product that can implement a time series data acquisition method based on a large model.

[0087] Time series data collection methods based on large models include: Collect time series data from various data sources and send it to edge computing nodes for data filtering, protocol conversion, and time synchronization to generate structured time series data; Preprocess the structured time series data, perform time series segmentation processing to generate standardized time series data, and save it; Read the standardized time series data, extract the signal features and deep features respectively, fuse the signal features and deep features, and then filter out the feature vectors; Select a neural network model based on the task type, use feature vectors to perform distributed training on the model, optimize and adjust the model's hyperparameters, and generate the target model; Deploy the target model to the production environment, collect time series data in real time, pre-process it and input it into the target model for inference, store the inference results, and optimize the target model based on the inference results.

[0088] In some possible implementations, the large model-based time series data acquisition method disclosed herein can be implemented in the form of a program product, which includes program code. When the program product is run on a terminal device, the program code is used to enable the terminal device to execute the steps described in the above "Exemplary Method" section of this specification according to various exemplary implementations of the present disclosure.

[0089] The storage medium of the present disclosure can adopt any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium can be, for example, but not limited to, a system, device or component of electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination thereof. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof.

[0090] The above description of the disclosed embodiments is intended to enable one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be readily apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention is not limited to the embodiments shown herein but is intended to conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A time series data collection method based on a large model, characterized in that: include: Collect time series data from various data sources and send it to edge computing nodes for data filtering, protocol conversion, and time synchronization to generate structured time series data; Preprocess the structured time series data, perform time series segmentation processing to generate standardized time series data, and save it; Read the standardized time series data, extract the signal features and deep features respectively, fuse the signal features and deep features, and then filter out the feature vectors; Select a neural network model based on the task type, use feature vectors to perform distributed training on the model, optimize and adjust the model's hyperparameters, and generate the target model; Deploy the target model to the production environment, collect time series data in real time, pre-process it and input it into the target model for inference, store the inference results, and optimize the target model based on the inference results.

2. The time series data acquisition method based on a large model according to claim 1 is characterized in that: The time series data is collected from various data sources and sent to the edge computing node for data filtering, protocol conversion and time synchronization processing to generate structured time series data, including: Collect sensor data from sensors through IoT protocols, extract timestamps, device IDs, and measurement values ​​from the sensor data, and generate time series data; Use log parsing tools to extract log data from log files, extract the time series fields, and generate time series data; Send the time series data to the edge computing node, and use threshold filtering to remove invalid values ​​in the time series data; Convert the time series data format to JSON format; Calibrate edge node clocks using the NTP protocol to reduce timestamp errors and generate structured time series data. Structured time series data is transferred in batches to the preset cloud data center for storage through the Kafka message queue or HDFS protocol.

3. The time series data acquisition method based on a large model according to claim 2 is characterized in that: The preprocessing of the structured time series data, performing time series segmentation processing to generate standardized time series data, and saving the data, includes: Read structured time series data through Kafka message queues or HDFS protocols; For normally distributed structured time series data, the Z-score method is used to identify outliers and then repair them by linear interpolation of the previous and next valid values. For non-normally distributed structured time series data, the IQR method is used to identify outliers and fill them with the median; Standardize or normalize structured time series data to unify the data scale; Split structured time series data by business cycle or sliding window to generate standardized time series data and store it in Hive tables or Parquet files.

4. The time series data acquisition method based on a large model according to claim 3 is characterized in that: The process of reading the standardized time series data and extracting the signal features and deep features thereof includes: Read standardized time series data, calculate the mean, variance, maximum, and minimum values ​​within the time window, and generate time series features; Calculate the power spectral density of the standardized time series data through Fourier transform to generate frequency domain features; Through wavelet transform, the signal of standardized time series data is decomposed into sub-bands of different frequencies, the energy proportion of each sub-band is extracted, and the time-frequency domain features are generated; Use deep learning models to automatically extract complex patterns and long-term dependencies from standardized time series data and generate deep features.

5. The time series data acquisition method based on a large model according to claim 4 is characterized in that: The step of fusing the signal features and the deep features to select a feature vector includes: Concatenate frequency domain features, time-frequency domain features, time series features, and deep features to generate feature vectors; The importance of the eigenvectors is calculated using the SHAP value, and the eigenvectors whose cumulative contribution is greater than the preset value are screened out and stored in the data table.

6. The time series data acquisition method based on a large model according to claim 5 is characterized in that: The process of selecting a neural network model based on the task type, performing distributed training of the model using feature vectors, optimizing and adjusting the model's hyperparameters, and generating a target model includes: Load the feature vectors in the data table; Identify the current task type; if the task type is a prediction task, select the LSTM model; if the task type is a classification task, select the CNN-RNN model; Based on the selected neural network model, PyTorch DDP is used to distribute training tasks on a multi-GPU cluster, feature vectors are used for distributed training of the model, training is accelerated through gradient synchronization and distributed sampling, and the Optuna tool is used to tune hyperparameters. After training is completed, the target model is generated and stored in the model warehouse.

7. The time series data collection method based on a large model according to claim 6, characterized in that: The target model is deployed in the production environment, time series data is collected in real time, pre-processed and then input into the target model for inference, the inference results are stored, and the target model is optimized based on the inference results, including: Deploy the target model as a RESTful API service to receive real-time time series data collected from multiple data sources in the production environment; After standardizing and windowing the real-time time series data, it is input into the target model for inference within the chassis. The inference results are then stored in the database for use by the business system. Generate training samples from real-time time series data and corresponding inference results; Regularly obtain a preset number of training samples, use the training samples to perform distributed training of the target model, and optimize and adjust the hyperparameters of the target model.

8. A time series data acquisition system based on a large model, characterized in that: The system adopts the time series data acquisition method based on the large model as described in any one of claims 1 to 7; The system comprises: The data acquisition module is used to collect time series data from various data sources and send it to the edge computing node for data filtering, protocol conversion, and time synchronization processing to generate structured time series data; The data preprocessing module is used to preprocess the structured time series data, generate standardized time series data after time series segmentation, and save it; The feature engineering module is used to read standardized time series data, extract signal features and deep features, fuse the signal features and deep features, and filter out feature vectors from them; The model selection and training module is used to select a neural network model based on the task type, perform distributed training of the model using feature vectors, optimize and adjust the model's hyperparameters, and generate the target model; The application deployment module is used to deploy the target model into the production environment, collect time series data in real time, input the data into the target model for inference after preprocessing, store the inference results, and optimize the target model based on the inference results.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the steps of the time series data acquisition method based on a large model as described in any one of claims 1 to 7 are implemented.

10. A storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the time series data acquisition method based on a large model as described in any one of claims 1 to 7 are implemented.

Images