An artificial intelligence model-based automatic data processing system and method

Abstract

The application discloses an automatic data processing system and method based on an artificial intelligence model, and relates to the technical field of data processing.The application comprises a data acquisition module, a data preprocessing module and a system state evaluation module.The data acquisition module is used for acquiring an acquisition interval, acquiring real-time business data according to the acquisition interval, and batch importing offline business data.The data preprocessing module is used for preprocessing the offline business data and the real-time business data.The system state evaluation module is used for acquiring current load data of the system.Through comprehensive analysis and dynamic adjustment of the acquisition interval, business and system data, the application constructs an intelligent closed loop of data acquisition.Data fluctuation and system state double-dimension driving interval optimization can accurately capture business abnormal details, adapt system load capacity, dynamically match data acquisition amount and system processing capacity during data processing, avoid data overload or insufficient acquisition, improve preprocessing efficiency and abnormal detection accuracy, and reduce invalid calculation consumption.

Description

Technical Field

[0001] This invention relates to the field of technology, specifically to an automated data processing system and method based on an artificial intelligence model. Background Technology

[0002] An automated data processing system is an intelligent system that integrates cutting-edge AI technologies such as machine learning, natural language processing, and computer vision to automate the entire process of processing massive amounts of data of various types and formats. It breaks through the limitations of traditional data processing, which relies on manual input, rigid rules, and low efficiency. Through the autonomous learning and iterative optimization of AI models, it achieves end-to-end automation from data collection, cleaning, and transformation to analysis, mining, and output.

[0003] In traditional data processing scenarios, data collection intervals are often fixed, making it difficult to adapt to the dynamic fluctuations of business data and the real-time changes in system load. Too wide an interval may miss details of business anomalies, while too narrow an interval can lead to data redundancy and increased system burden. Furthermore, the lack of linkage analysis between offline and real-time data, and the absence of system load status in the adjustment of collection strategies, results in low data preprocessing efficiency, insufficient anomaly detection accuracy, and reliance on frequent manual intervention to adjust intervals, leading to high maintenance costs and slow response times. These shortcomings fail to meet the demands of modern data processing for efficiency, stability, and intelligence. Therefore, this paper proposes an automated data processing system and method based on an artificial intelligence model to address these issues. Summary of the Invention

[0004] Technical problems to be solved

[0005] To address the shortcomings of existing technologies, this invention provides an automated data processing system and method based on an artificial intelligence model, which solves the problems mentioned in the background section.

[0006] Technical solution

[0007] To achieve the above objectives, the present invention provides the following technical solution: an automated data processing system based on an artificial intelligence model, comprising:

[0008] The system comprises the following modules: a data acquisition module for obtaining the acquisition interval, acquiring real-time business data based on the acquisition interval, and batch importing offline business data; a data preprocessing module for preprocessing offline and real-time business data; a system status assessment module for obtaining current system load data, assessing the computing resource adaptation status based on the current system load data, and evaluating the current system status based on the resource adaptation status; a period adjustment module for comprehensively analyzing offline business data, real-time business data, and current system load data to obtain a time adjustment value, and adjusting the acquisition interval based on the time adjustment value; and a data identification module for acquiring real-time business data based on the adjusted acquisition interval, performing anomaly detection, and storing the data in the database.

[0009] Preferably, the data acquisition module further includes a data storage unit, an interface unit for connecting to external devices, and a data verification unit for detecting data compliance.

[0010] Preferably, the specific steps for preprocessing offline and real-time business data are as follows:

[0011] Both offline and real-time business data are standardized. Offline business data is filled with historical averages, while real-time business data is filled with forward interpolation.

[0012] Preferably, the current system load data includes: CPU utilization, memory utilization, and disk speed.

[0013] Preferably, the specific steps for determining the computing resource adaptation status using the system's current load data are as follows:

[0014] The system status assessment value is obtained by comprehensively calculating the obtained CPU utilization, memory utilization, and disk speed.

[0015] Preferably, the specific steps for assessing the current system status through resource adaptation status are as follows: set an assessment threshold, compare the system status assessment value with the assessment threshold in real time, if the system status assessment value is greater than the assessment threshold, it indicates that the system status is poor, and if the system status assessment value is less than or equal to the assessment threshold, it indicates that the system status is stable.

[0016] Preferably, the specific steps for obtaining the time adjustment value are as follows:

[0017] By comprehensively analyzing offline and real-time business data, the data fluctuation value is obtained. By comprehensively analyzing the collection interval, data fluctuation value, and system status evaluation value, the interval adjustment value is obtained.

[0018] Set an adjustment threshold, combine the current acquisition interval, the interval adjustment value, and the adjustment threshold to obtain a time adjustment value, and adjust the acquisition interval according to the time adjustment value.

[0019] Preferably, the specific steps for obtaining the data fluctuation value are as follows:

[0020] The average values ​​of offline and real-time business data are calculated to obtain the offline business average and the real-time business average. The data fluctuation value is calculated based on the degree of deviation between the two.

[0021] Preferably, the standardization process includes standardizing the current system load data to eliminate the differences in the units of CPU utilization, memory utilization, and disk speed.

[0022] An automated data processing method based on an artificial intelligence model includes the following steps:

[0023] Step 1: Obtain the collection interval. Based on the collection interval, obtain real-time business data and import offline business data in batches.

[0024] Step two: Preprocess the offline and real-time business data;

[0025] Step 3: Obtain the current system load data, calculate the resource adaptation status based on the current system load data, and evaluate the current system status based on the resource adaptation status.

[0026] Step 4: Perform a comprehensive analysis of offline business data, real-time business data, and current system load data to obtain a time adjustment value, and adjust the collection interval based on the time adjustment value;

[0027] Step 5: Acquire real-time business data based on the adjusted collection interval, perform anomaly detection, and store the data in the database.

[0028] Beneficial effects

[0029] The present invention has the following beneficial effects:

[0030] (1) This automated data processing system and method based on artificial intelligence models can integrate scattered hardware indicators such as CPU utilization, memory usage, and disk speed into intuitive quantitative results by calculating system status evaluation values, accurately reflecting the overall load status of the system. It facilitates real-time monitoring of system health, quickly identifies high-risk states and triggers early warnings through preset thresholds, and provides data support for resource scheduling. At the same time, it can analyze load trends by combining historical evaluation values, proactively address periodic peaks, significantly improve system operation and maintenance efficiency and stability, and reduce business interruption losses caused by sudden failures.

[0031] (2) This automated data processing system and method based on artificial intelligence models constructs an intelligent closed loop for data acquisition through comprehensive analysis and dynamic adjustment of acquisition intervals, business data, and system data. The interval optimization is driven by both data fluctuations and system status, which not only accurately captures the details of business anomalies but also adapts to the system load capacity. At the same time, during data processing, the data acquisition volume and system processing capacity can be dynamically matched to avoid data overload or insufficient acquisition, improve preprocessing efficiency and anomaly detection accuracy, and reduce ineffective computational consumption.

[0032] Of course, any product implementing this invention does not necessarily need to achieve all of the advantages described above at the same time. Attached Figure Description

[0033] Figure 1 This is a structural diagram of an automated data processing system and method based on an artificial intelligence model according to the present invention;

[0034] Figure 2 This is a flowchart of an automated data processing method based on an artificial intelligence model according to the present invention. Detailed Implementation

[0035] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0036] This invention provides a technical solution: an automated data processing system based on an artificial intelligence model, comprising:

[0037] The data acquisition module is used to obtain the acquisition interval, acquire real-time business data based on the acquisition interval, and import offline business data in batches.

[0038] The data preprocessing module is used to preprocess offline business data and real-time business data;

[0039] The system status assessment module is used to obtain the current system load data, calculate the resource adaptation status based on the current system load data, and assess the current system status based on the resource adaptation status.

[0040] The period adjustment module is used to comprehensively analyze offline business data, real-time business data, and current system load data to obtain a time adjustment value, and then adjust the collection interval based on the time adjustment value.

[0041] The data identification module is used to acquire real-time business data based on the adjusted collection interval, perform anomaly detection, and store the data in the database. Specifically, it uses an artificial intelligence model to detect anomalies in the data and then stores them.

[0042] Specifically, the data acquisition module further includes a data storage unit, an interface unit for connecting to external devices, and a data verification unit for detecting data compliance. The data storage unit is used to store the acquired business data, offline business data, real-time business data, and historical acquisition interval records. The storage medium is a distributed database used to store time-series data. The interface unit is used to connect to the business system and monitor the server through the interface. The data verification unit is used to detect the rationality of the acquired data and whether the data fields are complete.

[0043] Specifically, the preprocessing steps for offline and real-time business data are as follows:

[0044] Offline and real-time business data are standardized, as are the current system load data. Standardization eliminates the influence of units, allowing for comparison of different data. Offline business data is filled with historical averages. When batch import of offline data results in missing fields, the missing parts can be filled by calculating historical averages. Real-time business data is filled with forward interpolation, specifically by reading real-time data from the time before the missing time from the data cache and filling in the missing data.

[0045] Specifically, the current system load data includes: CPU utilization, memory utilization, and disk speed, wherein CPU utilization, memory utilization, and disk speed can all be collected in real time through operating system commands.

[0046] Specifically, the steps for assessing the current system state through resource adaptation status are as follows:

[0047] The system status assessment value is obtained by comprehensively calculating the obtained CPU utilization, memory utilization, and disk speed.

[0048] The CPU utilization, memory utilization, and disk speed are standardized. Historical fault events in the system log are recorded and extracted to show the CPU utilization, memory utilization, and disk speed at the time of system faults. CPU thresholds, memory thresholds, and disk thresholds are set separately. The extracted CPU utilization, memory utilization, and disk speed are compared with the CPU threshold, memory threshold, and disk threshold in real time. The number of times the CPU, memory, and disk exceed the thresholds are recorded. The number of times the CPU, memory, and disk exceed the thresholds are added together, and the sum is compared with the corresponding number of times the CPU and memory exceed the thresholds to calculate the CPU impact value, memory impact value, and disk impact value, respectively.

[0049] The CPU impact value is obtained as follows:

[0050] ;

[0051] In the formula, This indicates the CPU impact value. The higher the CPU impact value, the greater the impact of CPU overload causing system abnormalities. Indicates the number of times the CPU exceeded its limit. Indicates the number of times memory exceeded the limit. This indicates the number of times the disk exceeded its limit. By calculating the number of times the CPU exceeded its limit, we can assess the degree of system abnormality caused by this number of times.

[0052] Furthermore, the methods for obtaining memory impact values ​​and disk impact values ​​are the same as those for obtaining CPU impact values;

[0053] The system status assessment value is obtained by comprehensively calculating the CPU impact value, memory impact value, and disk impact value together with the standardized CPU utilization, memory utilization, and disk speed.

[0054] The system status evaluation values ​​are obtained in the following ways:

[0055] ;

[0056] In the formula, This represents a system status assessment value, specifically a comprehensive quantitative indicator of the current operating status of the computer system. By integrating load information from key hardware such as CPU, memory, and disk, it intuitively reflects the system's load pressure. Indicates the CPU impact value. Indicates CPU utilization. Indicates the memory impact value. Indicates memory usage. Indicates the disk impact value. This indicates the disk speed.

[0057] Set an evaluation threshold and compare the system status evaluation value with the evaluation threshold in real time. If the system status evaluation value is greater than the evaluation threshold, it indicates that the system status is poor, and an alarm will be issued to warn the staff that the workload is too high and abnormal processes will be terminated. If the system status evaluation value is less than or equal to the evaluation threshold, it indicates that the system status is stable and no adjustment is required.

[0058] It's worth noting that the calculated system status assessment values ​​can integrate disparate hardware metrics such as CPU utilization, memory usage, and disk speed into intuitive quantitative results, accurately reflecting the overall system load status. This facilitates real-time monitoring of system health, quickly identifies high-risk states and triggers alerts through preset thresholds, provides data support for resource scheduling, and allows for load trend analysis by combining historical assessment values. This enables proactive responses to periodic peaks, significantly improving system operation and maintenance efficiency and stability, and reducing business interruption losses due to sudden failures.

[0059] Specifically, the steps for obtaining the time adjustment value are as follows:

[0060] The average values ​​of offline and real-time business data are calculated to obtain the real-time business average and the offline business average. The deviation between the real-time business average and the offline business average is comprehensively calculated to obtain the data fluctuation value.

[0061] The data fluctuation value is obtained in the following ways:

[0062] ;

[0063] In the formula, This represents the data fluctuation value, specifically a measure of the deviation between real-time business data and offline historical data. The larger the data fluctuation value, the more significant the deviation between the current real-time data and historical patterns. This represents the real-time average business value. This represents the average value of offline business operations. By multiplying it by 100%, the data fluctuation value can be converted into a percentage form, making it more intuitive and easier to understand.

[0064] A comprehensive analysis of the data collection interval, offline business data, real-time business data, and system status assessment values ​​is conducted to obtain the interval adjustment value.

[0065] The interval adjustment value is obtained as follows:

[0066] The interval adjustment value is obtained by comprehensively calculating the data fluctuation value and the system status assessment value; the interval adjustment value is obtained as follows: In the formula, Indicates the interval adjustment value. Indicates the data fluctuation value. This represents the system state assessment value;

[0067] Set the adjustment threshold, combine the current acquisition interval, the interval adjustment value, and the adjustment threshold to obtain the time adjustment value, add the current acquisition interval time to the time adjustment value to get the specific acquisition interval to be adjusted to;

[0068] The time adjustment value is obtained as follows:

[0069] ;

[0070] In the formula, This represents the time adjustment value, specifically the final time that needs to be shortened or increased. This indicates the current data collection interval, specifically the interval currently being used by the system. Indicates the interval adjustment value. This indicates that the threshold can be adjusted by setting it. Used to determine the direction of adjustment, and through The settings limit the adjustment range to prevent excessive adjustments from causing system malfunctions.

[0071] After the period adjustment module calculates the new acquisition interval, it pushes a configuration update command to the data acquisition module through the interface unit. The data acquisition module loads the new interval in real time and restarts the acquisition process.

[0072] It is worth noting that an intelligent closed loop for data acquisition has been constructed through comprehensive analysis and dynamic adjustment of the acquisition interval, business data, and system data. The interval optimization is driven by both data fluctuations and system status, which not only accurately captures the details of business anomalies but also adapts to the system load capacity. At the same time, during data processing, the data acquisition volume can be dynamically matched with the system processing capacity to avoid data overload or insufficient acquisition, improve preprocessing efficiency and anomaly detection accuracy, and reduce unnecessary computational consumption.

[0073] An automated data processing method based on an artificial intelligence model includes the following steps:

[0074] Step 1: Obtain the collection interval. Based on the collection interval, obtain real-time business data and import offline business data in batches.

[0075] Step two: Preprocess the offline and real-time business data;

[0076] Step 3: Obtain the current system load data, calculate the resource adaptation status based on the current system load data, and evaluate the current system status based on the resource adaptation status.

[0077] Step 4: Perform a comprehensive analysis of offline business data, real-time business data, and current system load data to obtain a time adjustment value, and adjust the collection interval based on the time adjustment value;

[0078] Step 5: Acquire real-time business data based on the adjusted acquisition interval, perform anomaly detection, and store the data in the database.

[0079] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus.

[0080] The preferred embodiments of the present invention disclosed above are merely illustrative of the invention. These preferred embodiments do not exhaustively describe all details, nor do they limit the invention to the specific implementations described. Clearly, many modifications and variations can be made based on the content of this specification. This specification selects and specifically describes these embodiments to better explain the principles and practical applications of the invention, thereby enabling those skilled in the art to better understand and utilize the invention. The invention is limited only by the claims and their full scope and equivalents.

Claims

1. An artificial intelligence model based automated data processing system, characterized in that, The application relates to a system state evaluation method and device. The application comprises the following steps: A data acquisition module is used to acquire an acquisition interval, acquire real-time service data according to the acquisition interval, and batch import offline service data; A data preprocessing module is used to preprocess the offline service data and the real-time service data; The specific steps of preprocessing the offline service data and the real-time service data are as follows: The offline service data and the real-time service data are subjected to standardization processing, the offline service data is filled with historical mean values, and the real-time service data is filled with forward interpolation values; The standardization processing comprises standardization processing of current system load data, so as to eliminate the dimensional differences of CPU usage rate, memory usage rate and disk rate; A system state evaluation module is used to acquire current system load data, the current system load data comprises CPU usage rate, memory usage rate and disk rate, a computing resource adaptation state is calculated through the current system load data, and a current system state is evaluated through the resource adaptation state; The specific steps of calculating the computing resource adaptation state through the current system load data are as follows: The CPU usage rate, the memory usage rate and the disk rate are comprehensively calculated to obtain a system state evaluation value; The system state evaluation value is obtained in the following mode: The CPU usage rate, the memory usage rate and the disk rate are subjected to standardization processing, the CPU usage rate, the memory usage rate and the disk rate at system fault are recorded and extracted, CPU threshold value, memory threshold value and disk threshold value are respectively set, the extracted CPU usage rate, memory usage rate and disk rate are respectively compared with the CPU threshold value, the memory threshold value and the disk threshold value in real time, the CPU over-standard times, the memory over-standard times and the disk over-standard times are recorded, the CPU over-standard times, the memory over-standard times and the disk over-standard times are added, and the addition results are respectively compared with the corresponding CPU over-standard times and memory over-standard times to obtain CPU influence value, memory influence value and disk influence value; ； In the formula, represents a system state evaluation value, specifically a comprehensive quantitative index of the current running state of the computer system, which intuitively reflects the load pressure of the system by integrating the load information of key hardware such as CPU, memory, and disk, represents a CPU influence value, represents CPU usage, represents a memory influence value, represents memory usage, represents a disk influence value, represents disk speed; The CPU influence value, the memory influence value and the disk influence value are comprehensively calculated with the CPU usage rate, the memory usage rate and the disk rate subjected to standardization processing to obtain the system state evaluation value; The specific steps of evaluating the current system state through the resource adaptation state are as follows: An evaluation threshold value is set, the system state evaluation value is compared with the evaluation threshold value in real time, if the system state evaluation value is greater than the evaluation threshold value, the system state is poor, and if the system state evaluation value is less than or equal to the evaluation threshold value, the system state is stable; A period adjustment module is used to comprehensively analyze the offline service data, the real-time service data and the current system load data, obtain a time adjustment value, and adjust the acquisition interval through the time adjustment value; The specific steps of obtaining the time adjustment value are as follows: The offline service data and the real-time service data are comprehensively analyzed to obtain a data fluctuation value, the acquisition interval, the data fluctuation value and the system state evaluation value are comprehensively analyzed to obtain an interval adjustment value; An adjustment threshold value is set, the current acquisition interval, the interval adjustment value and the adjustment threshold value are comprehensively calculated to obtain the time adjustment value, and the acquisition interval is adjusted according to the time adjustment value. The time adjustment value is obtained in the following manner: ； In the formula, represents the time adjustment value, specifically the time that needs to be shortened or increased finally, represents the current collection interval, specifically the collection interval currently used by the system, represents the interval adjustment value, represents the adjustment threshold, and the set by the adjustment threshold is used to determine the direction of adjustment, and the adjustment range is limited by the setting of . The data recognition module is configured to obtain real-time service data based on the adjusted collection interval, perform anomaly detection, and store the data in a database.

2. The system of claim 1, wherein: The specific steps for obtaining the data fluctuation value are as follows: The mean values of the offline service data and the real-time service data are calculated to obtain offline service mean value and real-time service mean value, and the data fluctuation value is calculated by the deviation degree of the two.

3. The system of claim 1, wherein: The data collection module further includes a data storage unit, an interface unit for interfacing with external devices, and a data verification unit for detecting data compliance.

4. An artificial intelligence model-based automatic data processing method for implementing the artificial intelligence model-based automatic data processing system of any one of claims 1-3, characterized in that: The method comprises the following steps: Step one, obtain the collection interval, obtain real-time service data according to the collection interval, and batch import offline service data; Step two, pre-process the offline service data and the real-time service data; The specific steps for pre-processing the offline service data and the real-time service data are as follows: The offline service data and the real-time service data are standardized, and the offline service data is filled with historical mean values, and the real-time service data is filled with forward interpolation values; The standardization process includes standardizing the current system load data to eliminate the dimensional differences of CPU usage, memory usage, and disk speed; Step three, obtain the current system load data, which includes CPU usage, memory usage, and disk speed, and calculate the resource adaptation state based on the current system load data, and evaluate the current system state based on the resource adaptation state; The specific steps for calculating the resource adaptation state based on the current system load data are as follows: The CPU usage, memory usage, and disk speed are calculated to obtain a system state evaluation value; The system state evaluation value is obtained in the following manner: The CPU usage, memory usage, and disk speed are standardized, the CPU usage, memory usage, and disk speed at system failure are recorded and extracted, CPU threshold, memory threshold, and disk threshold are set, the extracted CPU usage, memory usage, and disk speed are compared with the CPU threshold, memory threshold, and disk threshold in real time, the number of times of CPU exceeding, the number of times of memory exceeding, and the number of times of disk exceeding are recorded, the number of times of CPU exceeding, the number of times of memory exceeding, and the number of times of disk exceeding are added, and the sum is compared with the corresponding number of times of CPU exceeding, the number of times of memory exceeding, and the number of times of disk exceeding to obtain CPU influence value, memory influence value, and disk influence value; The CPU influence value, memory influence value, and disk influence value are calculated with the standardized CPU usage, memory usage, and disk speed to obtain a system state evaluation value; ； In the formula, represents a system state evaluation value, specifically a comprehensive quantitative index of the current running state of the computer system, which intuitively reflects the load pressure of the system by integrating the load information of key hardware such as CPU, memory, and disk, represents a CPU influence value, represents CPU usage, represents a memory influence value, represents memory usage, represents a disk influence value, represents disk speed; The specific steps for evaluating the current system state based on the resource adaptation state are as follows: An evaluation threshold is set, the system state evaluation value is compared with the evaluation threshold in real time, if the system state evaluation value is greater than the evaluation threshold, it indicates that the system state is poor, and if the system state evaluation value is less than or equal to the evaluation threshold, it indicates that the system state is stable; Step four, comprehensive analysis is made on the offline service data, real-time service data and current system load data to obtain a time adjustment value, and the collection interval is adjusted through the time adjustment value; The specific steps of obtaining the time adjustment value are as follows: Comprehensive analysis is made on the offline service data and real-time service data to obtain a data fluctuation value, and the collection interval, data fluctuation value and system state evaluation value are comprehensively analyzed to obtain an interval adjustment value; An adjustment threshold is set, the current collection interval, interval adjustment value and adjustment threshold are comprehensively calculated to obtain a time adjustment value, and the collection interval is adjusted according to the time adjustment value; The time adjustment value is obtained in the following manner: ； In the formula, represents the time adjustment value, specifically the time that needs to be shortened or increased finally, represents the current collection interval, specifically the collection interval currently used by the system, represents the interval adjustment value, represents the adjustment threshold, and the is set to determine the direction of adjustment, and the adjustment amplitude is limited by the setting of . Step five, real-time service data is obtained based on the adjusted collection interval, abnormality detection is performed and the data is stored in a database.

Images