Cloud computing big data real-time processing method and system fused with AI technology

Abstract

The invention discloses a cloud computing big data real-time processing method and system fused with an AI technology, and relates to the technical field of big data real-time processing methods, and the method comprises the steps: collecting original data from a plurality of heterogeneous data sources through employing a data access interface, and obtaining an original data set; preprocessing the original data set by adopting a distributed computing framework, and obtaining a preprocessed data set; constructing a load state prediction model based on a machine learning algorithm and the preprocessed data set, inputting the preprocessed data set, and outputting a load state prediction result; adjusting the cloud computing resource allocation quantity in the original data set based on the load state prediction result to obtain a resource allocation strategy; establishing a data processing channel by adopting a stream processing technology based on a resource allocation strategy, and executing an identification task in combination with an AI algorithm to obtain an analysis result; generating a decision report of the specific application scene based on the analysis result; the Kafka Connect connector is configured to be connected with a plurality of heterogeneous data sources, so that efficient integration of multi-source data is realized.

Description

Technical Field

[0001] The present invention relates to the technical field of big data real-time processing methods, and particularly to a cloud computing big data real-time processing method and system integrating AI technology. Background Art

[0002] The big data real-time processing method technology is an advanced framework focusing on processing large-scale data streams, aiming to achieve rapid collection, efficient processing, and instant analysis of massive data to support real-time decision-making. This technology utilizes a distributed computing framework and a message queue system, which can seamlessly integrate data from multiple heterogeneous data sources and achieve real-time data processing through a stream processing engine.

[0003] In the technical field of big data real-time processing methods, the load status prediction methods in traditional methods are often not accurate enough to effectively cope with dynamically changing workloads, resulting in unreasonable resource allocation, affecting system performance and service quality. Moreover, in traditional cloud computing environments, resource allocation is static and cannot be adjusted according to the real-time load status, leading to resource waste or insufficiency and affecting system efficiency. At the same time, in a big data environment, traditional batch processing methods are difficult to meet real-time requirements and cannot respond promptly to rapidly changing data streams. Summary of the Invention

[0004] In view of the above existing problems, the present invention is proposed.

[0005] Therefore, the present invention provides a cloud computing big data real-time processing method integrating AI technology to solve the problem that the load status prediction methods in traditional methods are often not accurate enough to effectively cope with dynamically changing workloads, resulting in unreasonable resource allocation, affecting system performance and service quality.

[0006] To solve the above technical problems, the present invention provides the following technical solutions: In a first aspect, the present invention provides a cloud computing big data real-time processing method integrating AI technology, which includes: Collecting raw data from multiple heterogeneous data sources using a data access interface to obtain a raw data set; Preprocessing the raw data set using a distributed computing framework to obtain a preprocessed data set; Constructing a load status prediction model based on a machine learning algorithm and the preprocessed data set, inputting the preprocessed data set, and outputting a load status prediction result; Adjusting the cloud computing resource allocation amount in the raw data set based on the load status prediction result to obtain a resource allocation strategy; Based on the resource allocation strategy, building a data processing channel using stream processing technology and performing an identification task in combination with an AI algorithm to obtain an analysis result; Generate a decision report for a specific application scenario based on the analysis results.

[0007] As a preferred solution of the cloud computing big data real-time processing method integrating AI technology according to the present invention, wherein: the step of collecting raw data from multiple heterogeneous data sources using a data access interface to obtain a raw data set is as follows: Configure Kafka Connect connectors to connect to each heterogeneous data source; The heterogeneous data sources include relational databases, NoSQL databases, and log files; Convert the relational databases, NoSQL databases, and log files in the heterogeneous data sources into message formats in Kafka topics, and obtain a raw data set.

[0008] As a preferred solution of the cloud computing big data real-time processing method integrating AI technology according to the present invention, wherein: the step of preprocessing the raw data set using a distributed computing framework to obtain a preprocessed data set is as follows: Use Spark SQL to clean the noise data in the raw data set and remove incomplete or incorrect data records; Use Spark's DataFrame API to convert the data into a unified format; For records with missing values, use a statistical model-based method to fill them, and obtain a preprocessed data set.

[0009] As a preferred solution of the cloud computing big data real-time processing method integrating AI technology according to the present invention, wherein: the step of constructing a load status prediction model based on machine learning algorithms and the preprocessed data set, inputting the preprocessed data set, and outputting a load status prediction result is as follows: Perform feature extraction and transformation on the preprocessed data set; Select the Gradient Boosting Decision Tree (GBDT) as the load status prediction model; Use the load status prediction model to predict the load status of new data points and calculate the load status prediction result.

[0010] As a preferred solution of the cloud computing big data real-time processing method integrating AI technology according to the present invention, wherein: the step of adjusting the allocation amount of cloud computing resources in the raw data set based on the load status prediction result to obtain a resource allocation strategy is as follows: Based on the load status prediction result , define a resource demand function to calculate the amount of cloud computing resources required at each time point; According to the resource demand function , adjust the allocation amount of cloud computing resources; Let is the total amount of cloud computing resources at the current moment ; is the amount of resources actually allocated to the platform; Based on resource allocation is performed on the platform, and a resource allocation strategy is obtained.

[0011] As a preferred solution of the cloud computing big data real-time processing method integrating AI technology according to the present invention, wherein: based on the resource allocation strategy, a data processing channel is built using stream processing technology, and an identification task is executed in combination with an AI algorithm to obtain an analysis result. The specific steps are as follows: Configure the data processing channel using Apache Kafka Streams; Let represent the stream processing channel, represent the input data stream; Build an AI pre-training model based on the AI algorithm , and in the stream processing channel, load the pre-trained AI model to execute complex pattern recognition tasks on each data record; Summarize the analysis results to obtain an analysis result set .

[0012] As a preferred solution of the cloud computing big data real-time processing method integrating AI technology according to the present invention, wherein: generating a decision report for a specific application scenario based on the analysis result. The specific steps are as follows: Based on the original data set and the analysis result set , define decision rules and thresholds; Generate a decision report according to the application decision rules.

[0013] In a second aspect, the present invention provides a cloud computing big data real-time processing system integrating AI technology, including: a data collection module, a preprocessing module, a load prediction module, a resource allocation module, an analysis module, and a decision generation module; The data collection module is used to configure a Kafka Connect connector to connect with multiple heterogeneous data sources, and convert the data in the data sources into the message format in the Kafka topic to obtain an original data set; The preprocessing module is used to clean the original data set using Spark SQL, remove noise and incomplete records, and convert the data into a unified format using the DataFrame API of Spark; The load prediction module is used to select the Gradient Boosting Decision Tree GBDT as a machine learning algorithm to build a load status prediction model, and perform feature extraction and conversion on the preprocessed data set; The resource allocation module is used to define a resource demand function based on the load status prediction result, calculate the amount of cloud computing resources required at each time point, and adjust the allocation amount of cloud computing resources according to the resource demand function; The real-time analysis module is used to configure a data processing channel using Apache Kafka Streams and load a pre-trained AI model in the stream processing channel; The decision generation module is used to define decision rules and thresholds based on the original data set and the analysis result set, and generate a decision report for a specific application scenario according to the applied decision rules.

[0014] In a third aspect, the present invention provides a computer device, including a memory and a processor, where the memory stores a computer program, and: when the computer program is executed by the processor, any step of the cloud computing big data real-time processing method integrating AI technology as described in the first aspect of the present invention is implemented.

[0015] In a fourth aspect, the present invention provides a computer-readable storage medium, on which a computer program is stored, and: when the computer program is executed by the processor, any step of the cloud computing big data real-time processing method integrating AI technology as described in the first aspect of the present invention is implemented.

[0016] The beneficial effects of the present invention are as follows: By configuring the Kafka Connect connector to connect with multiple heterogeneous data sources, the efficient integration of multi-source data is achieved. By selecting the Gradient Boosting Decision Tree (GBDT) as the load status prediction model and performing feature extraction and transformation on the preprocessed data set, the accurate prediction of future load status is realized. The accurate load status prediction helps the platform to identify peaks or valleys in resource requirements in advance, thereby optimizing the resource allocation strategy, avoiding resource waste or insufficiency, improving the response speed and service quality of the method. By defining a resource demand function based on the load status prediction result and adjusting the allocation amount of cloud computing resources according to the resource demand function, the dynamic optimization of resources is achieved. The dynamic adjustment mechanism enables the platform to maximize resource utilization while meeting real-time requirements, reduce unnecessary expenses, and ensure stable service under high load conditions. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following will briefly introduce the drawings required for the description of the embodiments. Obviously, the following drawings are only some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0018] Figure 1Flowchart of the cloud computing big data real-time processing method integrating AI technology in Embodiment 1.

[0019] Figure 2 Schematic diagram of the cloud computing big data real-time processing system integrating AI technology in Embodiment 1. Detailed implementation manners

[0020] To make the above objects, features, and advantages of the present invention more apparent and understandable, the following detailed description of the specific implementation manners of the present invention will be given with reference to the accompanying drawings of the specification.

[0021] In the following description, many specific details are set forth to fully understand the present invention. However, the present invention can also be implemented in other ways different from those described herein. Those skilled in the art can make similar generalizations without departing from the connotation of the present invention. Therefore, the present invention is not limited by the specific embodiments disclosed below.

[0022] Secondly, the so-called "one embodiment" or "embodiment" herein refers to a specific feature, structure, or characteristic that can be included in at least one implementation manner of the present invention. The phrase "in one embodiment" appearing in different places in this specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment that excludes other embodiments.

[0023] This embodiment also provides a cloud computing big data real-time processing system integrating AI technology, including: A data acquisition module, a preprocessing module, a load prediction module, a resource allocation module, an analysis module, and a decision generation module; The data acquisition module is used to configure the Kafka Connect connector to connect with multiple heterogeneous data sources, and convert the data in the data sources into the message format in the Kafka topic to obtain the original data set; The preprocessing module is used to clean the original data set using Spark SQL to remove noise and incomplete records, and convert the data into a unified format using the DataFrame API of Spark; The load prediction module is used to select Embodiment 1 and refer to Figure 1 and Figure 2 , which is the first embodiment of the present invention. This embodiment provides a cloud computing big data real-time processing method integrating AI technology, including the following steps: S1. Collect the original data from multiple heterogeneous data sources using the data access interface to obtain the original data set; Furthermore, configure the Kafka Connect connector to connect with each heterogeneous data source; The heterogeneous data sources include relational databases, NoSQL databases, and log files; Convert relational databases, NoSQL databases, and log files in heterogeneous data sources into message formats in Kafka topics, and obtain the original dataset; It should be noted that by configuring the Kafka Connect connector to seamlessly connect with various heterogeneous data sources, the present invention can ensure that data from different sources enters the system in a unified message format. This design not only improves the flexibility and efficiency of data integration but also provides a high-quality basic dataset for subsequent data processing, avoiding processing errors caused by inconsistent data formats.

[0024] S2. Use a distributed computing framework to preprocess the original dataset and obtain the preprocessed dataset; Furthermore, use Spark SQL to clean the noise data in the original dataset and remove incomplete or incorrect data records; Use the DataFrame API of Spark to convert the data into a unified format; For records with missing values, use a statistical model-based method to fill them and obtain the preprocessed dataset; It should be noted that in the data preprocessing stage, using Spark SQL and the DataFrame API for data cleaning and format conversion can efficiently remove noise and incomplete data records, convert the data into a unified format, and fill in missing values using a statistical model-based method, further improving the data quality, enabling subsequent machine learning models to be trained and predicted more accurately, and thus improving the overall performance and reliability of the system.

[0025] S3. Build a load status prediction model based on machine learning algorithms and the preprocessed dataset, input the preprocessed dataset, and output the load status prediction result; Furthermore, perform feature extraction and transformation on the preprocessed dataset; Select the Gradient Boosting Decision Tree (GBDT) as the load status prediction model, and the expression is: ; where, is the load status prediction model, is a single weak learner, is the parameter of this weak learner; Use the load status prediction model to predict the load status of new data points and calculate the load status prediction result. The expression is: ; where, is the predicted load status value of the new data point, is the prediction result of the load status prediction model applied to new data points; It should be noted that the Gradient Boosting Decision Tree (GBDT) is selected as the load status prediction model because of its high prediction accuracy and robustness. By performing feature extraction and transformation on the preprocessed data set and applying the GBDT model for training and prediction, it is possible to effectively identify the changing trends of future load statuses and provide a key basis for dynamic resource allocation, which helps optimize system resource allocation, improve response speed, and enhance service quality.

[0026] S4. Adjust the allocation amount of cloud computing resources in the original data set based on the load status prediction result to obtain a resource allocation strategy; Furthermore, based on the load status prediction result , define a resource demand function to calculate the amount of cloud computing resources required at each time point. The expression is:

[0027] where is the resource demand calculated based on the load status prediction value, and are adjustment factors used to adjust the relationship between resource demand and load status; According to the resource demand function , adjust the allocation amount of cloud computing resources; Let be the total amount of cloud computing resources at the current time , and be the amount of resources actually allocated to the platform. The expression is: where is the actual allocated resource amount determined according to the current total resource amount and the resource demand function, ensures that the actually allocated resource amount does not exceed the maximum available resource amount of the system currently, and is the total available resource amount of the system currently; Based on perform resource allocation for the platform and obtain a resource allocation strategy;

[0028] S5. Based on the resource allocation strategy, build a data processing channel using stream processing technology, and combine with AI algorithms to execute recognition tasks to obtain analysis results; Furthermore, configure the data processing channel using Apache Kafka Streams; Let represent the stream processing channel, represent the input data stream, and the expression is: ; where is the stream processing channel created according to the input data stream and the resource configuration ; is the resource allocation strategy determined in the previous step; Build an AI pre-trained model based on AI algorithms, and in the stream processing channel, load the pre-trained AI model to perform complex pattern recognition tasks on each data record. The expression is: ; where is the recognition result of the th data record after being processed by the stream processing channel, is the result of the AI pre-trained model processing the data record after stream processing; Summarize the analysis results and obtain an analysis result set ; It should be noted that configuring the data processing channel using Apache Kafka Streams and loading the pre-trained AI model to perform complex pattern recognition tasks on each data record can achieve real-time data processing and analysis. This method not only improves the speed and accuracy of data processing but also can timely discover potential problems and opportunities. By summarizing all recognition results and generating an analysis result set, the system can provide strong data support for subsequent decision-making.

[0029] S6. Generate a decision report for a specific application scenario based on the analysis results; Furthermore, based on the original data set and the analysis result set , define decision rules and thresholds. The expression is: ; where is the specific action determined based on the analysis result set and the threshold , is the specific measure to be taken when the corresponding conditions are met; Generate a decision report according to the decision rules applied. It should be noted that by defining decision rules and thresholds, and automatically generating a decision report for a specific application scenario based on the analysis results, the method can provide actionable suggestions in a timely manner. The method not only improves the scientificity and accuracy of decision-making, but also helps users quickly respond to changes and make optimal choices. The generated decision report contains detailed action suggestions and relevant background information, providing comprehensive support for users and enhancing the practicality of the system and the user experience.

[0030] Select the Gradient Boosting Decision Tree (GBDT) as the machine learning algorithm to construct a load status prediction model, and perform feature extraction and transformation on the preprocessed data set. A resource allocation module, which is used to define a resource demand function according to the load status prediction results to calculate the amount of cloud computing resources required at each time point, and adjust the allocation amount of cloud computing resources according to the resource demand function. A real-time analysis module, which is used to configure a data processing channel using Apache Kafka Streams and load a pre-trained AI model in the stream processing channel. A decision generation module, which is used to define decision rules and thresholds based on the original data set and the analysis result set, and generate a decision report for a specific application scenario according to the decision rules applied.

[0031] This embodiment also provides a computer device applicable to the case of a cloud computing big data real-time processing method integrating AI technology, including: a memory and a processor; the memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions to implement the cloud computing big data real-time processing method integrating AI technology as proposed in the above embodiment.

[0032] This computer device can be a terminal. The computer device includes a processor, a memory, a communication interface, a display screen, and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner. The wireless manner can be achieved through Wi-Fi, a carrier network, NFC (Near Field Communication), or other technologies. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen. The input device of the computer device can be a touch layer covering the display screen, or a button, a trackball, or a touchpad set on the shell of the computer device, or an external keyboard, touchpad, or mouse, etc.

[0033] This embodiment also provides a storage medium, on which a computer program is stored. When the program is executed by a processor, it implements the cloud computing big data real-time processing method for integrating AI technology as proposed in the above embodiment; the storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (Static Random Access Memory, abbreviated as SRAM), electrically erasable programmable read-only memory (Electrically Erasable Programmable Read-Only Memory, abbreviated as EEPROM), erasable programmable read-only memory (Erasable Programmable Read Only Memory, abbreviated as EPROM), programmable read-only memory (Programmable Red-Only Memory, abbreviated as PROM), read-only memory (Read-Only Memory, abbreviated as ROM), magnetic memory, flash memory, magnetic disk or optical disc.

[0034] In summary, the present invention connects with multiple heterogeneous data sources by configuring Kafka Connect connectors, realizing the efficient integration of multi-source data. By selecting the Gradient Boosting Decision Tree GBDT as the load status prediction model and performing feature extraction and transformation on the preprocessed data set, the accurate prediction of future load status is achieved. The accurate load status prediction helps the platform identify peaks or troughs in resource requirements in advance, thereby optimizing the resource allocation strategy, avoiding resource waste or insufficiency, improving the response speed and service quality of the method. By defining a resource demand function based on the load status prediction result and adjusting the allocation amount of cloud computing resources according to the resource demand function, the dynamic optimization of resources is realized. The dynamic adjustment mechanism enables the platform to maximize resource utilization while meeting real-time requirements, reduce unnecessary expenses, and ensure stable service under high load conditions.

[0035] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit them. Although the present invention has been described in detail with reference to the preferred embodiments, those of ordinary skill in the art should understand that the technical solutions of the present invention can be modified or equivalently replaced without departing from the spirit and scope of the technical solutions of the present invention, and they should all be covered by the scope of the claims of the present invention.

Claims

1. A real-time processing method for cloud computing big data integrating AI technology, characterized in that: including: collecting raw data from multiple heterogeneous data sources using a data access interface to obtain a raw data set; preprocessing the raw data set using a distributed computing framework and obtaining a preprocessed data set; constructing a load status prediction model based on a machine learning algorithm and the preprocessed data set, inputting the preprocessed data set, and outputting a load status prediction result; adjusting the allocation amount of cloud computing resources in the raw data set based on the load status prediction result to obtain a resource allocation strategy; building a data processing channel using stream processing technology based on the resource allocation strategy, and combining with an AI algorithm to execute an identification task to obtain an analysis result; generating a decision report for a specific application scenario based on the analysis result.

2. The real-time processing method of cloud computing big data integrating AI technology according to claim 1, characterized in that: The step of collecting raw data from multiple heterogeneous data sources using a data access interface to obtain a raw data set is specifically as follows: configuring a Kafka Connect connector to connect to each heterogeneous data source; the heterogeneous data sources include relational databases, NoSQL databases, and log files; converting the relational databases, NoSQL databases, and log files in the heterogeneous data sources into a message format in a Kafka topic and obtaining a raw data set.

3. The real-time processing method of cloud computing big data integrating AI technology according to claim 2, characterized in that: The step of preprocessing the raw data set using a distributed computing framework and obtaining a preprocessed data set is specifically as follows: using Spark SQL to clean the noise data in the raw data set and removing incomplete or incorrect data records; using the DataFrame API of Spark to convert the data into a unified format; for records with missing values, filling them using a statistical model-based method and obtaining a preprocessed data set.

4. The real-time processing method of cloud computing big data integrating AI technology according to claim 3, characterized in that: The step of constructing a load status prediction model based on a machine learning algorithm and the preprocessed data set, inputting the preprocessed data set, and outputting a load status prediction result is specifically as follows: performing feature extraction and transformation on the preprocessed data set; selecting the Gradient Boosting Decision Tree (GBDT) as the load status prediction model; using the load status prediction model to predict the load status of new data points and calculating the load status prediction result.

5. The real-time processing method of cloud computing big data integrating AI technology according to claim 4, characterized in that: The step of adjusting the allocation amount of cloud computing resources in the raw data set based on the load status prediction result to obtain a resource allocation strategy is specifically as follows: Based on the prediction result of the load status , define a resource demand function to calculate the amount of cloud computing resources required at each time point; According to the resource demand function , adjust the allocation amount of cloud computing resources; Let be the total amount of cloud computing resources at the current moment , and be the amount of resources actually allocated to the platform; Based on allocate resources to the platform and obtain a resource allocation strategy.

6. The real-time processing method of cloud computing big data integrating AI technology according to claim 5, characterized in that: The step of building a data processing channel using stream processing technology based on the resource allocation strategy and combining with an AI algorithm to execute an identification task to obtain an analysis result is specifically as follows: configuring a data processing channel using Apache Kafka Streams; Let represent a stream processing channel, and represent an input data stream; Construct an AI pre-trained model based on AI algorithms , and in the stream processing channel, load the pre-trained AI model to perform complex pattern recognition tasks on each data record; Summarize the analysis results to obtain an analysis result set .

7. The real-time processing method of cloud computing big data integrating AI technology according to claim 6, characterized in that: The step of generating a decision report for a specific application scenario based on the analysis result is specifically as follows: Based on the original dataset and the analysis result set , define decision rules and thresholds; generating a decision report according to the decision rules of the application.

8. A cloud computing big data real-time processing system integrating AI technology, based on the cloud computing big data real-time processing method integrating AI technology according to any one of claims 1 to 7, characterized in that: including: a data collection module, a preprocessing module, a load prediction module, a resource allocation module, an analysis module, and a decision generation module; the data collection module is used to configure a Kafka Connect connector to connect to multiple heterogeneous data sources and convert the data in the data sources into a message format in a Kafka topic to obtain a raw data set; The preprocessing module is used to clean the original data set using Spark SQL, remove noise and incomplete records, and convert the data into a unified format using the DataFrame API of Spark; The load prediction module is used to select the Gradient Boosting Decision Tree (GBDT) as the machine learning algorithm to build a load status prediction model, and perform feature extraction and transformation on the preprocessed data set; The resource allocation module is used to define a resource demand function according to the load status prediction result to calculate the amount of cloud computing resources required at each time point, and adjust the allocation amount of cloud computing resources according to the resource demand function; The real-time analysis module is used to configure a data processing channel using Apache Kafka Streams and load a pre-trained AI model in the stream processing channel; The decision generation module is used to define decision rules and thresholds based on the original data set and the analysis result set, and generate a decision report for a specific application scenario according to the applied decision rules.

9. A computer device, comprising a memory and a processor, the memory storing a computer program, characterized in that: When the processor executes the computer program, it implements the steps of the cloud computing big data real-time processing method integrating AI technology according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by the processor, it implements the steps of the cloud computing big data real-time processing method integrating AI technology according to any one of claims 1 to 7.

Images