A method and system for real-time processing and analysis of AI big data

Abstract

The present application provides an AI big data real-time processing and analysis method and system, wherein real-time data streams from different data sources are obtained, and the real-time data streams are mapped to a preset data structure framework to obtain an initial data stream, the initial data stream is fused and processed by a real-time data fusion engine to obtain a target real-time data stream, and then data mining and machine learning algorithms are used to perform pattern recognition and trend analysis on the target real-time data stream to generate a target insight report, and the target insight report is optimized and processed using an intelligent decision-making unit, strategy information is generated and executed to obtain the best analysis results. The technical solution provided by the present application not only solves the problem of integration and fusion of multi-source heterogeneous data, but also improves the real-time and accuracy of data analysis, realizes efficient conversion from data to decision-making, and significantly improves the intelligence level and response speed of business processes.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of big data analysis technology, and in particular to a method and system for real-time processing and analysis of AI big data. Background Art

[0002] With the diversification of data sources and the surge in data volume, traditional data processing methods have become unable to meet the requirements of real-time and accuracy. Real-time data streams from different data sources usually have different formats and structures, which brings huge challenges to data integration and analysis. In addition, the temporal and spatial correlations in data streams often require complex processing to be effectively utilized. Existing data fusion technology has shortcomings in dynamically adjusting the combination of data streams, resulting in poor data fusion effects and an inability to provide a high-quality data foundation for subsequent analysis. Summary of the Invention

[0003] The embodiments of the present application provide a method and system for real-time processing and analysis of AI big data, which is used to solve the problem that traditional data analysis methods in the prior art have difficulty in quickly identifying patterns and trends in data when faced with large-scale, high-dimensional real-time data, especially when processing multi-source heterogeneous data, and have difficulty in efficiently processing and analyzing and generating valuable analysis reports.

[0004] In a first aspect, an embodiment of the present invention provides a method for real-time processing and analysis of AI big data, comprising:

[0005] Acquire real-time data streams from different data sources, and map the real-time data streams into a preset data structure framework to obtain an initial data stream;

[0006] The real-time data fusion engine is used to fuse and process the initial data streams to obtain a target real-time data stream, wherein the real-time data fusion engine is used to dynamically adjust the combination mode of the initial data streams according to the time correlation and spatial correlation in the initial data streams;

[0007] Utilize data mining and machine learning algorithms to perform pattern recognition and trend analysis on the target real-time data stream to generate a target insight report;

[0008] The target insight report is optimized and processed using an intelligent decision-making unit to generate strategy information, and the strategy information is executed to obtain the best analysis result.

[0009] Optionally, the initial data streams are fused and processed by a real-time data fusion engine to obtain a target real-time data stream, wherein the real-time data fusion engine is configured to dynamically adjust the combination mode of the initial data streams according to the temporal correlation and spatial correlation in the initial data streams, including:

[0010] Processing the initial data stream using geographic information system technology to obtain a data stream with a unified spatial reference frame;

[0011] Using natural language processing technology and machine learning models, the data stream of the unified spatial reference frame is identified and parsed to obtain a data stream with time and space attributes;

[0012] Utilize complex event processing technology and pattern matching to dynamically detect and intelligently combine data streams with time and space attributes to obtain optimized data streams;

[0013] The optimized data stream is subjected to quality assurance processing using data cleaning and anomaly detection mechanisms, and is processed using standardized protocols and data formats to obtain the target real-time data stream.

[0014] Optionally, complex event processing technology and pattern matching are used to dynamically detect and intelligently combine data streams with time and space attributes to obtain optimized data streams, including:

[0015] Use complex event processing technology and advanced time series analysis algorithms to perform real-time monitoring, correlation analysis, and pattern recognition on events in the initial real-time data stream, identifying combined data streams with similar timestamps and geographic locations;

[0016] Use pattern matching algorithms and rule engines to perform multi-dimensional pattern recognition and rule matching on combined data streams, determine the association and combination methods between combined data streams, and dynamically adjust them through a custom rule library to obtain the adjusted combined data stream;

[0017] Dynamically grouping, aggregating, and filtering the adjusted combined data stream in combination with time window management and spatial proximity constraints, while removing redundant and noisy data to obtain an optimized combined data stream;

[0018] Dynamically adjust and optimize the optimized combined data stream through an adaptive adjustment mechanism and a machine learning model to improve the combination strategy and obtain the best combined data stream;

[0019] The best combined data stream is evaluated and optimized using a real-time feedback and performance monitoring mechanism to generate an optimized data stream.

[0020] Optionally, the optimized combined data stream is dynamically adjusted and optimized by an adaptive adjustment mechanism and a machine learning model to improve the combination strategy to obtain the best combined data stream, including:

[0021] Using an adaptive adjustment mechanism to monitor and analyze the optimal combination data stream in real time to obtain change trends and abnormal conditions;

[0022] Use machine learning models to predict and optimize the optimal combination data stream to obtain the initial combination strategy;

[0023] The initial combination strategy is simulated and evaluated using a reinforcement learning algorithm, and the initial combination strategy is iteratively optimized and improved to obtain a target combination strategy that adapts to the dynamic changes of the optimal combination data flow;

[0024] Using online learning technology to instantly learn and adjust the target combination strategy to obtain a dynamic combination strategy that quickly responds to data patterns;

[0025] The dynamic combination strategy is verified and adjusted using feedback and performance monitoring mechanisms to obtain the optimal combination strategy.

[0026] Optionally, a reinforcement learning algorithm is used to simulate and evaluate the initial combination strategy, and the initial combination strategy is iteratively optimized to obtain a target combination strategy that adapts to the dynamic changes of the data stream, including:

[0027] Using a reinforcement learning algorithm to perform multiple rounds of simulation on the initial combination strategy, identifying strategy performance and problems in different environments, and obtaining simulation results;

[0028] The simulation results are evaluated using a reward mechanism to generate a feedback signal, and the feedback signal is used to iteratively update the initial combination strategy, adjust and optimize the parameters of the initial combination strategy, and obtain an optimized combination strategy;

[0029] Through the simulation and evaluation process of the number of predictions, the optimized combination strategy is adapted to the best combination data flow to obtain the target combination strategy;

[0030] Optionally, data mining and machine learning algorithms are used to perform pattern recognition and trend analysis on the target real-time data stream to generate a target insight report, including:

[0031] Use data preprocessing technology to clean, denoise, and standardize the target real-time data stream to obtain analytical data;

[0032] Extracting key features based on the analyzed data, and selecting a feature subset from the key features to obtain a data representation;

[0033] Use clustering and classification algorithms to perform pattern recognition on data representation, identify implicit patterns and categories in data representation, and obtain pattern recognition results;

[0034] Performing trend analysis on the pattern recognition results to identify long-term trends and short-term fluctuations in the data representation to obtain trend analysis results;

[0035] Leverage natural language processing technology and report generation templates to integrate and interpret pattern recognition and trend analysis results to generate targeted insight reports.

[0036] Optionally, optimizing and processing the target insight report using an intelligent decision-making unit to generate strategy information and execute the strategy information to obtain an optimal analysis result includes:

[0037] Use natural language processing technology and semantic analysis to deeply understand target insight reports and extract key information to obtain structured data representation;

[0038] Performing pattern recognition, trend analysis, and risk assessment on the structured data representation to identify business impact factors and opportunities;

[0039] Perform logical reasoning, rule matching, and simulation optimization on identified business influencing factors and opportunities to generate preliminary decision recommendations;

[0040] Adjusting the preliminary decision suggestions to generate optimal strategy information;

[0041] Utilize automated execution systems and API interfaces to output optimal strategy information to the execution system in real time, and generate execution results based on real-time and performance indicators;

[0042] The execution results are interpreted and visualized in multiple dimensions to generate the best analysis results.

[0043] In a second aspect, embodiments of the present application provide a method and system for real-time processing and analysis of AI big data, including:

[0044] An acquisition module acquires real-time data streams from different data sources and maps the real-time data streams into a preset data structure framework to obtain an initial data stream;

[0045] a processing module for fusing the initial data streams through a real-time data fusion engine to obtain a target real-time data stream, wherein the real-time data fusion engine is used to dynamically adjust the combination mode of the initial data streams according to the temporal correlation and spatial correlation in the initial data streams;

[0046] A generation module, utilizing data mining and machine learning algorithms to perform pattern recognition and trend analysis on the target real-time data stream to generate a target insight report;

[0047] The execution module utilizes the intelligent decision-making unit to optimize and process the target insight report, generate strategy information, and execute the strategy information to obtain the best analysis result.

[0048] In a third aspect, an embodiment of the present invention further provides a computing device, characterized in that it includes a processing component and a storage component; the storage component stores one or more computer instructions; the one or more computer instructions are used to be called and executed by the processing component to implement the AI ​​big data real-time processing and analysis method described in the first aspect.

[0049] In a fourth aspect, an embodiment of the present invention further provides a computer storage medium, characterized in that a computer program is stored therein, and when the computer program is executed by a computer, the method for real-time processing and analysis of AI big data described in the first aspect is implemented.

[0050] In an embodiment of the present application, real-time data streams from different data sources are obtained, and the real-time data streams are mapped to a preset data structure framework to obtain an initial data stream. The initial data stream is fused and processed by a real-time data fusion engine to obtain a target real-time data stream. Data mining and machine learning algorithms are then used to perform pattern recognition and trend analysis on the target real-time data stream to generate a target insight report. The target insight report is optimized and processed using an intelligent decision-making unit, and strategy information is generated and executed to obtain the best analysis results. The technical solution provided by the present application not only solves the problem of integration and fusion of multi-source heterogeneous data, but also improves the real-time and accuracy of data analysis, realizes efficient conversion from data to decision-making, and significantly improves the intelligence level and response speed of business processes.

[0051] These and other aspects of the present application will become more readily apparent from the description of the following embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, a brief introduction will be given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0053] Figure 1 A flowchart of a method for real-time processing and analysis of AI big data provided in an embodiment of the present application;

[0054] Figure 2 A schematic diagram of the structure of an AI big data real-time processing and analysis system provided in an embodiment of the present application;

[0055] Figure 3 A schematic diagram of the structure of a computing device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0056] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application.

[0057] In some of the processes described in the specification and claims of this application and the above-mentioned figures, multiple operations that appear in a specific order are included, but it should be clearly understood that these operations may not be executed in the order in which they appear in this document or may be executed in parallel. The serial numbers of the operations, such as 101, 102, etc., are only used to distinguish between different operations, and the serial numbers themselves do not represent any order of execution. In addition, these processes may include more or fewer operations, and these operations may be executed in sequence or in parallel. It should be noted that the descriptions of "first", "second", etc. in this document are used to distinguish different messages, devices, modules, etc., and do not represent a sequential order, nor do they limit "first" and "second" to being different types.

[0058] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without making creative efforts are within the scope of protection of this application.

[0059] In modern enterprises and organizations, the processing and analysis of real-time data is becoming increasingly important. With the diversification of data sources and the surge in data volume, traditional data processing methods have been unable to meet the requirements of real-time and accuracy. Based on this, the present invention provides a text data statistical analysis method based on natural language processing, such as Figure 1 As shown, the method includes the following steps:

[0060] Step S1, obtaining real-time data streams from different data sources, and mapping the real-time data streams into a preset data structure framework to obtain an initial data stream;

[0061] In this step, advanced data acquisition tools are used to acquire real-time data streams from multiple data sources, such as sensors, log files, and social media. These data streams are then mapped into a unified data structure framework. Data cleaning techniques are used to remove noise, redundant information, and incomplete data to ensure data consistency and comparability, providing a standardized foundation for subsequent data fusion and analysis.

[0062] Step S2, fusing and processing the initial data streams through a real-time data fusion engine to obtain a target real-time data stream, wherein the real-time data fusion engine is used to dynamically adjust the combination mode of the initial data streams according to the temporal correlation and spatial correlation in the initial data streams;

[0063] In this step, the real-time data fusion engine is used to fuse the initial data streams. By combining advanced time series analysis algorithms and geographic information system technology, the timestamps and spatial location information in the data streams are identified and processed. Complex event processing technology and pattern matching are used to dynamically adjust the combination of data streams, eliminate redundancy and inconsistency, and generate high-quality target real-time data streams, ensuring the temporal and spatial consistency of the data.

[0064] Step S3, using data mining and machine learning algorithms to perform pattern recognition and trend analysis on the target real-time data stream to generate a target insight report;

[0065] In this step, advanced data mining and machine learning algorithms are used to conduct in-depth analysis of the target real-time data stream. Natural language processing, clustering, classification, and anomaly detection technologies are combined to identify patterns, trends, and anomalies in the data. Through time series analysis and predictive models, reports containing valuable insights are generated. Advanced visualization tools are used to present the analysis results in an intuitive way, enhancing the readability and practicality of the report.

[0066] Step S4: Optimizing and processing the target insight report using an intelligent decision-making unit to generate strategy information, and executing the strategy information to obtain an optimal analysis result;

[0067] In this step, the intelligent decision-making unit optimizes the insight report. It first uses natural language processing technology and semantic analysis to extract key information. It then combines machine learning, rule engines, expert systems, and reinforcement learning algorithms to generate preliminary decision recommendations. Multi-objective optimization techniques are used to weigh and adjust the recommendations, generating validated optimal strategy information. This strategy information is then delivered to the corresponding execution system via an automated execution system and API interface. Execution status and performance indicators are collected through real-time monitoring and feedback mechanisms. Finally, interpretive AI technology and advanced visualization tools are used to generate easy-to-understand and highly actionable analysis reports.

[0068] To sum up, it not only solves the problem of data integration and fusion, but also improves the real-time and accuracy of data analysis. In addition, by generating and executing policy information through intelligent decision-making units, a complete closed-loop mechanism from data collection to decision execution is realized, which significantly improves the intelligence level and response speed of business processes.

[0069] Furthermore, the present invention provides a specific embodiment, according to step S2, the initial data streams are fused and processed by a real-time data fusion engine to obtain a target real-time data stream, and the real-time data fusion engine is used to dynamically adjust the combination mode of the initial data streams according to the temporal correlation and spatial correlation in the initial data streams. The method includes the following steps:

[0070] Step S21, processing the initial data stream using geographic information system technology to obtain a data stream with a unified spatial reference frame;

[0071] In this step, the spatial location information in the initial data stream is extracted and converted through geographic information system technology to ensure that all data are based on a unified spatial reference frame, solving the problem of inconsistent spatial location information in different data sources and providing a basis for subsequent temporal and spatial correlation analysis. Through a unified spatial reference frame, geographic location-related data analysis and fusion can be performed more accurately.

[0072] Step S22, using natural language processing technology and machine learning models to identify and parse the data stream of the unified spatial reference frame to obtain a data stream with time and space attributes;

[0073] In this step, natural language processing technology and machine learning models are combined to further identify and parse the data stream of the unified spatial reference frame. Natural language processing technology is used to extract time and space-related information from text data, while machine learning models are used to identify patterns and trends in the data. Through these technologies, implicit time and space information is made explicit, generating data streams with clear timestamps and geographic locations, thereby improving the analyzability of the data.

[0074] Step S23, using complex event processing technology and pattern matching to dynamically detect and intelligently combine data streams with time and space attributes to obtain optimized data streams;

[0075] In this step, complex event processing technology and pattern matching algorithms are used to dynamically detect and intelligently combine data streams with temporal and spatial attributes. Complex event processing technology can monitor events in data streams in real time and perform correlation analysis based on predefined rules and patterns. Pattern matching algorithms are used to identify specific patterns and sequences in data streams, dynamically adjust the combination of data streams, eliminate redundancy and inconsistency, generate optimized data streams, and ensure data coherence and consistency in time and space.

[0076] Step S24 , performing quality assurance processing on the optimized data stream using data cleaning and anomaly detection mechanisms, and processing using standardized protocols and data formats to obtain a target real-time data stream.

[0077] In this step, the optimized data stream is quality-assured through data cleaning and anomaly detection mechanisms. Data cleaning techniques are used to remove noise, redundant information, and incomplete data, while anomaly detection mechanisms are used to identify and process outliers in the data. These data cleaning and anomaly detection mechanisms ensure data quality and accuracy. Finally, the data stream is processed using standardized protocols and data formats to generate a standard target real-time data stream, providing a high-quality data foundation for subsequent analysis and decision-making.

[0078] To sum up, it not only ensures the consistency and comparability of data, but also enhances the analyzability of data, and ultimately generates high-quality target real-time data streams, improving the real-time, accuracy and consistency of data processing.

[0079] Furthermore, the present invention also provides a specific embodiment. According to step S23, complex event processing technology and pattern matching are used to dynamically detect and intelligently combine data streams with time and space attributes to obtain optimized data streams. The method includes the following steps:

[0080] Step S31: Using complex event processing technology and advanced time series analysis algorithms, perform real-time monitoring, correlation analysis, and pattern recognition on events in the initial real-time data stream to identify combined data streams with similar timestamps and geographic locations.

[0081] In this step, complex event processing technology is used to identify the complex relationships between multiple simple events, and association analysis is performed based on predefined rules and patterns. At the same time, advanced time series analysis algorithms are used to identify and predict the time correlation in the data. By processing the timestamp and geographic location information in the initial real-time data stream, combined data streams with similar timestamps and geographic locations are identified.

[0082] Step S32, dynamically grouping, aggregating, and filtering the adjusted combined data stream in combination with time window management and spatial proximity constraints, while removing redundant and noisy data to obtain an optimized combined data stream;

[0083] In this step, the data stream is divided into multiple time segments through time window management, and the data in each segment is analyzed as a window;

[0084] Using spatial proximity constraints, data is filtered and processed based on geographic proximity, identifying geographically close data points. Dynamic grouping is then used to group data with similar timestamps and geographic locations together. Aggregation is then performed to generate higher-level statistical information. Filtering mechanisms are used to remove redundant and noisy data, resulting in an optimized combined data stream to ensure data quality and relevance.

[0085] Step S33, dynamically adjusting and optimizing the optimized combined data stream through an adaptive adjustment mechanism and a machine learning model to improve the combination strategy and obtain the best combined data stream;

[0086] In this step, the adaptive adjustment mechanism monitors the changes in the data stream in real time and automatically adjusts the parameters and rules according to these changes;

[0087] Machine learning models learn patterns and regularities from historical data and use these patterns to predict and optimize new data. Through dynamic adjustment and optimization, they continuously improve the combination strategy to ensure that it always matches the state of the current data stream, generating the best combination data stream in terms of both quality and relevance.

[0088] Step S34, using real-time feedback and performance monitoring mechanisms to evaluate and optimize the optimal combined data stream to generate an optimized data stream;

[0089] During this step, the real-time feedback mechanism instantly collects and analyzes system operation results and feeds these results back to the system for adjustment and optimization. The performance monitoring mechanism continuously monitors system performance indicators such as response time, throughput, and resource utilization to assess system health. These mechanisms enable the rapid detection of potential problems and anomalies, enabling appropriate adjustments and optimizations. Ultimately, this evaluation and optimization process generates the target real-time data stream, achieving optimal quality and performance.

[0090] In summary, the embodiments of the present invention not only improve the real-time and accuracy of big data processing while ensuring timely capture of key events and patterns, but also can flexibly respond to data changes in different scenarios, improve the flexibility and adaptability of data processing, and continuously learn and adapt to data changes through adaptive adjustment and optimization, providing a reliable data foundation for practical applications and decision support.

[0091] Furthermore, the present invention also provides a specific embodiment. According to step S33, the optimized combined data stream is dynamically adjusted and optimized by an adaptive adjustment mechanism and a machine learning model to improve the combination strategy, thereby obtaining the best combined data stream. The method includes the following steps:

[0092] Step S41, using an adaptive adjustment mechanism to monitor and analyze the optimal combined data stream in real time to obtain change trends and abnormal conditions;

[0093] In this step, the adaptive adjustment mechanism continuously collects and analyzes data streams, automatically adjusting system parameters and behaviors to respond to changes in data and the environment. By monitoring the optimal combination of data streams in real time, it can immediately detect changing trends in the data, such as upward, downward, or stable trends, and identify anomalies that do not meet expectations.

[0094] Furthermore, since traditional data analysis methods often have difficulty in quickly identifying patterns and trends in data when faced with large-scale, high-dimensional, real-time data, an adaptive adjustment mechanism is also provided in an embodiment of the present invention. The expression of the adaptive adjustment mechanism is:

[0095] ;

[0096] in, It's time The predicted value of Based on historical data and model parameters prediction function (such as LSTM or Transformer models), Based on the data change rate and parameters The trend analysis function of Based on anomaly detection results and parameters The anomaly correction function, Based on multimodal data and parameters The multimodal fusion function of Based on uncertainty estimates and parameters The uncertainty correction function is It is an adaptive adjustment coefficient that changes over time and can be dynamically updated through reinforcement learning or other adaptive control methods;

[0097] In this step, Use deep learning models (such as Transformer or LSTM) to model historical data to generate forecasts for future time points; By calculating the data change rate and using trend analysis models to extract the change trend; Use anomaly detection algorithms to identify abnormal data and correct predicted values; Combine multiple types of data (such as time series data, text data, image data, etc.) through multimodal fusion models to extract comprehensive features; Use Bayesian methods or other uncertainty estimation techniques to estimate the uncertainty of the prediction results, and modify the prediction values ​​according to the uncertainty estimation results, and adaptively adjust the coefficients It is used to dynamically adjust the weights of different components to ensure adaptive adjustments based on changes in the current data stream. These coefficients can be dynamically updated through reinforcement learning (such as Q-learning or DQN). Through this multi-dimensional comprehensive processing method, it can more effectively identify changing trends and anomalies in real-time monitoring and analysis, and make adaptive adjustments, thereby improving overall data processing capabilities and decision support levels.

[0098] Step S42: using a machine learning model to predict and optimize the optimal combination data stream to obtain an initial combination strategy;

[0099] In this step, the machine learning model learns from historical and current data to identify patterns and trends in the data and predict future data behavior. It also analyzes and adjusts the data stream, optimizing it to improve its quality and consistency. Finally, based on these predictions and optimization results, it generates a preliminary combination strategy, known as the initial combination strategy.

[0100] Step S43, using a reinforcement learning algorithm to simulate and evaluate the initial combination strategy, iteratively optimize and improve the initial combination strategy, and obtain a target combination strategy that adapts to the dynamic changes of the optimal combination data flow;

[0101] In this step, the reinforcement learning algorithm gradually optimizes the behavioral strategy through the interaction between the intelligent agent and the environment. By simulating and evaluating the initial combination strategy, the effectiveness and robustness of the strategy can be tested in a virtual environment. Based on the simulation results, multiple iterative optimizations are performed to continuously adjust and improve the strategy to make it more adaptable to the changes and needs of the actual data flow. Finally, after multiple rounds of iterative optimization, the target combination strategy that can adapt to the dynamic changes of the optimal combination data flow is generated;

[0102] Step S44, using online learning technology to instantly learn and adjust the target combination strategy to obtain a dynamic combination strategy that quickly responds to the data pattern;

[0103] In this step, online learning technology can learn and update model parameters in real time from continuous data streams, ensuring that it can quickly adapt to changes in the data stream. By learning the target combination strategy in real time, it can quickly capture new information and changes in the data stream and adjust the existing strategy based on this new information, including updating model parameters, adjusting rules, and improving strategies. This real-time learning and adjustment process enables the system to quickly respond to patterns and trends in the data stream, thereby generating a dynamic combination strategy that can be updated and optimized in real time.

[0104] Step S45: Verify and adjust the dynamic combination strategy using feedback and performance monitoring mechanisms to obtain the optimal combination strategy;

[0105] In this step, the feedback mechanism collects and analyzes the operating results, feeds back the results, and makes adjustments and optimizations. The performance monitoring mechanism continuously monitors the system's performance indicators, such as response time, throughput, and resource utilization, to evaluate the system's operating status. By verifying and evaluating the actual application effects of the dynamic combination strategy, necessary adjustments and optimizations can be made based on the feedback results. This verification and adjustment process ensures that the strategy operates effectively and efficiently in the actual environment, and potential problems are discovered and resolved in a timely manner. Finally, after multiple rounds of verification and adjustment, the optimal combination strategy that can maintain efficiency and accuracy under various conditions is generated.

[0106] In summary, the embodiments of the present invention can not only respond quickly to changes in data streams, but also generate more accurate and reliable prediction results. Moreover, through multiple rounds of simulation and evaluation processing, as well as real-time learning and adjustment, it can better adapt to various complex environments. Finally, through feedback and performance monitoring mechanisms, the strategy is continuously optimized to ensure its efficiency and reliability in practical applications.

[0107] Because existing data mining and machine learning algorithms still have limitations in real-time processing and complex pattern recognition, the present invention further provides an embodiment. According to step S43, a reinforcement learning algorithm is used to simulate and evaluate the initial combination strategy, and the initial combination strategy is iteratively optimized to obtain a target combination strategy that adapts to the dynamic changes of the data stream. The method includes the following steps:

[0108] Step S51, using a reinforcement learning algorithm to perform multiple rounds of simulation on the initial combination strategy, identifying strategy performance and problems in different environments, and obtaining simulation results;

[0109] In this step, by repeatedly testing and evaluating the initial combined strategy in a virtual environment, we gain a comprehensive understanding of the strategy's actual performance under various conditions, including its strengths and weaknesses. Reinforcement learning algorithms gradually optimize the behavior strategy through the interaction between the agent and the environment, while multiple rounds of simulation ensure that the strategy is thoroughly tested in different environments.

[0110] Step S52: using a reward mechanism to evaluate the simulation results and generate a feedback signal. The feedback signal is used to iteratively update the initial combination strategy, adjust and optimize the parameters of the initial combination strategy, and obtain an optimized combination strategy.

[0111] In this step, the reward mechanism guides the agent's behavior by giving it positive or negative feedback. The evaluation process quantifies and evaluates the performance of the strategy based on the simulation results. The generated feedback signal can tell the agent whether its current behavior is good or bad and guide it to take better actions in the future. Through multiple iterative update processes, the strategy parameters are gradually adjusted and optimized to generate an optimized combination strategy that can show higher performance and robustness under various conditions.

[0112] Step S53, through the simulation and evaluation process of the predicted number of times, the optimized combination strategy is adapted to the best combination data flow to obtain the target combination strategy;

[0113] In this step, the performance of the optimized combination strategy is repeatedly predicted and evaluated in multiple simulations to ensure its adaptability and effectiveness in the optimal combination data stream. Through this process, the performance of the strategy in different situations can be comprehensively tested and verified, and the strategy parameters can be gradually adjusted and optimized to make it more adaptable to the changes and needs of the actual data stream. Finally, after multiple rounds of prediction and evaluation, a target combination strategy is generated that can operate efficiently in the optimal combination data stream and adapt to various changes.

[0114] Step S54, verifying the target combination strategy using the verification set to obtain the optimal combination strategy;

[0115] In this step, the validation set is a set of new data independent of the training and test data, which is used to evaluate the performance of the target combination strategy in the actual environment. Through the validation process, the performance of the strategy on the new data is comprehensively tested, and necessary adjustments and optimizations are made based on the validation results. After the validation process, the best combination strategy is generated that can run efficiently in actual applications and provide optimal performance.

[0116] Furthermore, the present invention also provides an embodiment, according to step S3, using a reinforcement learning algorithm to simulate and evaluate the initial combination strategy, iteratively optimize and improve the initial combination strategy, and obtain a target combination strategy that adapts to the dynamic changes of the data stream. The method includes the following steps:

[0117] Step S61, using data preprocessing technology to clean, denoise and standardize the target real-time data stream to obtain analysis data;

[0118] In this step, data preprocessing techniques are used to clean the target real-time data stream, remove noise and outliers, and perform standardization to ensure data consistency and comparability. The cleaning process involves deleting or correcting incomplete, erroneous, or duplicate data records. Denoising involves filtering out random fluctuations and interference signals in the data. Standardization converts the data into a unified format and scale to generate high-quality analytical data.

[0119] Step S62: extracting key features based on the analyzed data, and selecting a feature subset from the key features to obtain a data representation;

[0120] In this step, key features that represent the essence of the data are extracted from the analyzed data. These features are the most informative part of the data and can reflect the main attributes and changes of the data. Feature selection technology is used to further select the most representative feature subset from these key features to form a data representation. Feature selection can reduce data dimensionality, improve model training efficiency and generalization ability, while retaining the most important information.

[0121] Step S63, performing pattern recognition processing on the data representation using clustering and classification algorithms to identify implicit patterns and categories in the data representation and obtain pattern recognition results;

[0122] In this step, clustering and classification algorithms are used to perform pattern recognition processing on the data representation. Clustering algorithms (such as K-means, DBSCAN, etc.) divide the data into multiple clusters, and the data in each cluster has similar characteristics. Classification algorithms (such as decision trees, support vector machines, neural networks, etc.) are used to classify the data into predefined categories. These algorithms are used to identify implicit patterns and categories in the data representation, reveal the internal structure and relationships of the data, and generate pattern recognition results.

[0123] Step S64, performing trend analysis on the pattern recognition results to identify long-term trends and short-term fluctuations in the data representation to obtain trend analysis results;

[0124] In this step, the pattern recognition results are processed for trend analysis to identify long-term trends and short-term fluctuations in the data. Trend analysis typically involves time series analysis methods such as moving averages, exponential smoothing, and ARIMA models. These methods reveal patterns in data over time and identify long-term trends (such as growth, decline, or stability) and short-term fluctuations (such as seasonality and cyclical fluctuations). Ultimately, trend analysis results are generated, providing a comprehensive understanding of data changes.

[0125] Step S65 , using natural language processing technology and a report generation template, the pattern recognition results and trend analysis results are integrated and interpreted to generate a target insight report.

[0126] In this step, natural language processing (NLP) technology and predefined report generation templates are used to integrate and interpret pattern recognition and trend analysis results. Natural language processing transforms complex data analysis results into easily understandable natural language descriptions, while report generation templates provide a structured framework to ensure clarity and consistency in report content. These technologies generate detailed, targeted insight reports that not only include the results of the data analysis but also provide explanations and recommendations, helping decision makers better understand and apply these insights.

[0127] Furthermore, the present invention also provides an embodiment, according to step S4, using an intelligent decision-making unit to optimize and process the target insight report, generate strategy information, and execute the strategy information to obtain the best analysis result, the method comprising the following steps:

[0128] Step S71: Using natural language processing technology and semantic analysis to deeply understand the target insight report and extract key information to obtain a structured data representation;

[0129] In this step, natural language processing technology and semantic analysis methods are used to deeply understand the generated target insight report. Natural language processing technology can parse text content, identify and extract key information, such as business indicators, trend descriptions, and pattern recognition results. Semantic analysis is used to understand the context and meaning of this information and convert it into structured data representation for subsequent processing and analysis.

[0130] Step S72: performing pattern recognition, trend analysis, and risk assessment on the structured data representation to identify business impact factors and opportunities;

[0131] In this step, structured data representations are further subjected to pattern recognition, trend analysis, and risk assessment. Pattern recognition is used to discover repetitive patterns and regularities in the data, trend analysis is used to identify long-term trends and short-term fluctuations, and risk assessment helps identify potential risk factors. These analyses identify key factors affecting the business and potential opportunities, providing important evidence for decision-making.

[0132] Step S73: Perform logical reasoning, rule matching, and simulation optimization on the identified business impact factors and opportunities to generate preliminary decision recommendations;

[0133] In this step, we leverage logical reasoning, rule matching, and simulation optimization techniques to conduct in-depth analysis of identified business impact factors and opportunities. Logical reasoning derives causal relationships and potential impacts, while rule matching verifies and screens based on predefined business rules. Simulation optimization simulates the effects of strategies in different scenarios to identify optimal solutions. Ultimately, preliminary decision recommendations are generated to guide actual operations.

[0134] Step S74, adjusting the preliminary decision suggestion to generate optimal strategy information;

[0135] In this step, preliminary decision recommendations are further adjusted and optimized to ensure they meet actual business needs and environmental changes. The adjustment process may involve parameter fine-tuning, strategy combination optimization, and interactive feedback with business experts to generate the final optimal strategy information and ensure its effectiveness and feasibility in actual application;

[0136] Step S75: Utilize the automated execution system and API interface to output the optimal strategy information to the execution system in real time, and generate execution results based on the real-time status and performance indicators;

[0137] In this step, the optimal strategy information is transmitted to the execution system in real time through the automated execution system and API interface. The automated execution strategy is converted into specific operational instructions and executed in the actual environment. At the same time, by monitoring the actual situation and performance indicators (such as response time, resource utilization, task completion rate, etc.), data from the execution process is collected and detailed execution results are generated for further analysis and evaluation.

[0138] Step S76, interpreting and visually displaying the execution results in multiple dimensions to generate the best analysis results;

[0139] In this step, the execution results are explained and visualized in multiple dimensions. Complex execution results are presented to users in an intuitive manner through charts, dashboards, and other visualization tools. This not only helps users better understand the execution effects, but also provides a multi-dimensional perspective, making it easier for users to analyze and interpret data from different angles, ultimately generating the best analysis results and providing users with comprehensive data support and decision-making basis.

[0140] In summary, the embodiments of the present invention provide a comprehensive and efficient decision support process. This method not only improves the accuracy and efficiency of decision-making, but also enhances the robustness and adaptability of the system, provides a solid data foundation for the operation and management of the enterprise, and helps to improve overall business performance and competitiveness.

[0141] Figure 2 The present invention provides a structural diagram of an AI big data real-time processing and analysis system, as shown in FIG. Figure 2 As shown, the system includes:

[0142] An acquisition module 21 acquires real-time data streams from different data sources and maps the real-time data streams into a preset data structure framework to obtain an initial data stream;

[0143] A fusion module 22 is configured to fuse and process the initial data streams through a real-time data fusion engine to obtain a target real-time data stream. The real-time data fusion engine is configured to dynamically adjust the combination of the initial data streams based on the temporal and spatial correlations in the initial data streams.

[0144] A generation module 23 performs pattern recognition and trend analysis on the target real-time data stream using data mining and machine learning algorithms to generate a target insight report;

[0145] The execution module 24 uses an intelligent decision-making unit to optimize and process the target insight report, generate strategy information, and execute the strategy information to obtain the best analysis result.

[0146] Figure 2 The AI ​​big data real-time processing and analysis system can perform Figure 1 The implementation principle and technical effects of the AI ​​big data real-time processing and analysis method described in the illustrated embodiment are not repeated here. The specific manner in which each module and unit performs operations in the AI ​​big data real-time processing and analysis system in the above embodiment has been described in detail in the embodiment of the method and will not be elaborated on here.

[0147] In one possible design, Figure 2 An AI big data real-time processing and analysis system of the embodiment shown can be implemented as a computing device, such as Figure 3 As shown, the computing device may include a storage component 31 and a processing component 32;

[0148] The storage component 31 stores one or more computer instructions, wherein the one or more computer instructions are called and executed by the processing component 32 .

[0149] The processing component 32 is used to: obtain real-time data streams from different data sources, and map the real-time data streams into a preset data structure framework to obtain an initial data stream; fuse and process the initial data stream through a real-time data fusion engine to obtain a target real-time data stream, and the real-time data fusion engine is used to dynamically adjust the combination method of the initial data stream according to the time correlation and spatial correlation in the initial data stream; use data mining and machine learning algorithms to perform pattern recognition and trend analysis on the target real-time data stream to generate a target insight report; use an intelligent decision-making unit to optimize the processing of the target insight report, generate strategy information, and execute the strategy information to obtain the best analysis result.

[0150] The processing component 32 may include one or more processors to execute computer instructions to complete all or part of the steps in the above method. Of course, the processing component may also be implemented as one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the above method.

[0151] The storage component 31 is configured to store various types of data to support operations at the terminal. The storage component can be implemented by any type of volatile or non-volatile memory device, or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk, or optical disk.

[0152] Of course, a computing device may also include other components, such as input / output interfaces, display components, communication components, etc.

[0153] The input / output interface provides an interface between the processing component and the peripheral interface module, which can be an output device, an input device, etc.

[0154] The communication component is configured to facilitate, among other things, wired or wireless communications between the computing device and other devices.

[0155] Among them, the computing device can be a physical device or an elastic computing host provided by a cloud computing platform, etc. In this case, the computing device can refer to a cloud server, and the above-mentioned processing components, storage components, etc. can be basic server resources rented or purchased from the cloud computing platform.

[0156] The present application also provides a computer storage medium storing a computer program, wherein the computer program can achieve the above-mentioned Figure 1 An AI big data real-time processing and analysis method according to the embodiment shown.

[0157] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0158] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, i.e., they may be located in one location or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of the present embodiment. Persons of ordinary skill in the art will be able to understand and implement the present invention without inventive effort.

[0159] Through the above description of the embodiments, those skilled in the art will clearly understand that each embodiment can be implemented using software plus a necessary general-purpose hardware platform, or of course, hardware. Based on this understanding, the essence of the above technical solution, or the portion that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a magnetic disk, or an optical disk, and includes a number of instructions for causing a computer device (such as a personal computer, server, or network device) to execute the methods described in each embodiment or certain portions of the embodiments.

[0160] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for real-time processing and analysis of AI big data, characterized in that: include: Step S1, obtaining real-time data streams from different data sources, and mapping the real-time data streams into a preset data structure framework to obtain an initial data stream; Different data sources include log files, social media; Step S2: The initial data streams are fused and processed by a real-time data fusion engine to obtain a target real-time data stream. The real-time data fusion engine is used to dynamically adjust the combination of the initial data streams according to the temporal correlation and spatial correlation in the initial data streams. Step S3: Using data mining and machine learning algorithms to perform pattern recognition and trend analysis on the target real-time data stream to generate a target insight report; Step S4: Utilize the intelligent decision-making unit to optimize and process the target insight report, generate strategy information, and execute the strategy information to obtain the best analysis result; Step S2 includes: Step S21, using geographic information system technology to process the initial data stream to obtain a data stream with a unified spatial reference frame; Step S22, using natural language processing technology to extract time and space related information from the text data, using machine learning models to identify patterns and trends in the data, and obtaining a data stream with time and space attributes; Step S23, using complex event processing technology and pattern matching to dynamically detect and intelligently combine data streams with time and space attributes to obtain optimized data streams; Step S24, using data cleaning and anomaly detection mechanisms to perform quality assurance processing on the optimized data stream, and using standardized protocols and data formats for processing to obtain the target real-time data stream; Step S23 includes: Step S31: Complex event processing techniques are used to identify complex relationships between multiple simple events, and association analysis is performed based on predefined rules and patterns. Time series analysis algorithms are also used to identify and predict temporal correlations in the data. By processing the timestamps and geographic location information in the initial real-time data stream, combined data streams with similar timestamps and geographic locations are identified. Step S32 , dynamically grouping, aggregating, and filtering the adjusted combined data stream in combination with time window management and spatial proximity constraints, while removing redundant and noisy data to obtain an optimized combined data stream; Step S33, dynamically adjusting and optimizing the optimized combined data stream through the adaptive adjustment mechanism and the machine learning model to improve the combination strategy and obtain the best combined data stream; Step S34, using real-time feedback and performance monitoring mechanisms to evaluate and optimize the best combined data flow to generate an optimized data flow; Step S33 includes: Step S41, using the adaptive adjustment mechanism to monitor and analyze the optimal combination data stream in real time to obtain change trends and abnormal conditions; Step S42: using a machine learning model to predict and optimize the optimal combination data stream to obtain an initial combination strategy; Step S43, using a reinforcement learning algorithm to simulate and evaluate the initial combination strategy, iteratively optimize and improve the initial combination strategy, and obtain a target combination strategy that adapts to the dynamic changes of the optimal combination data flow; Step S44, using online learning technology to conduct real-time learning and adjustment processing on the target combination strategy to obtain a dynamic combination strategy that quickly responds to the data pattern; Step S45: Verify and adjust the dynamic combination strategy using feedback and performance monitoring mechanisms to obtain the optimal combination strategy; Step S43 includes: Step S51, using a reinforcement learning algorithm to perform multiple rounds of simulation on the initial combination strategy, identifying strategy performance and problems in different environments, and obtaining simulation results; Step S52: Using the reward mechanism to evaluate the simulation results and generate a feedback signal. The feedback signal is then used to iteratively update the initial combination strategy, adjust and optimize the parameters of the initial combination strategy, and obtain an optimized combination strategy. Step S53, through the simulation and evaluation process of the predicted number of times, the optimized combination strategy is adapted to the best combination data flow to obtain the target combination strategy; Step S54: Verify the target combination strategy using the verification set to obtain the optimal combination strategy.

2. The method according to claim 1, characterized in that Use data mining and machine learning algorithms to perform pattern recognition and trend analysis on target real-time data streams to generate target insight reports, including: Use data preprocessing technology to clean, denoise, and standardize the target real-time data stream to obtain analytical data; Extract key features based on the analyzed data and select a subset of features from the key features to obtain data representation; Use clustering and classification algorithms to perform pattern recognition on data representation, identify implicit patterns and categories in data representation, and obtain pattern recognition results; Perform trend analysis on the pattern recognition results to identify long-term trends and short-term fluctuations in the data representation and obtain trend analysis results; Leverage natural language processing technology and report generation templates to integrate and interpret pattern recognition and trend analysis results to generate targeted insight reports.

3. The method according to claim 1, characterized in that Utilize intelligent decision-making units to optimize and process target insight reports, generate and execute strategic information, and obtain optimal analysis results, including: Use natural language processing technology and semantic analysis to deeply understand target insight reports and extract key information to obtain structured data representation; Perform pattern recognition, trend analysis, and risk assessment on structured data representation to identify business impact factors and opportunities; Perform logical reasoning, rule matching, and simulation optimization on identified business influencing factors and opportunities to generate preliminary decision recommendations; Adjust preliminary decision recommendations to generate optimal strategy information; Utilize automated execution systems and API interfaces to output optimal strategy information to the execution system in real time, and generate execution results based on real-time and performance indicators; Interpret and visualize the execution results in multiple dimensions to generate the best analysis results.

4. An AI big data real-time processing and analysis system, applied to the AI ​​big data real-time processing and analysis method according to any one of claims 1 to 3, comprising: The acquisition module is used to obtain real-time data streams from different data sources and map the real-time data streams into a preset data structure framework to obtain the initial data stream; Different data sources include log files, social media; The fusion module fuses the initial data streams through the real-time data fusion engine to obtain the target real-time data stream. The real-time data fusion engine is used to dynamically adjust the combination of the initial data streams according to the temporal correlation and spatial correlation in the initial data streams. A generation module is used to perform pattern recognition and trend analysis on the target real-time data stream using data mining and machine learning algorithms to generate target insight reports; The execution module is used to optimize the processing of target insight reports using intelligent decision-making units, generate strategy information, and execute the strategy information to obtain the best analysis results.

5. A computing device, characterized in that It includes a processing component and a storage component; the storage component stores one or more computer instructions; the one or more computer instructions are used to be called and executed by the processing component to implement an AI big data real-time processing and analysis method as described in any one of claims 1 to 3.

6. A computer storage medium, characterized in that A computer program is stored, and when the computer program is executed by a computer, an AI big data real-time processing and analysis method as described in any one of claims 1 to 3 is implemented.

Images