Method and system for processing and analyzing AI big data in real time

Abstract

The invention provides an AI big data real-time processing and analysis method and system, and the method comprises the steps: obtaining real-time data streams from different data sources, mapping the real-time data streams into a preset data structure frame, obtaining an initial data stream, carrying out the fusion processing of the initial data stream through a real-time data fusion engine, obtaining a target real-time data stream, and carrying out the analysis of the target real-time data stream. Performing pattern recognition and trend analysis processing on the target real-time data flow by using a data mining and machine learning algorithm, generating a target insight report, performing optimization processing on the target insight report by using an intelligent decision-making unit, generating and executing strategy information, and obtaining an optimal analysis result; according to the technical scheme provided by the invention, the problem of integration and fusion of multi-source heterogeneous data is solved, the real-time performance and accuracy of data analysis are also improved, efficient conversion from data to decision is realized, and the intelligent level and response speed of a business process are remarkably improved.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of big data analysis technology, and in particular, to a method and system for real-time processing and analysis of AI big data. Background Art

[0002] With the diversification of data sources and the surge in data volume, traditional data processing methods have been unable to meet the requirements of real-time and accuracy. Real-time data streams from different data sources usually have different formats and structures, which brings huge challenges to data integration and analysis. In addition, the temporal and spatial correlations in data streams often require complex processing to be effectively utilized, and existing data fusion technology has deficiencies in dynamically adjusting the combination of data streams, resulting in poor data fusion effects and failure to provide a high-quality data foundation for subsequent analysis. Summary of the invention

[0003] The embodiments of the present application provide a method and system for real-time processing and analysis of AI big data, which is used to solve the problem that traditional data analysis methods in the prior art are difficult to quickly identify patterns and trends in data when faced with large-scale, high-dimensional real-time data, especially when processing multi-source heterogeneous data, and it is difficult to efficiently process and analyze and generate valuable analysis reports.

[0004] In a first aspect, an embodiment of the present invention provides a method for real-time processing and analysis of AI big data, including: Acquire real-time data streams from different data sources, and map the real-time data streams into a preset data structure framework to obtain an initial data stream; The initial data streams are fused and processed by a real-time data fusion engine to obtain a target real-time data stream, wherein the real-time data fusion engine is used to dynamically adjust the combination mode of the initial data streams according to the time correlation and space correlation in the initial data streams; Using data mining and machine learning algorithms to perform pattern recognition and trend analysis on the target real-time data stream to generate a target insight report; The target insight report is optimized and processed using an intelligent decision-making unit, strategy information is generated, and the strategy information is executed to obtain the best analysis result.

[0005] Optionally, the initial data streams are fused and processed by a real-time data fusion engine to obtain a target real-time data stream, and the real-time data fusion engine is used to dynamically adjust the combination mode of the initial data streams according to the time correlation and spatial correlation in the initial data streams, including: Processing the initial data stream using geographic information system technology to obtain a data stream with a unified spatial reference frame; Using natural language processing technology and machine learning models, the data stream of the unified spatial reference frame is identified and parsed to obtain a data stream with time and space attributes. Use complex event processing technology and pattern matching to dynamically detect and intelligently combine data streams with time and space attributes to obtain optimized data streams; The optimized data stream is subjected to quality assurance processing using data cleaning and anomaly detection mechanisms, and is processed using standardized protocols and data formats to obtain a target real-time data stream.

[0006] Optionally, complex event processing technology and pattern matching are used to dynamically detect and intelligently combine data streams with time and space attributes to obtain optimized data streams, including: Use complex event processing technology and advanced time series analysis algorithms to perform real-time monitoring, correlation analysis, and pattern recognition processing on events in the initial real-time data stream, and identify combined data streams with similar timestamps and geographic locations; Use pattern matching algorithms and rule engines to perform multi-dimensional pattern recognition and rule matching on combined data streams, determine the association relationship and combination method between combined data streams, and dynamically adjust them through a custom rule library to obtain the adjusted combined data stream; Dynamically grouping, aggregating and filtering the adjusted combined data stream in combination with time window management and spatial proximity constraints, while removing redundant and noisy data to obtain an optimized combined data stream; Dynamically adjust and optimize the optimized combined data stream through an adaptive adjustment mechanism and a machine learning model to improve the combination strategy and obtain the best combined data stream; The best combined data stream is evaluated and optimized using a real-time feedback and performance monitoring mechanism to generate an optimized data stream.

[0007] Optionally, the optimized combined data stream is dynamically adjusted and optimized by an adaptive adjustment mechanism and a machine learning model to improve the combination strategy, so as to obtain the best combined data stream, including: Using an adaptive adjustment mechanism to monitor and analyze the optimal combined data stream in real time to obtain change trends and abnormal conditions; Use machine learning models to predict and optimize the best combination data stream to obtain the initial combination strategy; The initial combination strategy is simulated and evaluated by using a reinforcement learning algorithm, and the initial combination strategy is iteratively optimized and improved to obtain a target combination strategy that adapts to the dynamic changes of the optimal combination data flow; Using online learning technology to instantly learn and adjust the target combination strategy, to obtain a dynamic combination strategy that quickly responds to data patterns; The dynamic combination strategy is verified and adjusted by using feedback and performance monitoring mechanisms to obtain the best combination strategy.

[0008] Optionally, the initial combination strategy is simulated and evaluated using a reinforcement learning algorithm, and the initial combination strategy is iteratively optimized and improved to obtain a target combination strategy that adapts to the dynamic changes of the data stream, including: Using a reinforcement learning algorithm to perform multiple rounds of simulation on the initial combination strategy, identifying strategy performance and problems in different environments, and obtaining simulation results; The simulation results are evaluated and processed by using a reward mechanism to generate a feedback signal, and the feedback signal is used to iteratively update the initial combination strategy, adjust and optimize the parameters of the initial combination strategy, and obtain an optimized combination strategy; Through the simulation and evaluation process of the forecast times, the optimized combination strategy is adapted to the best combination data flow to obtain the target combination strategy; Optionally, data mining and machine learning algorithms are used to perform pattern recognition and trend analysis on the target real-time data stream to generate a target insight report, including: Use data preprocessing technology to clean, denoise and standardize the target real-time data stream to obtain analytical data; Extracting key features based on the analysis data, and selecting a feature subset from the key features to obtain data representation; Use clustering and classification algorithms to perform pattern recognition on data representation, identify implicit patterns and categories in data representation, and obtain pattern recognition results; Performing trend analysis on the pattern recognition results to identify long-term trends and short-term fluctuations in the data representation and obtain trend analysis results; Use natural language processing technology and report generation templates to integrate and interpret pattern recognition and trend analysis results to generate targeted insight reports.

[0009] Optionally, the target insight report is optimized and processed by an intelligent decision-making unit to generate strategy information, and the strategy information is executed to obtain the best analysis result, including: Use natural language processing technology and semantic analysis to deeply understand and extract key information from target insight reports to obtain structured data representation; Performing pattern recognition, trend analysis, and risk assessment on the structured data representation to identify business impact factors and opportunities; Perform logical reasoning, rule matching, and simulation optimization on the identified business influencing factors and opportunities to generate preliminary decision recommendations; Adjusting the preliminary decision suggestions to generate optimal strategy information; Use the automated execution system and API interface to output the best strategy information to the execution system in real time, and generate execution results through real-time and performance indicators; The execution results are explained and visualized in multiple dimensions to generate the best analysis results.

[0010] In a second aspect, the embodiments of the present application provide a method and system for real-time processing and analysis of AI big data, including: An acquisition module acquires real-time data streams from different data sources and maps the real-time data streams into a preset data structure framework to obtain an initial data stream; A processing module, which fuses and processes the initial data streams through a real-time data fusion engine to obtain a target real-time data stream, wherein the real-time data fusion engine is used to dynamically adjust the combination mode of the initial data streams according to the time correlation and space correlation in the initial data streams; A generation module, using data mining and machine learning algorithms to perform pattern recognition and trend analysis on the target real-time data stream to generate a target insight report; The execution module uses the intelligent decision-making unit to optimize and process the target insight report, generate strategy information, and execute the strategy information to obtain the best analysis result.

[0011] In a third aspect, an embodiment of the present invention further provides a computing device, characterized in that it includes a processing component and a storage component; the storage component stores one or more computer instructions; the one or more computer instructions are used to be called and executed by the processing component to implement the AI ​​big data real-time processing and analysis method described in the first aspect.

[0012] In a fourth aspect, an embodiment of the present invention further provides a computer storage medium, characterized in that a computer program is stored therein, and when the computer program is executed by a computer, the method for real-time processing and analysis of AI big data described in the first aspect is implemented.

[0013] In an embodiment of the present application, real-time data streams are obtained from different data sources, and the real-time data streams are mapped into a preset data structure framework to obtain an initial data stream. The initial data stream is fused and processed by a real-time data fusion engine to obtain a target real-time data stream. Then, data mining and machine learning algorithms are used to perform pattern recognition and trend analysis on the target real-time data stream to generate a target insight report. The target insight report is optimized and processed using an intelligent decision-making unit, and strategy information is generated and executed to obtain the best analysis result. The technical solution provided by the present application not only solves the problem of integration and fusion of multi-source heterogeneous data, but also improves the real-time and accuracy of data analysis, realizes efficient conversion from data to decision, and significantly improves the intelligence level and response speed of business processes.

[0014] These and other aspects of the present application will become more clearly understood in the description of the following embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, a brief introduction will be given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0016] Figure 1 A flowchart of a method for real-time processing and analysis of AI big data provided in an embodiment of the present application; Figure 2 A schematic diagram of the structure of an AI big data real-time processing and analysis system provided in an embodiment of the present application; Figure 3 A schematic diagram of the structure of a computing device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0017] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application.

[0018] In some of the processes described in the specification and claims of this application and the above-mentioned figures, multiple operations that appear in a specific order are included, but it should be clearly understood that these operations may not be executed in the order in which they appear in this article or executed in parallel. The serial numbers of the operations, such as 101, 102, etc., are only used to distinguish between different operations, and the serial numbers themselves do not represent any execution order. In addition, these processes may include more or fewer operations, and these operations may be executed in sequence or in parallel. It should be noted that the descriptions of "first", "second", etc. in this article are used to distinguish different messages, devices, modules, etc., do not represent the order of precedence, and do not limit the "first" and "second" to be different types.

[0019] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of this application.

[0020] In modern enterprises and organizations, the processing and analysis of real-time data is becoming more and more important. With the diversification of data sources and the surge in data volume, traditional data processing methods have been unable to meet the requirements of real-time and accuracy. Based on this, the present invention provides a text data statistical analysis method based on natural language processing, such as Figure 1 As shown, the method comprises the following steps: Step S1, obtaining real-time data streams from different data sources, and mapping the real-time data streams into a preset data structure framework to obtain an initial data stream; In this step, advanced data collection tools are used to obtain real-time data streams from multiple different data sources such as sensors, log files, and social media, and then mapped into a unified data structure framework. Data cleaning techniques are used to remove noise, redundant information, and incomplete data to ensure data consistency and comparability, thus providing a standardized basis for subsequent data fusion and analysis. Step S2, fusing and processing the initial data streams through a real-time data fusion engine to obtain a target real-time data stream, wherein the real-time data fusion engine is used to dynamically adjust the combination mode of the initial data streams according to the time correlation and spatial correlation in the initial data streams; In this step, the real-time data fusion engine is used to fuse the initial data stream, and the timestamp and spatial location information in the data stream are identified and processed by combining advanced time series analysis algorithms and geographic information system technology. Through complex event processing technology and pattern matching, the combination of data streams is dynamically adjusted to eliminate redundancy and inconsistency, generate high-quality target real-time data streams, and ensure the temporal and spatial consistency of data; Step S3, using data mining and machine learning algorithms to perform pattern recognition and trend analysis on the target real-time data stream to generate a target insight report; In this step, advanced data mining and machine learning algorithms are used to conduct in-depth analysis of the target real-time data stream. Natural language processing, clustering, classification, and anomaly detection techniques are combined to identify patterns, trends, and anomalies in the data. Through time series analysis and predictive models, reports containing valuable insights are generated. Advanced visualization tools are used to present the analysis results in an intuitive way, enhancing the readability and practicality of the report.

[0021] Step S4, using an intelligent decision-making unit to optimize and process the target insight report, generate strategy information, and execute the strategy information to obtain the best analysis result; In this step, the insight report is optimized through the intelligent decision-making unit. First, natural language processing technology and semantic analysis are used to extract key information. Then, machine learning, rule engines, expert systems, and reinforcement learning algorithms are combined to generate preliminary decision recommendations. Multi-objective optimization technology is used to make trade-offs and adjustments to generate verified optimal strategy information. This strategy information is transmitted to the corresponding execution system through the automated execution system and API interface, and the execution status and performance indicators are collected through real-time monitoring and feedback mechanisms. Finally, interpretive AI technology and advanced visualization tools are used to generate easy-to-understand and highly operational analysis reports. To sum up, it not only solves the problem of data integration and fusion, but also improves the real-time and accuracy of data analysis. In addition, by generating and executing policy information through intelligent decision-making units, a complete closed-loop mechanism from data collection to decision-making execution is realized, which significantly improves the intelligence level and response speed of business processes.

[0022] Furthermore, the present invention provides a specific embodiment. According to step S2, the initial data stream is fused and processed by a real-time data fusion engine to obtain a target real-time data stream. The real-time data fusion engine is used to dynamically adjust the combination mode of the initial data stream according to the time correlation and spatial correlation in the initial data stream. The method includes the following steps: Step S21, using geographic information system technology to process the initial data stream to obtain a data stream with a unified spatial reference frame; In this step, the spatial location information in the initial data stream is extracted and converted through geographic information system technology to ensure that all data are based on a unified spatial reference frame, which solves the problem of inconsistent spatial location information in different data sources and provides a basis for subsequent temporal and spatial correlation analysis. Through a unified spatial reference frame, geographic location-related data analysis and fusion can be performed more accurately.

[0023] Step S22, using natural language processing technology and machine learning models to identify and parse the data stream of the unified spatial reference frame to obtain a data stream with time and space attributes; In this step, natural language processing technology and machine learning models are combined to further identify and parse the data stream of the unified spatial reference frame. Natural language processing technology is used to extract time and space related information from text data, while machine learning models are used to identify patterns and trends in the data. Through these technologies, implicit time and space information is made explicit, generating data streams with clear timestamps and geographic locations, thereby improving the analyzability of the data.

[0024] Step S23, using complex event processing technology and pattern matching to dynamically detect and intelligently combine data streams with time and space attributes to obtain optimized data streams; In this step, complex event processing technology and pattern matching algorithms are used to dynamically detect and intelligently combine data streams with time and space attributes. Complex event processing technology can monitor events in data streams in real time and perform correlation analysis based on predefined rules and patterns. Pattern matching algorithms are used to identify specific patterns and sequences in data streams, dynamically adjust the combination of data streams, eliminate redundancy and inconsistency, generate optimized data streams, and ensure the coherence and consistency of data in time and space. Step S24, using data cleaning and anomaly detection mechanisms to perform quality assurance processing on the optimized data stream, and using standardized protocols and data formats for processing to obtain the target real-time data stream.

[0025] In this step, the optimized data stream is quality-assured through data cleaning and anomaly detection mechanisms. Data cleaning technology is used to remove noise, redundant information, and incomplete data, while anomaly detection mechanisms are used to identify and process outliers in the data. These data cleaning and anomaly detection mechanisms ensure the quality and accuracy of the data. Finally, the data stream is processed using standardized protocols and data formats to generate target real-time data streams that meet the standards, providing a high-quality data foundation for subsequent analysis and decision-making. To sum up, it not only ensures the consistency and comparability of data, but also enhances the analyzability of data, and ultimately generates high-quality target real-time data streams, improving the real-time, accuracy and consistency of data processing.

[0026] Furthermore, the embodiment of the present invention also provides a specific embodiment. According to step S23, complex event processing technology and pattern matching are used to dynamically detect and intelligently combine data streams with time and space attributes to obtain optimized data streams. The method includes the following steps: Step S31, using complex event processing technology and advanced time series analysis algorithms to perform real-time monitoring, correlation analysis, and pattern recognition processing on events in the initial real-time data stream, and identifying combined data streams with similar timestamps and geographic locations; In this step, complex event processing technology is used to identify the complex relationships between multiple simple events, and correlation analysis is performed according to predefined rules and patterns. At the same time, advanced time series analysis algorithms are used to identify and predict the time correlation in the data. By processing the timestamp and geographic location information in the initial real-time data stream, combined data streams with similar timestamps and geographic locations are identified.

[0027] Step S32, dynamically grouping, aggregating and filtering the adjusted combined data stream in combination with time window management and spatial proximity constraints, while removing redundant and noisy data to obtain an optimized combined data stream; In this step, the data stream is divided into multiple time segments through time window management, and the data in each segment is analyzed as a window; Using spatial proximity constraints, we filter and process data based on the proximity of geographical locations, identify data points that are geographically close to each other, and then group data with similar timestamps and geographical locations together through dynamic grouping. We then aggregate data to generate higher-level statistical information, and remove redundant and noisy data through filtering mechanisms to obtain an optimized combined data stream to ensure data quality and relevance. Step S33, dynamically adjusting and optimizing the optimized combined data stream through an adaptive adjustment mechanism and a machine learning model to improve the combination strategy and obtain the best combined data stream; In this step, the adaptive adjustment mechanism monitors the changes in the data stream in real time and automatically adjusts the parameters and rules according to these changes; The machine learning model learns patterns and rules from historical data, and uses these patterns to predict and optimize new data. Through dynamic adjustment and optimization, it continuously improves the combination strategy to ensure that it always matches the state of the current data stream and generates the best combination data stream in terms of quality and relevance. Step S34, using real-time feedback and performance monitoring mechanisms to evaluate and optimize the optimal combined data stream to generate an optimized data stream; In this step, the real-time feedback mechanism can instantly collect and analyze the system operation results and feed these results back to the system for adjustment and optimization. The performance monitoring mechanism continuously monitors the system's performance indicators, such as response time, throughput, and resource utilization, to evaluate the system's operating status. Through these mechanisms, potential problems and anomalies can be quickly detected, and corresponding adjustments and optimizations can be made. Finally, after evaluation and optimization, the target real-time data stream is generated with the best quality and performance.

[0028] In summary, the embodiments of the present invention not only improve the real-time and accuracy of big data processing while ensuring timely capture of key events and patterns, but also can flexibly respond to data changes in different scenarios, improve the flexibility and adaptability of data processing, and continuously learn and adapt to data changes through adaptive adjustment and optimization, providing a reliable data foundation for practical applications and decision support.

[0029] Further, the present invention also provides a specific embodiment. According to the step S33, the optimized combined data stream is dynamically adjusted and optimized by an adaptive adjustment mechanism and a machine learning model to improve the combination strategy, so as to obtain the best combined data stream. The method comprises the following steps: Step S41, using an adaptive adjustment mechanism to perform real-time monitoring and analysis on the optimal combined data stream to obtain change trends and abnormal conditions; In this step, the adaptive adjustment mechanism automatically adjusts system parameters and behaviors to cope with changes in data and environment by continuously collecting and analyzing data streams. Through real-time monitoring of the best combined data stream, it can instantly discover changing trends in the data, such as rising, falling or stable trends, and identify abnormal situations that do not meet expectations; Furthermore, since traditional data analysis methods often have difficulty in quickly identifying patterns and trends in data when faced with large-scale, high-dimensional real-time data, an adaptive adjustment mechanism is also provided in an embodiment of the present invention. The expression of the adaptive adjustment mechanism is: ; in, It's time point The predicted value of Based on historical data and model parameters prediction function (such as LSTM or Transformer model), Based on the data change rate and parameters The trend analysis function of Based on the anomaly detection results and parameters The anomaly correction function of Based on multimodal data and parameters The multimodal fusion function of Based on the uncertainty estimate and parameters The uncertainty correction function is is an adaptive adjustment coefficient that changes over time and can be dynamically updated through reinforcement learning or other adaptive control methods; In this step, Use deep learning models (such as Transformer or LSTM) to model historical data to generate forecasts for future time points; By calculating the data change rate and using trend analysis models to extract the change trend; Use anomaly detection algorithms to identify abnormal data and correct the predicted values; Combine multiple types of data (such as time series data, text data, image data, etc.) through multimodal fusion models to extract comprehensive features; Use Bayesian methods or other uncertainty estimation techniques to estimate the uncertainty of the prediction results, and modify the prediction values ​​according to the uncertainty estimation results, and adaptively adjust the coefficients Used to dynamically adjust the weights of different components to ensure that they can be adaptively adjusted according to changes in the current data stream. These coefficients can be dynamically updated through reinforcement learning (such as Q-learning or DQN). Through this multi-dimensional comprehensive processing method, it can more effectively identify changing trends and abnormal situations in real-time monitoring and analysis, and make adaptive adjustments, thereby improving the overall data processing capabilities and decision support levels.

[0030] Step S42, using a machine learning model to predict and optimize the best combination data stream to obtain an initial combination strategy; In this step, the machine learning model identifies patterns and trends in the data by learning from historical and current data, and predicts the behavior of future data. At the same time, it optimizes and analyzes the data flow to improve the quality and consistency of the data flow. Finally, based on the results of these predictions and optimization processes, a preliminary combination strategy, namely the initial combination strategy, is generated. Step S43, using a reinforcement learning algorithm to simulate and evaluate the initial combination strategy, iteratively optimize and improve the initial combination strategy, and obtain a target combination strategy that adapts to the dynamic changes of the optimal combination data flow; In this step, the reinforcement learning algorithm gradually optimizes the behavior strategy through the interaction between the agent and the environment. By simulating and evaluating the initial combination strategy, the effectiveness and robustness of the strategy can be tested in a virtual environment. Based on the simulation results, multiple iterations of optimization are performed to continuously adjust and improve the strategy to make it more adaptable to the changes and needs of the actual data flow. Finally, after multiple rounds of iterative optimization, a target combination strategy that can adapt to the dynamic changes of the optimal combination data flow is generated; Step S44, using online learning technology to instantly learn and adjust the target combination strategy to obtain a dynamic combination strategy that quickly responds to the data pattern; In this step, online learning technology can learn and update model parameters from continuous data streams in real time, ensuring that it can quickly adapt to changes in the data stream. By learning the target combination strategy in real time, it can quickly capture new information and changes in the data stream and adjust the existing strategy based on this new information, including updating model parameters, adjusting rules and improving strategies. This real-time learning and adjustment process enables the system to quickly respond to patterns and trends in the data stream, thereby generating a dynamic combination strategy that can be updated and optimized in real time. Step S45, verifying and adjusting the dynamic combination strategy using feedback and performance monitoring mechanisms to obtain the best combination strategy; In this step, the feedback mechanism collects and analyzes the operation results, feeds back the results and makes adjustments and optimizations; the performance monitoring mechanism continuously monitors the system's performance indicators, such as response time, throughput and resource utilization, to evaluate the system's operation status. By verifying and evaluating the actual application effect of the dynamic combination strategy, necessary adjustments and optimizations can be made according to the feedback results. This verification and adjustment process can ensure that the strategy runs effectively and efficiently in the actual environment, and potential problems are discovered and resolved in a timely manner. Finally, after multiple rounds of verification and adjustment, the best combination strategy that can maintain high efficiency and accuracy under various conditions is generated; In summary, the embodiments of the present invention can not only respond quickly to changes in data streams, but also generate more accurate and reliable prediction results. Moreover, through multiple rounds of simulation and evaluation processing, as well as real-time learning and adjustment, it can better adapt to various complex environments. Finally, through feedback and performance monitoring mechanisms, the strategy is continuously optimized to ensure its efficiency and reliability in practical applications.

[0031] Since existing data mining and machine learning algorithms still have limitations in real-time processing and complex pattern recognition, the present invention also provides an embodiment. According to step S43, the initial combination strategy is simulated and evaluated using a reinforcement learning algorithm, and the initial combination strategy is iteratively optimized to obtain a target combination strategy that adapts to dynamic changes in data streams. The method includes the following steps: Step S51, using a reinforcement learning algorithm to perform multiple rounds of simulation processing on the initial combination strategy, identifying strategy performance and problems in different environments, and obtaining simulation results; In this step, by testing and evaluating the initial combination strategy multiple times in a virtual environment, we can fully understand the actual operation effect of the strategy under various conditions, including its advantages and disadvantages. The reinforcement learning algorithm gradually optimizes the behavior strategy through the interaction between the agent and the environment, and multiple rounds of simulation processing ensure that the strategy is fully tested in different environments; Step S52, using a reward mechanism to evaluate the simulation results and generate a feedback signal, and at the same time, the feedback signal iteratively updates the initial combination strategy, adjusts and optimizes the parameters of the initial combination strategy, and obtains an optimized combination strategy; In this step, the reward mechanism guides the agent’s behavior by giving it positive or negative feedback, and the evaluation process quantifies and evaluates the performance of the strategy based on the simulation results. The generated feedback signal can tell the agent whether its current behavior is good or bad, and guide it to take better actions in the future. Through multiple iterative update processes, the strategy parameters are gradually adjusted and optimized to generate an optimized combination strategy that can show higher performance and robustness under various conditions. Step S53, through the simulation and evaluation process of the prediction number, the optimized combination strategy is adapted to the best combination data flow to obtain the target combination strategy; In this step, the performance of the optimized combination strategy is repeatedly predicted and evaluated in multiple simulations to ensure its adaptability and effectiveness in the optimal combination data stream. Through this process, the performance of the strategy in different situations can be fully tested and verified, and the strategy parameters can be gradually adjusted and optimized to make it more adaptable to the changes and needs of the actual data stream. Finally, after multiple rounds of prediction and evaluation, a target combination strategy is generated that can run efficiently in the optimal combination data stream and adapt to various changes; Step S54, using the verification set to verify the target combination strategy to obtain the best combination strategy; In this step, the validation set is a set of new data independent of the training and test data, which is used to evaluate the performance of the target combination strategy in the actual environment. The performance of the strategy on the new data is fully tested through the validation process, and necessary adjustments and optimizations are made based on the validation results. After the validation process, the best combination strategy is generated that can run efficiently in actual applications and provide optimal performance.

[0032] Furthermore, the present invention also provides an embodiment, according to step S3, using a reinforcement learning algorithm to simulate and evaluate the initial combination strategy, iteratively optimize and improve the initial combination strategy, and obtain a target combination strategy that adapts to the dynamic changes of the data stream. The method includes the following steps: Step S61, using data preprocessing technology to clean, denoise and standardize the target real-time data stream to obtain analysis data; In this step, data preprocessing technology is used to clean the target real-time data stream, remove noise and outliers, and perform standardization to ensure data consistency and comparability. The cleaning process includes deleting or correcting incomplete, erroneous or duplicate data records, while denoising involves filtering out random fluctuations and interference signals in the data. Standardization converts the data into a unified format and scale to generate high-quality analytical data. Step S62: extracting key features based on the analysis data, and selecting a feature subset from the key features to obtain data representation; In this step, key features that can represent the essence of the data are extracted from the analyzed data. These features are the most informative part of the data and can reflect the main attributes and changes of the data. Through feature selection technology, the most representative feature subset is further selected from these key features to form data representation. Feature selection can reduce data dimensions, improve model training efficiency and generalization ability, and retain the most important information. Step S63, using clustering and classification algorithms to perform pattern recognition processing on the data representation, identifying implicit patterns and categories in the data representation, and obtaining pattern recognition results; In this step, clustering and classification algorithms are used to perform pattern recognition processing on the data representation. Clustering algorithms (such as K-means, DBSCAN, etc.) divide the data into multiple clusters, and the data in each cluster has similar characteristics. Classification algorithms (such as decision trees, support vector machines, neural networks, etc.) are used to classify the data into predefined categories. These algorithms are used to identify implicit patterns and categories in the data representation, reveal the internal structure and relationships of the data, and generate pattern recognition results.

[0033] Step S64, performing trend analysis processing on the pattern recognition result, identifying long-term trends and short-term fluctuations in the data representation, and obtaining trend analysis results; In this step, the pattern recognition results are processed for trend analysis to identify long-term trends and short-term fluctuations in the data. Trend analysis usually involves time series analysis methods, such as moving average, exponential smoothing, ARIMA model, etc. These methods are used to discover the change patterns of data over time and identify long-term trends (such as growth, decline or stability) and short-term fluctuations (such as seasonal changes and cyclical fluctuations). Finally, trend analysis results are generated to provide a comprehensive understanding of data changes.

[0034] Step S65, using natural language processing technology and report generation templates, the pattern recognition results and trend analysis results are integrated and interpreted to generate a target insight report.

[0035] In this step, natural language processing technology and predefined report generation templates are used to integrate and interpret pattern recognition results and trend analysis results. Natural language processing technology can transform complex data analysis results into easy-to-understand natural language descriptions, and report generation templates provide a structured framework to ensure the clarity and consistency of report content. These technologies are used to generate detailed target insight reports that not only contain the results of data analysis, but also provide explanations and suggestions for the results, helping decision makers better understand and apply these insights.

[0036] Furthermore, the present invention also provides an embodiment, according to the step S4, the target insight report is optimized and processed by using an intelligent decision-making unit, strategy information is generated, and the strategy information is executed to obtain the best analysis result, the method comprising the following steps: Step S71, using natural language processing technology and semantic analysis to deeply understand the target insight report and extract key information to obtain a structured data representation; In this step, natural language processing technology and semantic analysis methods are used to deeply understand the generated target insight report. Natural language processing technology can parse text content, identify and extract key information, such as business indicators, trend descriptions, pattern recognition results, etc. Through semantic analysis, the context and meaning of this information are understood and converted into structured data representation for subsequent processing and analysis. Step S72, performing pattern recognition, trend analysis and risk assessment on the structured data representation to identify business influencing factors and opportunities; In this step, further pattern recognition, trend analysis and risk assessment are performed on the structured data representation. Pattern recognition is used to discover repeated patterns and regularities in the data, trend analysis is used to identify long-term trends and short-term fluctuations, and risk assessment helps identify potential risk factors. Through these analyses, key factors affecting the business and potential opportunities are identified, providing important basis for decision-making; Step S73, performing logical reasoning, rule matching, and simulation optimization on the identified business influencing factors and opportunities to generate preliminary decision suggestions; In this step, we use logical reasoning, rule matching and simulation optimization technology to conduct in-depth analysis of the identified business influencing factors and opportunities. Logical reasoning derives causal relationships and potential impacts, rule matching verifies and screens according to predefined business rules, and simulation optimization simulates the effects of strategies in different scenarios to find the optimal solution. Finally, we generate preliminary decision recommendations to provide guidance for actual operations. Step S74, adjusting the preliminary decision suggestion to generate optimal strategy information; In this step, the preliminary decision suggestions are further adjusted and optimized to ensure that they meet actual business needs and environmental changes. The adjustment process may involve parameter fine-tuning, strategy combination optimization, and interactive feedback with business experts to generate the final optimal strategy information to ensure its effectiveness and feasibility in actual applications; Step S75, using the automated execution system and API interface to output the best strategy information to the execution system in real time, and generate execution results through real-time and performance indicators; In this step, the best strategy information is transmitted to the execution system in real time through the automated execution system and API interface, and the automated execution strategy is converted into specific operation instructions and executed in the actual environment. At the same time, by monitoring the actual situation and performance indicators (such as response time, resource utilization, task completion rate, etc.), data during the execution process is collected and detailed execution results are generated for further analysis and evaluation; Step S76, interpreting and visually displaying the execution results in multiple dimensions to generate the best analysis results; In this step, the execution results are explained and visualized in multiple dimensions. The complex execution results are presented to users in an intuitive manner through charts, dashboards and other visualization tools. This not only helps users better understand the execution effects, but also provides a multi-dimensional perspective, making it easier for users to analyze and interpret data from different angles, ultimately generating the best analysis results and providing users with comprehensive data support and decision-making basis.

[0037] In summary, the embodiments of the present invention provide a comprehensive and efficient decision support process, which not only improves the accuracy and efficiency of decision-making, but also enhances the robustness and adaptability of the system, provides a solid data foundation for the operation and management of the enterprise, and helps to improve overall business performance and competitiveness.

[0038] Figure 2 A structural diagram of an AI big data real-time processing and analysis system is provided for an embodiment of the present application, such as Figure 2 As shown, the system includes: The acquisition module 21 acquires real-time data streams from different data sources and maps the real-time data streams into a preset data structure framework to obtain an initial data stream; A fusion module 22, which fuses and processes the initial data streams through a real-time data fusion engine to obtain a target real-time data stream, wherein the real-time data fusion engine is used to dynamically adjust the combination mode of the initial data streams according to the time correlation and space correlation in the initial data streams; A generation module 23 performs pattern recognition and trend analysis on the target real-time data stream using data mining and machine learning algorithms to generate a target insight report; The execution module 24 uses an intelligent decision-making unit to optimize and process the target insight report, generate strategy information, and execute the strategy information to obtain the best analysis result.

[0039] Figure 2 The AI ​​big data real-time processing and analysis system can perform Figure 1 The implementation principle and technical effects of the AI ​​big data real-time processing and analysis method described in the embodiment are not repeated here. The specific way in which each module and unit performs operations in the AI ​​big data real-time processing and analysis system in the above embodiment has been described in detail in the embodiment of the method, and will not be elaborated here.

[0040] In one possible design, Figure 2 The AI ​​big data real-time processing and analysis system of the embodiment shown can be implemented as a computing device, such as Figure 3 As shown, the computing device may include a storage component 31 and a processing component 32; The storage component 31 stores one or more computer instructions, wherein the one or more computer instructions are called and executed by the processing component 32 .

[0041] The processing component 32 is used to: obtain real-time data streams from different data sources, and map the real-time data streams into a preset data structure framework to obtain an initial data stream; fuse and process the initial data stream through a real-time data fusion engine to obtain a target real-time data stream, and the real-time data fusion engine is used to dynamically adjust the combination of the initial data streams according to the time correlation and spatial correlation in the initial data stream; use data mining and machine learning algorithms to perform pattern recognition and trend analysis on the target real-time data stream to generate a target insight report; use an intelligent decision-making unit to optimize the processing of the target insight report, generate strategy information, and execute the strategy information to obtain the best analysis result.

[0042] The processing component 32 may include one or more processors to execute computer instructions to complete all or part of the steps in the above method. Of course, the processing component may also be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors or other electronic components to perform the above method.

[0043] The storage component 31 is configured to store various types of data to support operations at the terminal. The storage component can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0044] Of course, the computing device may also include other components, such as input / output interfaces, display components, communication components, etc.

[0045] The input / output interface provides an interface between the processing component and the peripheral interface module, which may be an output device, an input device, etc.

[0046] The communication component is configured to facilitate, among other things, wired or wireless communications between the computing device and other devices.

[0047] Among them, the computing device can be a physical device or an elastic computing host provided by a cloud computing platform, etc. In this case, the computing device can refer to a cloud server, and the above-mentioned processing components, storage components, etc. can be basic server resources rented or purchased from the cloud computing platform.

[0048] The present application also provides a computer storage medium storing a computer program, wherein the computer program can achieve the above-mentioned Figure 1 An AI big data real-time processing and analysis method of the illustrated embodiment.

[0049] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0050] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0051] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0052] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for real-time processing and analysis of AI big data, characterized in that: include: Acquire real-time data streams from different data sources, and map the real-time data streams into a preset data structure framework to obtain an initial data stream; The initial data streams are fused and processed by a real-time data fusion engine to obtain a target real-time data stream, wherein the real-time data fusion engine is used to dynamically adjust the combination mode of the initial data streams according to the time correlation and space correlation in the initial data streams; Using data mining and machine learning algorithms to perform pattern recognition and trend analysis on the target real-time data stream to generate a target insight report; The target insight report is optimized and processed using an intelligent decision-making unit to generate strategy information and execute the strategy information to obtain the best analysis result.

2. The method according to claim 1, characterized in that The initial data stream is fused and processed by a real-time data fusion engine to obtain a target real-time data stream, wherein the real-time data fusion engine is used to dynamically adjust the combination mode of the initial data stream according to the time correlation and space correlation in the initial data stream, including: Processing the initial data stream using geographic information system technology to obtain a data stream with a unified spatial reference frame; Using natural language processing technology and machine learning models, the data stream of the unified spatial reference frame is identified and parsed to obtain a data stream with time and space attributes. Use complex event processing technology and pattern matching to dynamically detect and intelligently combine data streams with time and space attributes to obtain optimized data streams; The optimized data stream is subjected to quality assurance processing using data cleaning and anomaly detection mechanisms, and is processed using standardized protocols and data formats to obtain a target real-time data stream.

3. The method according to claim 2, characterized in that Complex event processing technology and pattern matching are used to dynamically detect and intelligently combine data streams with time and space attributes to obtain optimized data streams, including: Use complex event processing technology and advanced time series analysis algorithms to perform real-time monitoring, correlation analysis, and pattern recognition processing on events in the initial real-time data stream, and identify combined data streams with similar timestamps and geographic locations; Use pattern matching algorithms and rule engines to perform multi-dimensional pattern recognition and rule matching on combined data streams, determine the association relationship and combination method between combined data streams, and dynamically adjust them through a custom rule library to obtain the adjusted combined data stream; Dynamically grouping, aggregating and filtering the adjusted combined data stream in combination with time window management and spatial proximity constraints, while removing redundant and noisy data to obtain an optimized combined data stream; Dynamically adjust and optimize the optimized combined data stream through an adaptive adjustment mechanism and a machine learning model to improve the combination strategy and obtain the best combined data stream; The best combined data stream is evaluated and optimized using a real-time feedback and performance monitoring mechanism to generate an optimized data stream.

4. The method according to claim 3, characterized in that The optimized combined data stream is dynamically adjusted and optimized by an adaptive adjustment mechanism and a machine learning model to improve the combination strategy, thereby obtaining the best combined data stream, including: Using an adaptive adjustment mechanism to monitor and analyze the optimal combined data stream in real time to obtain change trends and abnormal conditions; Use machine learning models to predict and optimize the best combination data stream to obtain the initial combination strategy; The initial combination strategy is simulated and evaluated by using a reinforcement learning algorithm, and the initial combination strategy is iteratively optimized and improved to obtain a target combination strategy that adapts to the dynamic changes of the optimal combination data flow; Using online learning technology to instantly learn and adjust the target combination strategy, to obtain a dynamic combination strategy that quickly responds to data patterns; The dynamic combination strategy is verified and adjusted by using feedback and performance monitoring mechanisms to obtain the best combination strategy.

5. The method according to claim 4, characterized in that The reinforcement learning algorithm is used to simulate and evaluate the initial combination strategy, and the initial combination strategy is iteratively optimized to obtain the target combination strategy that adapts to the dynamic changes of data flow, including: Using a reinforcement learning algorithm to perform multiple rounds of simulation on the initial combination strategy, identifying strategy performance and problems in different environments, and obtaining simulation results; The simulation results are evaluated and processed by using a reward mechanism to generate a feedback signal, and the feedback signal is used to iteratively update the initial combination strategy, adjust and optimize the parameters of the initial combination strategy, and obtain an optimized combination strategy; Through the simulation and evaluation process of the forecast times, the optimized combination strategy is adapted to the best combination data flow to obtain the target combination strategy.

6. The method according to claim 1, characterized in that Use data mining and machine learning algorithms to perform pattern recognition and trend analysis on the target real-time data stream to generate a target insight report, including: Use data preprocessing technology to clean, denoise and standardize the target real-time data stream to obtain analytical data; Extracting key features based on the analysis data, and selecting a feature subset from the key features to obtain data representation; Use clustering and classification algorithms to perform pattern recognition on data representation, identify implicit patterns and categories in data representation, and obtain pattern recognition results; Performing trend analysis on the pattern recognition results to identify long-term trends and short-term fluctuations in the data representation and obtain trend analysis results; Use natural language processing technology and report generation templates to integrate and interpret pattern recognition and trend analysis results to generate targeted insight reports.

7. The method according to claim 1, characterized in that The target insight report is optimized and processed using an intelligent decision-making unit to generate strategy information and execute the strategy information to obtain the best analysis result, including: Use natural language processing technology and semantic analysis to deeply understand and extract key information from target insight reports to obtain structured data representation; Performing pattern recognition, trend analysis, and risk assessment on the structured data representation to identify business impact factors and opportunities; Perform logical reasoning, rule matching, and simulation optimization on the identified business influencing factors and opportunities to generate preliminary decision recommendations; Adjusting the preliminary decision suggestions to generate optimal strategy information; Use the automated execution system and API interface to output the best strategy information to the execution system in real time, and generate execution results through real-time and performance indicators; The execution results are explained and visualized in multiple dimensions to generate the best analysis results.

8. An AI big data real-time processing and analysis system, characterized in that: include: An acquisition module acquires real-time data streams from different data sources and maps the real-time data streams into a preset data structure framework to obtain an initial data stream; A fusion module, which fuses and processes the initial data streams through a real-time data fusion engine to obtain a target real-time data stream, wherein the real-time data fusion engine is used to dynamically adjust the combination mode of the initial data streams according to the time correlation and space correlation in the initial data streams; A generation module, using data mining and machine learning algorithms to perform pattern recognition and trend analysis on the target real-time data stream to generate a target insight report; The execution module uses an intelligent decision-making unit to optimize and process the target insight report, generate strategy information, and execute the strategy information to obtain the best analysis result.

9. A computing device, characterized in that It comprises a processing component and a storage component; the storage component stores one or more computer instructions; the one or more computer instructions are used to be called and executed by the processing component to implement an AI big data real-time processing and analysis method as described in any one of claims 1 to 7.

10. A computer storage medium, characterized in that: A computer program is stored, and when the computer program is executed by a computer, an AI big data real-time processing and analysis method as described in any one of claims 1 to 7 is implemented.

Images