Big data analysis processing method and system based on artificial intelligence

Abstract

The invention provides a big data analysis processing method and system based on artificial intelligence. Wherein business demand information of a user is collected in multiple ways; comprehensively analyzing the business demand information based on an artificial intelligence technology, and generating a data analysis target; based on the data analysis target, utilizing an association rule learning technology to construct an association relationship between data, and adopting an incremental learning algorithm to process a newly added data stream to generate an optimized data structure; based on the optimized data structure, deep analysis is carried out by using a multi-modal analysis model, model parameters are dynamically adjusted by using an online transfer learning algorithm, and a high-quality analysis intermediate result is generated; and based on the high-quality analysis intermediate result, constructing a multi-dimensional visual interface, and generating an intelligent decision support system. According to the big data analysis processing method and system based on artificial intelligence, multi-source data can be comprehensively and efficiently processed and analyzed, high-quality intelligent decision support is provided, and the business operation efficiency is remarkably improved.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of big data analysis technology, and in particular to a big data analysis and processing method and system based on artificial intelligence. Background Art

[0002] With the development of big data and artificial intelligence technology, enterprises need to obtain and integrate business demand information from multiple channels to ensure the comprehensiveness and accuracy of the information. Based on this information, enterprises need to generate clear data analysis goals for in-depth data analysis and decision support. In addition, real-time integration of multi-source raw data, building relationships between data, and processing newly added data streams are the keys to achieving efficient data management. Through multimodal analysis models and online transfer learning algorithms, enterprises can deeply explore potential value information, generate high-quality intermediate analysis results, and ultimately build a multi-dimensional visualization interface to provide predictive analysis and intelligent decision support systems.

[0003] Currently, many companies have adopted a variety of technologies and methods to process and analyze business demand information; through various data collection tools and platforms, business demand information is collected from multiple channels, and preliminary data cleaning and integration is performed; traditional statistical analysis and machine learning methods are used to analyze the collected data and generate preliminary analysis reports; the analysis results are presented to users in the form of charts and reports, supporting basic interactive exploration.

[0004] Although existing solutions have met some of the needs of enterprises to a certain extent, they often rely on a single data source and lack comprehensive integration of multi-source data, resulting in one-sided and incomplete information; traditional statistical analysis and machine learning methods are difficult to deeply explore the complex relationships between data and cannot provide deep insights and valuable information; existing solutions usually adopt batch processing methods and cannot achieve real-time data processing and dynamic adjustment, resulting in limited adaptability and predictive ability of the model; existing visualization tools are relatively simple in function and lack multi-dimensional interactive exploration capabilities, making it difficult for users to obtain a comprehensive analytical perspective and support. Summary of the invention

[0005] The embodiments of the present application provide a big data analysis and processing method and system based on artificial intelligence to solve the problem of lack of comprehensive integration of multi-source data in the prior art.

[0006] In a first aspect, the present application provides an artificial intelligence-based big data analysis and processing method, including: Collect users' business demand information through multiple channels; Based on artificial intelligence technology and combined with external influencing factors, the business demand information is comprehensively analyzed to generate data analysis goals; Based on the data analysis objectives, the original data from multiple sources are integrated in real time, the association relationship between data is constructed using association rule learning technology, and the newly added data stream is processed using incremental learning algorithms to generate optimized data structures; Based on the optimized data structure, a multimodal analysis model is used to conduct in-depth analysis, mine potential valuable information, and an online transfer learning algorithm is used to dynamically adjust model parameters to generate high-quality intermediate analysis results; Based on the high-quality intermediate analysis results, a multi-dimensional visualization interface is constructed to support user interactive exploration, provide predictive analysis, and generate an intelligent decision support system.

[0007] In a second aspect, the present application provides an artificial intelligence-based big data analysis and processing system, including: Collection module, used to collect users' business demand information through multiple channels; An analysis module is used to comprehensively analyze the business demand information based on artificial intelligence technology and in combination with external influencing factors to generate data analysis targets; A processing module is used to integrate multi-source raw data in real time based on the data analysis target, use association rule learning technology to build association relationships between data, use incremental learning algorithms to process newly added data streams, and generate optimized data structures; An adjustment module is used to use a multimodal analysis model to conduct in-depth analysis based on the optimized data structure, mine potential valuable information, and dynamically adjust model parameters using an online transfer learning algorithm to generate high-quality intermediate analysis results; The prediction module is used to build a multi-dimensional visualization interface based on the high-quality intermediate analysis results, support user interactive exploration, provide predictive analysis, and generate an intelligent decision support system.

[0008] In a third aspect, an embodiment of the present application provides a computing device, comprising a processing component and a storage component; the storage component stores one or more computer instructions; the one or more computer instructions are used to be called and executed by the processing component to implement an artificial intelligence-based big data analysis and processing method as described in any one of claims 1 to 7.

[0009] In a fourth aspect, an embodiment of the present application provides a computer storage medium storing a computer program. When the computer program is executed by a computer, it implements an artificial intelligence-based big data analysis and processing method as described in any one of claims 1 to 7.

[0010] In an embodiment of the present application, business demand information of users is collected through multiple channels; based on artificial intelligence technology, the business demand information is comprehensively analyzed in combination with external influencing factors to generate data analysis targets; based on the data analysis targets, multi-source raw data is integrated in real time, association rule learning technology is used to build association relationships between data, and an incremental learning algorithm is used to process newly added data streams to generate an optimized data structure; based on the optimized data structure, a multimodal analysis model is used for in-depth analysis to mine potential valuable information, and an online transfer learning algorithm is used to dynamically adjust model parameters to generate high-quality analysis intermediate results; based on the high-quality analysis intermediate results, a multi-dimensional visualization interface is constructed to support user interactive exploration, provide predictive analysis, and generate an intelligent decision support system. Collect users' business demand information through multiple channels to ensure the comprehensiveness and accuracy of the demand, and provide a solid foundation for subsequent analysis and decision-making; Based on artificial intelligence technology, combined with external influencing factors, comprehensively analyze business demand information, generate clear data analysis goals, and improve the pertinence and effectiveness of analysis; Integrate multi-source raw data in real time, use association rule learning technology to build the association relationship between data, ensure the integrity and consistency of data, and provide high-quality data support for subsequent processing; Use incremental learning algorithms to process newly added data streams and generate optimized data structures to ensure that the model can quickly adapt to new data and improve the real-time and flexibility of the system; Based on the optimized data structure, use multimodal analysis models for in-depth analysis, mine potential value information, provide more comprehensive and in-depth insights, and support more accurate decision-making; Use online transfer learning algorithms to dynamically adjust model parameters to ensure continuous optimization and adaptability of the model, and improve the model's prediction accuracy and generalization ability; Build a multi-dimensional visualization interface to support user interactive exploration, provide intuitive data display and analysis tools, and enhance user experience and the convenience of decision support; Provide predictive analysis and generate intelligent decision support systems to help users identify potential risks and opportunities in advance, formulate scientific and reasonable strategies, and improve the scientificity and effectiveness of decision-making.

[0011] Based on the data analysis objectives, real-time integration of multi-source raw data, use of association rule learning technology to build data association relationships, use of incremental learning algorithms to process newly added data streams, and generate optimized data structures. Specifically, it includes: real-time monitoring and collection of multi-source data to generate comprehensive data sets; data preprocessing, use of association rule learning technology to discover implicit association patterns, and generate data association models; use of incremental learning algorithms to dynamically process new data streams, update data models in real time, and generate dynamic data association models; and adjustment of data structures through optimization algorithms to eliminate data redundancy and ensure data processing efficiency and accuracy. Real-time monitoring and collection of multi-source data ensures the timeliness and comprehensiveness of data and promptly reflects the latest business status; performs data preprocessing to improve data quality, reduce noise and errors, and ensure the accuracy of subsequent analysis; uses association rule learning technology to discover implicit association patterns between data and provide deeper business insights; uses incremental learning algorithms to dynamically process new data streams and update data models in real time to ensure the adaptability and predictive accuracy of the models; adjusts data structures through optimization algorithms to eliminate data redundancy, improves data processing efficiency and accuracy, and ensures high performance and reliability of the system; optimizes data structures to reduce the waste of storage and computing resources and improves the overall operating efficiency of the system; high-quality data processing and analysis results provide more reliable support for decision makers and improve the scientific nature and effectiveness of decision-making.

[0012] Based on the optimized data structure, use the multimodal analysis model to generate a multimodal analysis architecture; conduct in-depth analysis of the optimized data structure, explore potential valuable information, and generate initial analysis data; based on the initial analysis data, use the online transfer learning algorithm combined with the new data stream to dynamically adjust the model parameters and generate an adaptive model parameter configuration; continuously monitor and evaluate the model performance, use advanced data processing methods to optimize the initial analysis data, and generate high-quality intermediate analysis results. Use multimodal analysis models to comprehensively process various types of data features, generate multimodal analysis architectures, and improve the comprehensiveness and accuracy of data processing; conduct in-depth analysis of optimized data structures, explore potential valuable information, and provide richer business insights; use online transfer learning algorithms combined with new data streams to dynamically adjust model parameters to ensure the adaptability and predictive capabilities of the model and improve the real-time and accuracy of the model; continuously monitor and evaluate model performance, use advanced data processing methods to optimize initial analysis data, generate high-quality analysis intermediate results, and provide a reliable basis for decision-making; ensure the performance and stability of the model and improve the overall performance of the system by dynamically adjusting model parameters and continuous monitoring; high-quality analysis intermediate results provide decision makers with more accurate and comprehensive information, enhancing the scientific nature and effectiveness of decision-making; optimized data processing methods reduce the waste of computing resources and improve the operating efficiency and response speed of the system.

[0013] These and other aspects of the present application will become more clearly understood in the description of the following embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, a brief introduction will be given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0015] Figure 1 A flowchart of a big data analysis and processing method based on artificial intelligence provided in an embodiment of the present application; Figure 2 A schematic diagram of the structure of a big data analysis and processing system based on artificial intelligence provided in an embodiment of the present application; Figure 3 A schematic diagram of the structure of a computing device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0016] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application.

[0017] In some of the processes described in the specification and claims of this application and the above-mentioned figures, multiple operations that appear in a specific order are included, but it should be clearly understood that these operations may not be executed in the order in which they appear in this article or executed in parallel. The serial numbers of the operations, such as 101, 102, etc., are only used to distinguish between different operations, and the serial numbers themselves do not represent any execution order. In addition, these processes may include more or fewer operations, and these operations may be executed in sequence or in parallel. It should be noted that the descriptions of "first", "second", etc. in this article are used to distinguish different messages, devices, modules, etc., do not represent the order of precedence, and do not limit the "first" and "second" to be different types.

[0018] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of this application.

[0019] Figure 1 A flowchart of a big data analysis and processing method based on artificial intelligence is provided for the embodiment of the present application, such as Figure 1As shown, the method includes: 101. Collect users’ business demand information through multiple channels; Ensure that comprehensive and accurate business demand information is collected; collect users' business demand information through multiple channels; and provide basic data support for subsequent data analysis and decision-making.

[0020] 102. Based on artificial intelligence technology and combined with external influencing factors, comprehensively analyze the business demand information and generate data analysis objectives; Generate clear data analysis goals to guide subsequent data processing and analysis work; use artificial intelligence technology, combined with external influencing factors, to conduct a comprehensive analysis of business demand information; ensure the comprehensiveness and accuracy of analysis goals, and provide a clear direction for subsequent steps.

[0021] 103. Based on the data analysis objectives, real-time integration of multi-source raw data, using association rule learning technology to build association relationships between data, using incremental learning algorithms to process newly added data streams, and generating optimized data structures; Generate optimized data structures to ensure the real-time and accuracy of data; monitor and collect multiple data sources in real time to generate comprehensive data sets; perform data cleaning, formatting, missing value processing and other operations to improve data quality; use association rule learning technology to build associations between data and discover implicit association patterns; use incremental learning algorithms to dynamically process newly added data streams and update data models in real time.

[0022] Optionally, the step 103, based on the data analysis target, real-time integration of multi-source raw data, building association relationships between data using association rule learning technology, and using incremental learning algorithms to process newly added data streams to generate optimized data structures, includes: Based on the data analysis objectives, real-time monitoring and collection of multiple data sources are performed, and multi-source raw data are integrated to generate a comprehensive data set; Based on the comprehensive data set, data preprocessing is performed to improve data quality, association rule learning technology is used to construct association relationships between data, implicit association patterns between data are discovered, and a data association model is generated; Based on the data association model, an incremental learning algorithm is used to dynamically process newly added data streams, update the data model in real time, ensure that the data structure reflects the latest business status in real time, and generate a dynamic data association model; Based on the dynamic data association model, the data structure is adjusted through an optimization algorithm to eliminate data redundancy, ensure the efficiency and accuracy of data processing, and generate an optimized data structure.

[0023] Suppose there is a national grid company that needs to monitor and optimize grid operation in real time to ensure the stability and efficiency of power supply; Through sensors and data acquisition systems, the required data is monitored and collected in real time, and a comprehensive data set is generated to ensure the comprehensiveness and timeliness of the data, providing real-time data support for power grid operation; data cleaning is performed on the comprehensive data set to remove invalid and erroneous data; formatting is performed to ensure data consistency and compatibility; missing values ​​are processed, and blank values ​​are reasonably filled or deleted to ensure data accuracy and consistency, reduce noise and errors, and improve the reliability of subsequent analysis; association rule learning technology is used to analyze the association between power grid operation data, user electricity consumption data, and weather data, discover potential association patterns, generate data association models, provide deeper business insights, and discover potential problems and optimization opportunities in power grid operation; based on the data association model, an incremental learning algorithm is used to dynamically process newly added data streams, update the data model in real time, and generate a dynamic data association model; ensure that the data structure reflects the latest power grid operation status in real time, improve the adaptability and prediction ability of the model, and support real-time decision-making; based on the dynamic data association model, the data structure is adjusted through an optimization algorithm, data redundancy is eliminated, the efficiency and accuracy of data processing are ensured, and an optimized data structure is generated; the efficiency and accuracy of data processing are improved, resource waste is reduced, the high performance and reliability of the system are ensured, and the efficient operation and management of the power grid are supported.

[0024] Through the above steps, State Grid Corporation of China can monitor and process grid operation data in real time, identify potential problems, optimize grid operation, improve energy utilization efficiency, and ensure the safe and stable operation of the grid.

[0025] This application takes into account that the formula is used to calculate the nonlinear dynamic adjustment factor, and the dynamic data association model is generated through dynamic processing; in power companies, data processing and model optimization is a continuous process, and when faced with a large number of new data streams, traditional static models are difficult to adapt to new business conditions; by combining the Fisher information matrix and the nonlinear dynamic adjustment factor, it is ensured that the model can reflect the latest business data in real time; Optionally, based on the data association model, an incremental learning algorithm is used to dynamically process newly added data streams, update the data model in real time, ensure that the data structure reflects the latest business status in real time, and generate a dynamic data association model, including: Define the task loss function , providing timely feedback during model training, accelerating model convergence, and providing support for the calculation of elastic weight consolidation loss function; Setting the Fisher information matrix , dynamically evaluate the importance of parameters to ensure the accuracy and stability of model parameter updates; The elastic weight consolidation loss function is calculated using the following formula: ; in, Consolidate loss functions for elastic weights; is the task loss function, which is the mean square error or cross entropy loss; is the th model parameter; For the previous task The best values ​​of the model parameters; is the diagonal element of the Fisher information matrix, which is used to measure the importance of the parameters; is a regularization parameter used to balance the weight of task loss and parameter stability; is the weight of the nonlinear term, which is used to adjust the impact of the nonlinear term; is the parameter of nonlinear transformation, which is used to adjust the shape of nonlinear transformation; is the index of the model parameter, from 1 to ; is the total number of model parameters; The Fisher information matrix is ​​calculated using the following formula: ; in, is the Fisher information matrix, The diagonal elements represent; For input data; is the data distribution; is the task loss function; For the model parameters; is the weight of the sine term, which is used to adjust the influence of the sine term; is the frequency of the sine function; is the phase shift of the sine function; is the task loss function For Model parameters The partial derivative of reflects the parameter The degree of influence on the loss function; For all input data According to its distribution Calculated expectations; Based on the elastic weight consolidation loss function and the Fisher information matrix, the Gaussian term is introduced to construct a multi-factor comprehensive nonlinear relationship, and the nonlinear dynamic adjustment factor is calculated by the following formula: ; in, For the Nonlinear dynamic adjustment factors for model parameters; is the learning rate, which controls the step size of parameter update; Consolidate loss functions for elastic weights; For the model parameters; For the previous task The best values ​​of the model parameters; is the weight of the sine term, which is used to adjust the influence of the sine term; is the frequency of the sine function; is the phase shift of the sine function; is the weight of the logarithmic term, which is used to adjust the impact of the logarithmic term; is the weight of the hyperbolic tangent term, which is used to adjust the influence of the hyperbolic tangent term; is the slope of the hyperbolic tangent function; is the weight of the Fisher information matrix item; is the Fisher information matrix, The diagonal elements represent; is the weight of the Gaussian term, which is used to adjust the influence of the Gaussian term; is the standard deviation of the Gaussian function; Consolidating loss functions for elastic weights For Model parameters The partial derivative of reflects the parameter The degree of influence on the elastic weight consolidation loss function; Based on the nonlinear dynamic adjustment factor , continuously updating the model parameters , dynamically adjust in combination with new data streams to ensure that the data structure reflects the latest business conditions in real time and generate a dynamic data association model.

[0026] This method aims to dynamically process newly added data streams and update the data model in real time to ensure that the model can quickly adapt to new business conditions, while maintaining effective memory of old data to avoid catastrophic forgetting; in this way, it can ensure that the data structure reflects the latest business conditions in real time, improving the timeliness and accuracy of the model.

[0027] Consolidation loss function in elastic weights In the task loss function : reflects the difference between the model's predicted output and the actual output, and is the basis for optimizing model performance by minimizing , which can ensure the performance of the model on new tasks; the regularization term : By penalizing the deviation between the model parameters and the optimal parameters of the old task, the model is prevented from excessively changing the important parameters in the old task when learning the new task, thereby reducing catastrophic forgetting; nonlinear terms : Introducing nonlinear components enables the model to explore more possible solutions in the parameter space and enhance the generalization ability of the model; In the Fisher Information Matrix In, expect : By calculating the expectation of all input data according to their distribution, the statistical significance of the Fisher information matrix is ​​ensured and the accuracy of parameter importance assessment is improved; partial derivatives : reflects the influence of model parameters on task loss and is the key basis for evaluating the importance of parameters; the sine term : This item is introduced to increase the randomness in the process of updating model parameters, which helps to jump out of the local optimal solution and find the global optimal solution; In the nonlinear dynamic adjustment factor In the learning rate : Control the speed of parameter update. Too high a learning rate may cause model oscillation, while too low a learning rate will prolong the training time. : represents the sensitivity of the elastic weight consolidation loss function to the model parameters, guiding the direction of parameter update; the sine term : Increase the randomness of parameter updates to help escape from local optimal solutions; logarithmic terms : Emphasizes the degree to which the parameter deviates from its optimal value, and gives greater adjustment strength to the parameters far from the optimal value; the hyperbolic tangent term : Provides a smooth parameter update method to make the update process more gradual and controllable; Fisher information matrix term :Take the importance of parameters into consideration and ensure the stability of important parameters; Gaussian term : By introducing Gaussian distribution, parameter updates are more in line with the laws of nature, which helps the model fit the data better; in, :Model parameters are obtained; : Calculated from the best parameter value saved after the previous task training; and : Task loss and elastic weight consolidation loss calculated during training based on the difference between the model prediction and the true label; partial derivatives and : Calculated by the back propagation algorithm, used to guide the update direction and speed of parameters; other parameters :It needs to be fine-tuned according to the requirements of the specific task or optimized through automated methods. Usually, different parameter combinations are systematically tried within a predefined parameter range through a grid search method to select the parameter combination with the best performance.

[0028] Suppose there is a power company that needs to optimize its power usage forecast model to reflect the latest business conditions in real time; Assuming real power usage and predicting electricity usage 5 samples of Sample 1 is ; Task loss function ; Assume that the weight of the sine term ; Frequency of the sine function ; Phase shift of the sine function ; Fisher information matrix ; Regularization parameter ; Weight of nonlinear term ; Parameters of nonlinear transformation ; Model parameters ; The best values ​​of the model parameters in the previous task ; Weight consolidation loss function ; Assuming the learning rate ; Weight of the sine term ; Frequency of the sine function ; Phase shift of the sine function ; Weight of logarithmic terms ; Weight of the hyperbolic tangent term ; The slope of the hyperbolic tangent function ; Weights of Fisher information matrix entries ; The weight of the Gaussian term ; Standard deviation of the Gaussian function ; Nonlinear dynamic adjustment factor ; Through the above steps, power companies can introduce elastic weights to consolidate the loss function and Fisher information matrix, combined with nonlinear dynamic adjustment factors, update the data model in real time, ensure that the data structure reflects the latest business conditions in real time, and generate a dynamic data association model.

[0029] Optionally, based on the comprehensive data set, data preprocessing is performed to improve data quality, association rule learning technology is used to construct association relationships between data, implicit association patterns between data are discovered, and a data association model is generated, including: Based on the comprehensive data set, performing a data cleaning operation to remove invalid and erroneous data records to generate a clean data set; Based on the clean data set, formatting is performed to unify the data format, ensure the consistency and compatibility of the data, and generate a standardized data set; Based on the standardized data set, applying missing value processing technology, reasonably filling or deleting blank values ​​in the data set, and generating a complete data set; Based on the complete data set, apply statistical analysis methods to conduct preliminary exploratory analysis of the data, identify data distribution characteristics and abnormal points, and generate preliminary analysis results; Based on the preliminary analysis results, the association rule learning technology is used to conduct in-depth analysis of the data, explore the implicit association patterns between the data, and generate a data association model.

[0030] Suppose there is a national grid company that needs to improve the reliability and efficiency of grid operation through advanced data analysis technology; Through the deployment of intelligent sensors and data acquisition systems in substations, transmission lines and user terminals, data is collected in real time, and multi-source data is integrated into a comprehensive data set through a data integration platform to ensure the comprehensiveness and timeliness of the data, providing a basis for subsequent data processing and analysis; data cleaning is performed on the comprehensive data set to remove invalid and erroneous data records, such as eliminating abnormal data caused by sensor failure, to ensure the accuracy and reliability of the data, and reduce noise and errors; the clean data set is formatted to ensure that the format of all data is consistent, improve the processability and analyzability of the data, and ensure the accuracy of subsequent analysis; missing value processing technology is applied to the standardized data set to reasonably fill or delete blank values ​​in the data set to improve the quality and availability of the data; statistical analysis methods are applied to the complete data set to conduct preliminary exploratory analysis, identify data distribution characteristics and abnormal points, understand the basic characteristics and abnormal conditions of the data, and provide a basis for subsequent in-depth analysis; association rule learning technology is used to conduct in-depth analysis of the preliminary analysis results, explore implicit association patterns between data, discover potential associations between data, provide deeper business insights, and support power grid optimization and fault prediction; Through the above steps, State Grid Corporation of China can effectively improve the reliability and efficiency of grid operation, timely discover and solve potential problems, optimize resource allocation, and improve service quality.

[0031] Optionally, based on the data association model, an incremental learning algorithm is used to dynamically process newly added data streams, update the data model in real time, ensure that the data structure reflects the latest business status in real time, and generate a dynamic data association model, including: Based on the data association model, newly added data streams are monitored in real time to ensure timely capture of new business data and generate real-time data streams; Based on the real-time data stream, an incremental learning algorithm is used to perform analysis and processing, and model parameters are dynamically adjusted to quickly adapt to data changes, ensure the timeliness and accuracy of the model, and generate an adaptive data association model; Based on the adaptive data association model, the model performance is continuously monitored, the new data processing capability is improved through performance optimization technology, and an efficient data association model is generated; Based on the efficient data association model, the data processing process is iterated to optimize the data processing efficiency and accuracy, and an optimized data processing process is generated; Based on the optimized data processing flow, the data association model is dynamically adjusted to ensure that the data structure reflects the latest business status in real time and generate a dynamic data association model.

[0032] Suppose there is a national grid company that needs to optimize grid management and improve service quality through advanced information processing technology. The specific implementation steps are as follows: Through the deployment of intelligent terminals and information collection systems in distribution centers, user homes and industrial facilities, information is collected in real time and summarized into a comprehensive information set; the comprehensive information set is cleaned to remove invalid and erroneous information records; the clean information set is formatted to ensure that the format of all information is consistent; the missing value processing technology is applied to the standardized information set to reasonably fill or delete the blank values ​​in the information set; the statistical analysis method is applied to the complete information set to conduct preliminary exploratory analysis to identify information distribution characteristics and anomalies; the association rule learning technology is used to conduct in-depth analysis of the preliminary analysis results to explore the implicit association patterns between information; based on the association rule model, the newly added information flow is monitored in real time to ensure that new business business information, generating a real-time information flow; based on the real-time information flow, using incremental learning algorithms to analyze and process, dynamically adjusting model parameters to quickly adapt to information changes, ensuring the timeliness and accuracy of the model, and generating an adaptive information association model; based on the adaptive information association model, continuously monitoring model performance, improving new information processing capabilities through performance optimization technology, and generating an efficient information association model; based on the efficient information association model, iterating the information processing process, optimizing information processing efficiency and accuracy, and generating an optimized information processing process; based on the optimized information processing process, dynamically adjusting the information association model to ensure that the information structure reflects the latest business conditions in real time, and generating a dynamic information association model; Through the above steps, State Grid Corporation of China can effectively optimize grid management and improve service quality, timely discover and solve potential problems, and improve user satisfaction and operational efficiency.

[0033] 104. Based on the optimized data structure, use the multimodal analysis model to conduct in-depth analysis, mine potential valuable information, and use the online transfer learning algorithm to dynamically adjust the model parameters to generate high-quality intermediate analysis results; Generate high-quality intermediate analysis results and provide in-depth business insights; use multimodal analysis models to comprehensively process various types of data features and generate multimodal analysis architectures; conduct in-depth analysis of optimized data structures, mine potential valuable information, and generate initial analysis data; use online transfer learning algorithms in combination with new data streams to dynamically adjust model parameters and generate adaptive model parameter configurations; continuously monitor and evaluate model performance, and use advanced data processing methods to optimize initial analysis data; ensure the adaptability and predictive capabilities of the model, generate high-quality intermediate analysis results, and provide a reliable basis for decision-making.

[0034] Optionally, based on the optimized data structure, a multimodal analysis model is used to conduct in-depth analysis, mine potential valuable information, and an online transfer learning algorithm is used to dynamically adjust model parameters to generate high-quality intermediate analysis results, including: Based on the optimized data structure, a motion multimodal analysis model is used to comprehensively process multiple types of data features to generate a multimodal analysis architecture; Based on the multimodal analysis framework, the optimized data structure is deeply analyzed to mine potential valuable information and generate initial analysis data; Based on the initial analysis data, using an online transfer learning algorithm, dynamically adjusting model parameters in combination with new data streams to ensure the adaptability and predictive ability of the model, and generating an adaptive model parameter configuration; Based on the adaptive model parameter configuration, the model performance is continuously monitored and evaluated, and advanced data processing methods are used to optimize the initial analysis data to generate high-quality intermediate analysis results.

[0035] Suppose there is an electric power company that needs to improve its data processing capabilities and business insights through advanced information processing methods; Collect and integrate various data sources to ensure data integrity and consistency; use data cleaning technology to remove invalid and erroneous data records; format data to ensure that all data formats are consistent, such as unifying all timestamps into a standard format; apply multimodal analysis models to comprehensively process various types of data features and generate a multimodal analysis architecture; apply the multimodal analysis architecture to actual business scenarios; based on the multimodal analysis architecture, conduct preliminary exploratory analysis on the optimized data structure to identify data distribution characteristics and anomalies; apply statistical analysis methods to further explore potential patterns and associations in the data; use association rule learning technology to discover implicit association patterns between data, such as identifying the relationship between peak power usage hours and specific environmental conditions; generate initial analysis data and Initial analysis data is stored in the database for subsequent use; initial model parameters are set based on the initial analysis data; new data streams are monitored in real time to ensure timely capture of new business data; online transfer learning algorithms are used to dynamically adjust model parameters in combination with new data streams to ensure the adaptability and predictive ability of the model; the prediction accuracy of the model is regularly evaluated to ensure the adaptability and predictive ability of the model; adaptive model parameter configuration is generated, for example, when new electricity usage data arrives, model parameters are dynamically adjusted to adapt to seasonal changes; the performance of the model is continuously monitored and the prediction error of the model is recorded; advanced data processing methods are used to optimize the initial analysis data; the prediction accuracy of the model is regularly evaluated, the analysis data is optimized, and high-quality analysis intermediate results are generated; high-quality analysis intermediate results are applied to actual business decisions; Through the above steps, power companies can effectively improve their data processing capabilities and business insights, and maintain continuous optimization and development of their businesses.

[0036] This application takes into account that the formula is used to calculate the update amount of the model parameters to generate an adaptive model parameter configuration; in the field of energy and power, data processing and model optimization are the key to ensuring stable operation and efficient management of the system, and introduces online transfer learning algorithms and multimodal feature enhancement technologies, combined with a variety of nonlinear mapping and regularization technologies, to achieve real-time data processing and model optimization; Optionally, based on the initial analysis data, an online transfer learning algorithm is used to dynamically adjust the model parameters in combination with the new data stream to ensure the adaptability and predictive ability of the model, and to generate an adaptive model parameter configuration, including: Perform data preprocessing on multi-source data streams, extract multimodal features, combine information from different modalities, and enhance the expressiveness of features; A multi-layer perceptron is used to perform nonlinear feature transformation to convert the multimodal features into a nonlinear feature representation of a higher dimension; The multimodal feature enhancement vector is calculated using the following formula: ; in, For the Multimodal feature enhancement vector of data points; For the The data point modal eigenvectors; For the The weight of each modal feature; is the weight of the sine term, which is used to adjust the influence of the sine term; is the frequency of the sine function; is the phase shift of the sine function; is the weight of the logarithmic term, which is used to adjust the impact of the logarithmic term; is the weight of the hyperbolic tangent term, which is used to adjust the influence of the hyperbolic tangent term; is the slope of the hyperbolic tangent function; is the index of the modal feature, from 1 to ; is the total number of modal features; For the The feature vector of the data point; The feature fusion and dynamic adjustment vector are calculated using the following formula: ; in, For the Feature fusion and dynamic adjustment vector of data points; For the The data point feature enhancement vector; For the The weight of the feature enhancement vector; is the weight of the Gaussian term, which is used to adjust the influence of the Gaussian term; Enhance the vector for the initial features; is the standard deviation of the Gaussian function; is the weight of the sine term, which is used to adjust the influence of the sine term; is the weight of the logarithmic term, which is used to adjust the impact of the logarithmic term; is the index of the feature enhancement vector, from 1 to ; is the total number of feature enhancement vectors; is the phase shift of the sine function; Based on the multimodal feature enhancement vector and the feature fusion and dynamic adjustment vector, combined with a variety of nonlinear mapping and regularization techniques, the update amount of the model parameters is calculated by the following formula: ; in, For the The update amount of each model parameter; is the learning rate, which controls the step size of parameter update; is the loss function for online transfer learning; For the model parameters; is the weight of the sine term, which is used to adjust the influence of the sine term; is the frequency of the sine function; is the phase shift of the sine function; is the weight of the logarithmic term, which is used to adjust the impact of the logarithmic term; is the weight of the hyperbolic tangent term, which is used to adjust the influence of the hyperbolic tangent term; is the slope of the hyperbolic tangent function; is the weight of feature fusion and dynamic adjustment items; For the Feature fusion and dynamic adjustment vector of data points; For the The data point eigenvalues; is the weight of the Gaussian term, which is used to adjust the influence of the Gaussian term; is the standard deviation of the Gaussian function; is the weight of the sine term combination, which is used to adjust the influence of the sine term combination; is the weight of the logarithmic term, which is used to adjust the impact of the logarithmic term; is the total number of model parameters; is the index of the data point, from 1 to ; is the total number of data points; Indicates Initial values ​​of model parameters or best estimates at a particular reference point; For the model parameters; Based on the update amount of the model parameters , perform multiple parameter smoothing processes and evaluate the validation set to ensure the adaptability and predictive ability of the model parameters and generate an adaptive model parameter configuration.

[0037] This method aims to ensure the adaptability and predictive ability of the model by comprehensively processing multi-source data streams, extracting multimodal features, and dynamically adjusting model parameters in combination with online transfer learning algorithms; in this way, the expressive power of features can be enhanced, the performance of the model in new data streams can be improved, and adaptive model parameter configurations can be generated.

[0038] In multimodal feature enhancement vector In the linear combination : By linearly combining the features of different modes, the contribution of each mode is comprehensively considered to form a comprehensive feature representation; the sine term : Increase the nonlinear component of the feature and improve the generalization ability of the model; logarithmic term : Emphasizes the absolute size of the eigenvalues ​​and gives greater weight to larger eigenvalues; hyperbolic tangent term : Provide smooth nonlinear transformation to make feature representation more flexible; In feature fusion and dynamic adjustment vector In the linear combination : By linearly combining different feature enhancement vectors, the contribution of each feature is comprehensively considered to form a comprehensive feature fusion representation; Gaussian term : Through Gaussian distribution, feature fusion is more in line with the laws of nature and the robustness of the model is enhanced; sine term : Increase the randomness of feature fusion to help escape the local optimal solution; logarithmic term : Emphasize the absolute size of the eigenvalues ​​after feature fusion, and give greater weight to larger eigenvalues; The update amount of the model parameters In the learning rate : Control the speed of parameter update. Too high a learning rate may cause model oscillation, while too low a learning rate will prolong the training time. : represents the sensitivity of the online transfer learning loss function to the model parameters and guides the direction of parameter update; the sine term : Increase the randomness of parameter updates to help escape from local optimal solutions; logarithmic terms : Emphasizes the degree to which the parameter deviates from its initial value, giving greater adjustment strength to the parameter far from the initial value; hyperbolic tangent term : Provides a smooth parameter update method to make the update process more gradual and controllable; feature fusion and dynamic adjustment items : Taking feature fusion and dynamic adjustment vector into consideration, enhancing the adaptability of the model; Gaussian term : Through Gaussian distribution, the parameter update is more in line with the laws of nature and the robustness of the model is enhanced; sine term combination : Increase the complexity of parameter update and improve the flexibility of the model; logarithmic terms : Emphasize feature fusion and dynamically adjust the absolute size of the vector, giving greater weight to larger values; in, :The kth modal eigenvector of the ith data point is obtained, : The multimodal feature enhancement vector is calculated; : Feature enhancement vector is calculated; : Feature fusion and dynamic adjustment vector calculation; : The j-th feature value of the i-th data point is obtained; : Model parameter generation; : Initial model parameter generation; :The partial derivatives of the online transfer learning loss function with respect to the model parameters are calculated; other parameters : Usually it is necessary to manually set the parameters according to the specific application scenario or automatically tune them through hyperparameter search technology. The selection of these parameters often depends on experimental results, and the most suitable values ​​are found through trial and error to achieve the best model performance.

[0039] Suppose a power company needs to adjust its power usage forecast model to respond to the latest business needs in real time; Assume that the weight of the sine term is ; Frequency of the sine function ; Phase of the sine function ; Weight of logarithmic terms ; Weight of the hyperbolic tangent term ; The slope of the hyperbolic tangent function ; Weight of modal features Assumptions ; Multimodal feature enhancement vector ; Assume the weight of the feature enhancement vector ; The weight of the Gaussian term ; Initial feature enhancement vector ; Standard deviation of the Gaussian function ; Weight of the sine term ; Weight of logarithmic terms Assumptions ; Feature fusion and dynamic adjustment vector ; Assuming the learning rate ; Online transfer learning loss function ; Weight of the sine term ; Frequency of the sine function ; Phase shift of the sine function ; Weight of logarithmic terms ; Weight of the hyperbolic tangent term ; The slope of the hyperbolic tangent function ; Feature fusion and dynamic adjustment item weights ; The weight of the Gaussian term ; Standard deviation of the Gaussian function ; Weights of the sine term combinations ; Weight of logarithmic terms ; Model parameters ; Initial model parameters ; Total number of data points ; The update amount of model parameters ; Through the above steps, power companies can extract multimodal features, apply online transfer learning algorithms, update and optimize their prediction models in real time, ensure the high accuracy and reliability of the models in new data environments, and generate adaptive model parameter configurations.

[0040] Optionally, based on the initial analysis data, using an online transfer learning algorithm, dynamically adjusting model parameters in combination with new data streams to ensure the adaptability and predictive ability of the model, and generating an adaptive model parameter configuration, includes: Based on the initial analysis data, using an online transfer learning algorithm, preliminarily adjusting the model parameters to generate preliminarily adjusted model parameters; Based on the model parameters initially adjusted, combined with feature changes in the new data stream, key parameters in the model are dynamically identified and adjusted to generate dynamically optimized model parameters; Based on the dynamically optimized model parameters, the model's new data processing performance is continuously monitored, and high prediction capability and stability are ensured by automatically optimizing the model parameters to generate a stable model parameter configuration; Based on the stable model parameter configuration, the multi-scenario adaptability of the model is tested, and a comprehensive evaluation is performed in combination with prediction accuracy and computational efficiency to ensure high adaptability and generate an adaptive model parameter configuration.

[0041] Suppose there is an electric power company that needs to optimize its business processes through intelligent data processing methods; Based on the preliminary analysis data, the online transfer learning algorithm is used to make preliminary adjustments to the model parameters and generate preliminary adjusted model parameters. Specifically, it includes extracting key features from the preliminary analysis data, setting the initial model parameters, making preliminary adjustments using the online transfer learning algorithm, and verifying the adjustment effect; based on the preliminary adjusted model parameters, combined with the feature changes in the new data stream, dynamically identifying and adjusting the key parameters in the model, and generating dynamically optimized model parameters. Specifically, it includes real-time monitoring of new data streams, dynamic identification of feature changes, dynamic adjustment of key parameters, and real-time updating of model parameters; based on the dynamically optimized model parameters, continuously monitoring the performance of the model when processing new data, automatically optimizing the model parameters, ensuring high prediction ability and stability, and generating stable model parameter configuration. Specifically, it includes continuously monitoring model performance, recording prediction errors, automatically optimizing model parameters, generating stable model parameter configuration, and regularly evaluating model performance; based on the stable model parameter configuration, testing the adaptability of the model in different scenarios, combining prediction accuracy and computational efficiency for comprehensive evaluation, ensuring that the model has high adaptability, and generating the final adaptive model parameter configuration. Specifically, it includes designing multiple test scenarios, evaluating the prediction accuracy and computational efficiency of the model, conducting a comprehensive evaluation based on all test results, generating the final adaptive model parameter configuration, and conducting actual verification tests.

[0042] Through the above steps, power companies can obtain adaptive model parameter configuration, implement intelligent data processing methods, optimize business processes, and provide technical support for long-term development.

[0043] 105. Based on the high-quality intermediate analysis results, a multi-dimensional visualization interface is constructed to support user interactive exploration, provide predictive analysis, and generate an intelligent decision support system.

[0044] Generate an intelligent decision support system to improve user experience and decision-making efficiency; design and develop visualization components to display key indicators and analyze trends, and generate a multi-dimensional visualization interface; integrate data screening and multi-perspective switching functions to support users in interactive exploration and generate an interactive exploration environment; provide predictive analysis to predict future trends and potential risks, and generate predictive analysis reports; integrate intelligent recommendation functions to ensure personalized decision suggestions and action plans, and generate an intelligent decision support system; provide intuitive and interactive data display and analysis tools to support users in making scientific decisions and improve the scientific nature and effectiveness of decisions.

[0045] Optionally, based on the high-quality intermediate analysis results, a multi-dimensional visualization interface is constructed to support user interactive exploration, provide predictive analysis, and generate an intelligent decision support system, including: Based on the high-quality intermediate analysis results, design and develop visualization components to display key indicators and analyze trends, and generate a multi-dimensional visualization interface; Based on the multi-dimensional visualization interface, data screening and multi-perspective switching functions are integrated to support users to conduct interactive exploration and generate an interactive exploration environment; Based on the interactive exploration environment, provide predictive analysis, predict future trends and potential risks, and generate predictive analysis reports; Based on the predictive analysis report, an intelligent recommendation function is integrated to ensure personalization of decision suggestions and action plans, and to generate an intelligent decision support system.

[0046] Suppose an electric power company needs to build an intelligent decision support system to reflect the latest business conditions in real time and provide scientific decision support; The original data is obtained from various power facilities, and after cleaning, conversion and standardization, high-quality intermediate analysis results are formed to lay the foundation for subsequent analysis; based on the above intermediate analysis results, a series of customized visualization components are designed and implemented to display key performance indicators from different angles, supporting management to quickly grasp the overall status and changing trends of the power system; advanced filtering and multi-view switching functions are added to the visualization interface, so that users can flexibly view according to actual needs; through interactive methods such as clicking and dragging chart elements, users can deeply explore the details behind the data and obtain more accurate insights; using advanced algorithms and technologies, the collected data is deeply mined to accurately predict future power demand and equipment performance, and to discover potential safety hazards or signs of failure in advance; the system automatically generates predictive analysis reports, clearly presents the expected development direction and possible problems, and proposes corresponding preventive measures; for different application scenarios and user groups, an intelligent recommendation engine is developed to provide tailored suggestions and services based on user-specific needs and past decision records, effectively improving work efficiency and decision-making quality; Through the above steps, the power company has established an intelligent decision support system integrating multi-dimensional functions, enhancing its competitiveness in power system management and optimization.

[0047] Figure 2 A structural diagram of a big data analysis and processing system based on artificial intelligence is provided for the embodiment of the present application, such as Figure 2 As shown, the device comprises: A collection module 21 is used to collect user's business demand information through multiple channels; An analysis module 22 is used to comprehensively analyze the business demand information based on artificial intelligence technology and in combination with external influencing factors to generate data analysis targets; The processing module 23 is used to integrate the original data from multiple sources in real time based on the data analysis target, construct the association relationship between the data using the association rule learning technology, and process the newly added data stream using the incremental learning algorithm to generate an optimized data structure; An adjustment module 24 is used to use a multimodal analysis model to conduct in-depth analysis based on the optimized data structure, mine potential valuable information, and dynamically adjust model parameters using an online transfer learning algorithm to generate high-quality intermediate analysis results; The prediction module 25 is used to build a multi-dimensional visualization interface based on the high-quality intermediate analysis results, support user interactive exploration, provide predictive analysis, and generate an intelligent decision support system.

[0048] Figure 2 The big data analysis and processing system based on artificial intelligence can perform Figure 1 The implementation principle and technical effect of the big data analysis and processing method based on artificial intelligence described in the embodiment shown will not be repeated. The specific way in which each module and unit performs operations in the big data analysis and processing system based on artificial intelligence in the above embodiment has been described in detail in the embodiment of the method, and will not be elaborated here.

[0049] In one possible design, Figure 2 The big data analysis and processing system based on artificial intelligence in the embodiment shown can be implemented as a computing device, such as Figure 3 As shown, the computing device may include a storage component 31 and a processing component 32; The storage component 31 stores one or more computer instructions, wherein the one or more computer instructions are called and executed by the processing component 32 .

[0050] The processing component 32 is used to: collect user business demand information through multiple channels; based on artificial intelligence technology, combined with external influencing factors, comprehensively analyze the business demand information and generate data analysis goals; based on the data analysis goals, real-time integrate multi-source original data, use association rule learning technology to build data association relationships, use incremental learning algorithms to process newly added data streams, and generate optimized data structures; based on the optimized data structure, use multimodal analysis models to conduct in-depth analysis, mine potential value information, use online transfer learning algorithms to dynamically adjust model parameters, and generate high-quality analysis intermediate results; based on the high-quality analysis intermediate results, build a multi-dimensional visualization interface to support user interactive exploration, provide predictive analysis, and generate an intelligent decision support system.

[0051] The processing component 32 may include one or more processors to execute computer instructions to complete all or part of the steps in the above method. Of course, the processing component may also be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors or other electronic components to perform the above method.

[0052] The storage component 31 is configured to store various types of data to support operations at the terminal. The storage component can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0053] Of course, the computing device may also include other components, such as input / output interfaces, display components, communication components, etc.

[0054] The input / output interface provides an interface between the processing component and the peripheral interface module, which may be an output device, an input device, etc.

[0055] The communication component is configured to facilitate, among other things, wired or wireless communications between the computing device and other devices.

[0056] Among them, the computing device can be a physical device or an elastic computing host provided by a cloud computing platform, etc. In this case, the computing device can refer to a cloud server, and the above-mentioned processing components, storage components, etc. can be basic server resources rented or purchased from the cloud computing platform.

[0057] The present application also provides a computer storage medium storing a computer program, wherein the computer program can achieve the above-mentioned Figure 1 The illustrated embodiment is a method for analyzing and processing big data based on artificial intelligence.

[0058] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0059] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0060] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0061] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A big data analysis and processing method based on artificial intelligence, characterized in that: include: Collect users' business demand information through multiple channels; Based on artificial intelligence technology and combined with external influencing factors, the business demand information is comprehensively analyzed to generate data analysis goals; Based on the data analysis objectives, the original data from multiple sources are integrated in real time, the association relationship between data is constructed using association rule learning technology, and the newly added data stream is processed using incremental learning algorithms to generate optimized data structures; Based on the optimized data structure, a multimodal analysis model is used to conduct in-depth analysis, mine potential valuable information, and an online transfer learning algorithm is used to dynamically adjust model parameters to generate high-quality intermediate analysis results; Based on the high-quality intermediate analysis results, a multi-dimensional visualization interface is constructed to support user interactive exploration, provide predictive analysis, and generate an intelligent decision support system.

2. The method according to claim 1, characterized in that Based on the data analysis target, the multi-source raw data is integrated in real time, the association relationship between data is constructed using association rule learning technology, and the newly added data stream is processed using an incremental learning algorithm to generate an optimized data structure, including: Based on the data analysis objectives, real-time monitoring and collection of multiple data sources are performed, and multi-source raw data are integrated to generate a comprehensive data set; Based on the comprehensive data set, data preprocessing is performed to improve data quality, association rule learning technology is used to construct association relationships between data, implicit association patterns between data are discovered, and a data association model is generated; Based on the data association model, an incremental learning algorithm is used to dynamically process newly added data streams, update the data model in real time, ensure that the data structure reflects the latest business status in real time, and generate a dynamic data association model; Based on the dynamic data association model, the data structure is adjusted through an optimization algorithm to eliminate data redundancy, ensure the efficiency and accuracy of data processing, and generate an optimized data structure; Wherein, based on the data association model, an incremental learning algorithm is used to dynamically process newly added data streams, update the data model in real time, ensure that the data structure reflects the latest business status in real time, and generate a dynamic data association model, including: Define the task loss function , providing timely feedback during model training, accelerating model convergence, and providing support for the calculation of elastic weight consolidation loss function; Setting the Fisher information matrix , dynamically evaluate the importance of parameters to ensure the accuracy and stability of model parameter updates; The elastic weight consolidation loss function is calculated using the following formula: ； in, Consolidate loss functions for elastic weights; is the task loss function, which is the mean square error or cross entropy loss; For the model parameters; For the previous task The best values ​​of the model parameters; is the diagonal element of the Fisher information matrix, which is used to measure the importance of the parameters; is a regularization parameter used to balance the weight of task loss and parameter stability; is the weight of the nonlinear term, which is used to adjust the impact of the nonlinear term; is the parameter of nonlinear transformation, which is used to adjust the shape of nonlinear transformation; is the index of the model parameter, from 1 to ; is the total number of model parameters; The Fisher information matrix is ​​calculated using the following formula: ； in, is the diagonal element of the Fisher information matrix, which is used to measure the importance of the parameters; For input data; is the data distribution; is the task loss function; For the model parameters; is the weight of the sine term, which is used to adjust the influence of the sine term; is the frequency of the sine function; is the phase shift of the sine function; is the task loss function For Model parameters The partial derivative of reflects the parameter The degree of influence on the loss function; For all input data According to its distribution Calculated expectations; Based on the elastic weight consolidation loss function and the Fisher information matrix, the Gaussian term is introduced to construct a multi-factor comprehensive nonlinear relationship, and the nonlinear dynamic adjustment factor is calculated by the following formula: ； in, For the Nonlinear dynamic adjustment factors for model parameters; is the learning rate, which controls the step size of parameter update; Consolidate loss functions for elastic weights; For the model parameters; For the previous task The best values ​​of the model parameters; is the weight of the sine term, which is used to adjust the influence of the sine term; is the frequency of the sine function; is the phase shift of the sine function; is the weight of the logarithmic term, which is used to adjust the impact of the logarithmic term; is the weight of the hyperbolic tangent term, which is used to adjust the influence of the hyperbolic tangent term; is the slope of the hyperbolic tangent function; is the weight of the Fisher information matrix item; is the Fisher information matrix, The diagonal elements represent; is the weight of the Gaussian term, which is used to adjust the influence of the Gaussian term; is the standard deviation of the Gaussian function; Consolidating loss functions for elastic weights For Model parameters The partial derivative of reflects the parameter The degree of influence on the elastic weight consolidation loss function; Based on the nonlinear dynamic adjustment factor , continuously updating the model parameters , dynamically adjust in combination with new data streams to ensure that the data structure reflects the latest business conditions in real time and generate a dynamic data association model.

3. The method according to claim 2, characterized in that Based on the comprehensive data set, data preprocessing is performed to improve data quality, association rule learning technology is used to construct association relationships between data, implicit association patterns between data are discovered, and a data association model is generated, including: Based on the comprehensive data set, performing a data cleaning operation to remove invalid and erroneous data records to generate a clean data set; Based on the clean data set, formatting is performed to unify the data format, ensure the consistency and compatibility of the data, and generate a standardized data set; Based on the standardized data set, applying missing value processing technology, reasonably filling or deleting blank values ​​in the data set, and generating a complete data set; Based on the complete data set, apply statistical analysis methods to conduct preliminary exploratory analysis of the data, identify data distribution characteristics and abnormal points, and generate preliminary analysis results; Based on the preliminary analysis results, the association rule learning technology is used to conduct in-depth analysis of the data, explore the implicit association patterns between the data, and generate a data association model.

4. The method according to claim 2, characterized in that: Based on the data association model, an incremental learning algorithm is used to dynamically process the newly added data stream, update the data model in real time, ensure that the data structure reflects the latest business status in real time, and generate a dynamic data association model, including: Based on the data association model, newly added data streams are monitored in real time to ensure timely capture of new business data and generate real-time data streams; Based on the real-time data stream, an incremental learning algorithm is used for analysis and processing, and model parameters are dynamically adjusted to quickly adapt to data changes, ensure the timeliness and accuracy of the model, and generate an adaptive data association model; Based on the adaptive data association model, the model performance is continuously monitored, the new data processing capability is improved through performance optimization technology, and an efficient data association model is generated; Based on the efficient data association model, the data processing process is iterated to optimize the data processing efficiency and accuracy, and an optimized data processing process is generated; Based on the optimized data processing flow, the data association model is dynamically adjusted to ensure that the data structure reflects the latest business status in real time and generate a dynamic data association model.

5. The method according to claim 1, characterized in that: Based on the optimized data structure, the multimodal analysis model is used for in-depth analysis to mine potential valuable information, and the model parameters are dynamically adjusted using the online transfer learning algorithm to generate high-quality intermediate analysis results, including: Based on the optimized data structure, a motion multimodal analysis model is used to comprehensively process multiple types of data features to generate a multimodal analysis architecture; Based on the multimodal analysis framework, the optimized data structure is deeply analyzed to mine potential valuable information and generate initial analysis data; Based on the initial analysis data, using an online transfer learning algorithm, dynamically adjusting model parameters in combination with new data streams to ensure the adaptability and predictive ability of the model, and generating an adaptive model parameter configuration; Based on the adaptive model parameter configuration, the model performance is continuously monitored and evaluated, and the initial analysis data is optimized using advanced data processing methods to generate high-quality intermediate analysis results; Based on the initial analysis data, an online transfer learning algorithm is used to dynamically adjust the model parameters in combination with the new data stream to ensure the adaptability and predictive ability of the model and generate an adaptive model parameter configuration, including: Perform data preprocessing on multi-source data streams, extract multimodal features, combine information from different modalities, and enhance the expressiveness of features; A multi-layer perceptron is used to perform nonlinear feature transformation to convert the multimodal features into a nonlinear feature representation of a higher dimension; The multimodal feature enhancement vector is calculated using the following formula: ； in, For the Multimodal feature enhancement vector of data points; For the The data point modal eigenvectors; For the The weight of each modal feature; is the weight of the sine term, which is used to adjust the influence of the sine term; is the frequency of the sine function; is the phase shift of the sine function; is the weight of the logarithmic term, which is used to adjust the impact of the logarithmic term; is the weight of the hyperbolic tangent term, which is used to adjust the influence of the hyperbolic tangent term; is the slope of the hyperbolic tangent function; is the index of the modal feature, from 1 to ; is the total number of modal features; For the The feature vector of the data point; The feature fusion and dynamic adjustment vector are calculated using the following formula: ； in, For the Feature fusion and dynamic adjustment vector of data points; For the The data point feature enhancement vector; For the The weight of the feature enhancement vector; is the weight of the Gaussian term, which is used to adjust the influence of the Gaussian term; Enhance the vector for the initial features; is the standard deviation of the Gaussian function; is the weight of the sine term, which is used to adjust the influence of the sine term; is the weight of the logarithmic term, which is used to adjust the impact of the logarithmic term; is the index of the feature enhancement vector, from 1 to ; is the total number of feature enhancement vectors; is the phase shift of the sine function; Based on the multimodal feature enhancement vector and the feature fusion and dynamic adjustment vector, combined with a variety of nonlinear mapping and regularization techniques, the update amount of the model parameters is calculated by the following formula: ； in, For the The update amount of each model parameter; is the learning rate, which controls the step size of parameter update; is the loss function for online transfer learning; For the model parameters; is the weight of the sine term, which is used to adjust the influence of the sine term; is the frequency of the sine function; is the phase shift of the sine function; is the weight of the logarithmic term, which is used to adjust the impact of the logarithmic term; is the weight of the hyperbolic tangent term, which is used to adjust the influence of the hyperbolic tangent term; is the slope of the hyperbolic tangent function; is the weight of feature fusion and dynamic adjustment items; For the Feature fusion and dynamic adjustment vector of data points; For the The data point eigenvalues; is the weight of the Gaussian term, which is used to adjust the influence of the Gaussian term; is the standard deviation of the Gaussian function; is the weight of the sine term combination, which is used to adjust the influence of the sine term combination; is the weight of the logarithmic term, which is used to adjust the impact of the logarithmic term; is the total number of model parameters; is the index of the data point, from 1 to ; is the total number of data points; Indicates Initial values ​​of model parameters or best estimates at a particular reference point; For the model parameters; Based on the update amount of the model parameters , perform multiple parameter smoothing processes and evaluate the validation set to ensure the adaptability and predictive ability of the model parameters and generate an adaptive model parameter configuration.

6. The method according to claim 5, characterized in that Based on the initial analysis data, the online transfer learning algorithm is used to dynamically adjust the model parameters in combination with the new data stream to ensure the adaptability and predictive ability of the model and generate an adaptive model parameter configuration, including: Based on the initial analysis data, using an online transfer learning algorithm, preliminarily adjusting the model parameters to generate preliminarily adjusted model parameters; Based on the model parameters initially adjusted, combined with feature changes in the new data stream, key parameters in the model are dynamically identified and adjusted to generate dynamically optimized model parameters; Based on the dynamically optimized model parameters, the model's new data processing performance is continuously monitored, and high prediction capability and stability are ensured by automatically optimizing the model parameters to generate a stable model parameter configuration; Based on the stable model parameter configuration, the multi-scenario adaptability of the model is tested, and a comprehensive evaluation is performed in combination with prediction accuracy and computational efficiency to ensure high adaptability and generate an adaptive model parameter configuration.

7. The method according to claim 1, characterized in that Based on the high-quality intermediate analysis results, a multi-dimensional visualization interface is constructed to support user interactive exploration, provide predictive analysis, and generate an intelligent decision support system, including: Based on the high-quality intermediate analysis results, design and develop visualization components to display key indicators and analyze trends, and generate a multi-dimensional visualization interface; Based on the multi-dimensional visualization interface, data screening and multi-perspective switching functions are integrated to support users to conduct interactive exploration and generate an interactive exploration environment; Based on the interactive exploration environment, provide predictive analysis, predict future trends and potential risks, and generate predictive analysis reports; Based on the predictive analysis report, an intelligent recommendation function is integrated to ensure personalization of decision suggestions and action plans, and to generate an intelligent decision support system.

8. A big data analysis and processing system based on artificial intelligence, characterized in that: include: Collection module, used to collect users' business demand information through multiple channels; An analysis module is used to comprehensively analyze the business demand information based on artificial intelligence technology and in combination with external influencing factors to generate data analysis targets; A processing module is used to integrate multi-source raw data in real time based on the data analysis target, use association rule learning technology to build association relationships between data, use incremental learning algorithms to process newly added data streams, and generate optimized data structures; An adjustment module is used to use a multimodal analysis model to conduct in-depth analysis based on the optimized data structure, mine potential valuable information, and dynamically adjust model parameters using an online transfer learning algorithm to generate high-quality intermediate analysis results; The prediction module is used to build a multi-dimensional visualization interface based on the high-quality intermediate analysis results, support user interactive exploration, provide predictive analysis, and generate an intelligent decision support system.

9. A computing device, characterized in that It comprises a processing component and a storage component; the storage component stores one or more computer instructions; the one or more computer instructions are used to be called and executed by the processing component to implement an artificial intelligence-based big data analysis and processing method as described in any one of claims 1 to 7.

10. A computer storage medium, characterized in that: A computer program is stored, and when the computer program is executed by a computer, an artificial intelligence-based big data analysis and processing method as described in any one of claims 1 to 7 is implemented.

Images