Enterprise big data analysis processing method and system based on cloud computing

Abstract

The invention provides an enterprise big data analysis processing method and system based on cloud computing, relates to the technical field of enterprise analysis, and is used for solving the problems of poor security, low flexibility, low automation degree and poor user experience in the prior art. The method comprises the following steps: receiving a customized data analysis request of an enterprise user; dynamically constructing a target virtual private cloud environment based on the basic demand information and the business logic rule, and loading cross-source data of different data sources and related to the basic demand information into the target virtual private cloud environment; carrying out data processing on the cross-source data in the target virtual private cloud environment by adopting a self-adaptive distributed computing framework; and converting the data processing result into corresponding text description, presenting the text description to the enterprise user through the cloud platform in combination with a graphical display mode, and optimizing the data processing flow according to feedback suggestion information of the enterprise user on the data analysis result. According to the invention, more flexible, efficient and intelligent data processing service can be provided.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of enterprise analysis technology, and more particularly to a cloud computing-based enterprise big data analysis and processing method and system. Background Art

[0002] With the rapid development of information technology, enterprises have accumulated a large amount of data resources. These data cover many aspects of enterprise operations, dynamic markets, customer service, etc., and are of great value in improving the competitiveness and decision-making level of enterprises. However, traditional data analysis and processing methods are often limited by the limitations of local cloud computing resources and the singleness of data processing technology, and it is difficult to meet the high standards of modern enterprises in data processing speed, accuracy and security.

[0003] In recent years, cloud computing technology has provided enterprises with powerful data processing capabilities and flexible service models. Through the cloud computing platform, enterprises can obtain sufficient cloud computing resources according to actual needs, thereby realizing efficient processing of large-scale data. However, the existing cloud computing services still have deficiencies in security, flexibility, automation and user experience when analyzing and processing enterprise big data. In other words, since enterprise data usually contains sensitive information, ensuring data security and privacy in the cloud computing environment has become a key issue that needs to be solved urgently. The existing data processing process is often relatively fixed and cannot quickly respond to and adapt to the diverse and changing data analysis needs of enterprises. Many data processing links still require manual intervention, which not only reduces processing efficiency but also increases the risk of errors. The presentation form of data analysis results is too single, lacking in intuitiveness and interactivity, which is not conducive to enterprise users' in-depth understanding of data and making decision support based on it.

[0004] Therefore, there is an urgent need for a new cloud computing-based enterprise big data analysis and processing method that can provide more flexible, efficient, and intelligent data processing services while ensuring data security to meet the ever-changing data analysis needs of enterprises. Summary of the invention

[0005] The embodiments of the present application provide a cloud computing-based enterprise big data analysis and processing method and system to solve the problems of poor security, low flexibility, low degree of automation and poor user experience in the prior art.

[0006] In a first aspect, the present application provides a method for analyzing and processing enterprise big data based on cloud computing, including: Receiving a customized data analysis request submitted by an enterprise user through a cloud platform, wherein the customized data analysis request includes: basic demand information and advanced demand information, wherein the advanced demand information includes business logic rules specific to the enterprise user; Based on the basic demand information and the business logic rules, dynamically construct a target virtual private cloud environment; Identify and aggregate cross-source data that is scattered in different data sources and is related to the basic demand information to obtain a data set, perform data consistency checks on all cross-source data in the data set to ensure the accuracy and integrity of the cross-source data, and load the data set into the target virtual private cloud environment after all cross-source data pass the data consistency check; Using an adaptive distributed computing framework in the target virtual private cloud environment to process all cross-source data in the data set; After the data is processed, natural language generation technology is used to convert the data analysis results into corresponding text descriptions, and then combined with graphical display methods, it is presented to enterprise users through the cloud platform; The continuous learning mechanism is used to collect feedback and suggestion information from enterprise users on the data analysis results, and the data processing flow is optimized based on the feedback and suggestion information.

[0007] Optionally, dynamically constructing a target virtual private cloud environment based on the basic demand information and the business logic rule includes: Generate a security policy specified by the enterprise user according to the business logic rules and the enterprise user-specific and privacy protection level requirements in the high-level demand information; Dynamically create an initial virtual private cloud environment based on the security policy specified by the enterprise user and the basic demand information; A multi-tenant isolation strategy is introduced into the initial virtual private cloud environment to obtain a target virtual private cloud environment, and the multi-tenant isolation strategy enables the target virtual private cloud environment to achieve isolation between tenants and isolation within tenants.

[0008] Optionally, dynamically creating an initial virtual private cloud environment based on the security policy specified by the enterprise user and the basic requirement information includes: Convert the security policies and basic demand information specified by the enterprise user into structured configuration parameters, generate a personalized infrastructure as code template according to the configuration parameters, and configure cloud computing resources according to the personalized infrastructure as code template to generate a virtual private cloud environment to be optimized; Based on the access control list, combined with the role-based access control strategy and the attribute-based access control strategy, the resource access rights in the virtual private cloud environment to be optimized are configured and processed, and the validity and security of the resource access rights are verified through the simulation test tool to obtain a preliminary optimized virtual private cloud environment; Combined with future resource demands, an adaptive expansion algorithm is used to dynamically adjust the cloud computing resources of the initial optimized virtual private cloud environment to obtain a secondary optimized virtual private cloud environment.

[0009] Optionally, based on the access control list, in combination with the role-based access control policy and the attribute-based access control policy, the resource access rights in the virtual private cloud environment to be optimized are configured and processed, including: Based on the access control list and the role-based access control policy, the user roles are hierarchically divided to obtain the role permission set. Based on the attribute-based access control policy, the attribute permission configuration result is generated by combining the user attributes, cloud computing resource attributes and environmental conditions. The role permission set and the attribute permission configuration result are merged using a weighted average method to form a comprehensive access control strategy; According to the comprehensive access control strategy, the cloud computing resources are processed by using the access control list application method to obtain the access control list rules applied to the virtual private cloud environment to be optimized; Based on the access control list rules and in combination with dynamic permission adjustment suggestions, a time series analysis model is used to adjust the resource access rights in the virtual private cloud environment to be optimized; the dynamic permission adjustment suggestions are obtained by tracking enterprise user behaviors and resource access patterns.

[0010] Optionally, the adaptive expansion algorithm includes an expansion algorithm based on queue length, and the adaptive expansion algorithm is used to dynamically adjust the cloud computing resources of the preliminary optimized virtual private cloud environment in combination with future resource requirements to obtain a secondary optimized virtual private cloud environment, including: Based on the historical data analysis requests of enterprise users and the historical cloud computing resource allocation results corresponding to the historical data analysis requests, the autoregressive integral moving average model is trained, and the autoregressive integral moving average model is optimized by combining business logic rules and seasonal factors to obtain a prediction model for predicting future resource demand, and the prediction model is used to output future resource demand; According to the future resource demand, a threshold setting method is used to determine the scope of the automatic expansion trigger condition and define the instance quantity limit; Use automated tools to configure adaptive scaling groups based on the range of automatic scaling trigger conditions and instance quantity limits; An expansion algorithm based on queue length is used to select the actually required expansion group from the adaptive expansion group; The actually required extension groups are added to the initially optimized virtual private cloud environment to achieve dynamic adjustment of cloud computing resources and obtain a secondary optimized virtual private cloud environment.

[0011] Optionally, a multi-tenant isolation strategy is introduced into the initial virtual private cloud environment to obtain a target virtual private cloud environment, wherein the multi-tenant isolation strategy enables the target virtual private cloud environment to achieve inter-tenant isolation and intra-tenant isolation, including: Define tenant boundaries, where the tenant boundaries are used to reflect resource allocation and service scope of the corresponding tenant. One tenant includes multiple enterprise users. Evaluate the similarity of business processes of different tenants, and generate corresponding multi-tenant isolation strategies according to the similarity, wherein the multi-tenant isolation strategies include virtual network isolation strategies, cloud computing resource isolation strategies, and storage resource isolation strategies; A virtual network isolation strategy, a cloud computing resource isolation strategy and a storage resource isolation strategy are introduced into the initial virtual private cloud environment to obtain a target virtual private cloud environment. The multi-tenant isolation strategy enables the target virtual private cloud environment to achieve virtual network isolation, cloud computing resource isolation and storage resource isolation between tenants, as well as virtual network isolation, cloud computing resource isolation and storage resource isolation within a tenant.

[0012] Optionally, the adopting an adaptive distributed computing framework in the target virtual private cloud environment to process all cross-source data in the data set includes: Calculate the complexity of the data processing task, and if the complexity is greater than a preset complexity threshold, divide the data processing task into multiple subtasks in combination with business logic rules, and determine the priority of each subtask in combination with the association relationship between the subtasks; According to the task priority of each subtask, the execution order of each subtask is determined, and cloud computing resources are allocated according to the resource availability of computing nodes in the adaptive distributed computing framework, the data size of cross-source data required by the subtask and the execution order of each subtask, and the cloud computing resource allocation result is obtained; In the target virtual private cloud environment, combined with the cloud computing resource allocation results, an adaptive distributed computing framework is used to process all cross-source data in the data set, and the execution status of each subtask is monitored in real time during the data processing process. When the execution status of any subtask is detected to be abnormal, the cloud computing resource allocation results are adjusted to generate data analysis results within a preset time.

[0013] In a second aspect, the embodiment of the present application provides an enterprise big data analysis and processing system based on cloud computing, including: A receiving module, configured to receive a customized data analysis request submitted by an enterprise user through a cloud platform, wherein the customized data analysis request includes: basic demand information and advanced demand information, wherein the advanced demand information includes business logic rules specific to the enterprise user; A construction module, used to dynamically construct a target virtual private cloud environment based on the basic demand information and the business logic rules; An identification and verification module is used to identify and aggregate cross-source data that is scattered in different data sources and is related to the basic demand information to obtain a data set, and perform data consistency verification on all cross-source data in the data set to ensure the accuracy and integrity of the cross-source data. After all cross-source data pass the data consistency verification, the data set is loaded into the target virtual private cloud environment; A data processing module, used for processing all cross-source data in the data set by using an adaptive distributed computing framework in a target virtual private cloud environment; The conversion module is used to convert the data analysis results into corresponding text descriptions using natural language generation technology after the data processing is completed, and present them to enterprise users through the cloud platform in combination with graphical display methods; The collection and optimization module is used to collect feedback and suggestion information from enterprise users on the data analysis results through a continuous learning mechanism, and optimize the data processing process according to the feedback and suggestion information.

[0014] In a third aspect, an embodiment of the present application provides a computing device, comprising a processing component and a storage component; the storage component stores one or more computer instructions; the one or more computer instructions are used to be called and executed by the processing component to implement a cloud computing-based enterprise big data analysis and processing method as described in any one of the first aspects.

[0015] In a fourth aspect, an embodiment of the present application provides a computer storage medium storing a computer program. When the computer program is executed by a computer, it implements a cloud computing-based enterprise big data analysis and processing method as described in any one of the first aspects.

[0016] In an embodiment of the present application, a cloud computing-based enterprise big data analysis and processing method is provided, including: receiving a customized data analysis request submitted by an enterprise user through a cloud platform, the customized data analysis request including: basic demand information and advanced demand information, the advanced demand information including business logic rules specific to the enterprise user; dynamically building a target virtual private cloud environment based on the basic demand information and the business logic rules; identifying and aggregating cross-source data that is scattered in different data sources and is related to the basic demand information to obtain a data set, performing data consistency checks on all cross-source data in the data set to ensure the accuracy and completeness of the cross-source data, and after all cross-source data pass the data consistency check, loading the data set into the target virtual private cloud environment; using an adaptive distributed computing framework in the target virtual private cloud environment to process all cross-source data in the data set; after the data processing is completed, using natural language generation technology to convert the data analysis results into corresponding text descriptions, and presenting them to the enterprise user through the cloud platform in combination with a graphical display method; collecting feedback and suggestion information on the data analysis results from enterprise users through a continuous learning mechanism, and optimizing the data processing process based on the feedback and suggestion information.

[0017] The embodiment of the present application realizes the automation and intelligence of the entire process from data collection, processing to result presentation. Through multi-tenant isolation technology and continuous optimization mechanism, it ensures the security of data and the high quality of data analysis results. Through the adaptive distributed computing framework, it improves data processing efficiency, bringing better big data analysis and processing experience to enterprise users.

[0018] These and other aspects of the present application will become more clearly understood in the description of the following embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, a brief introduction will be given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0020] Figure 1 A flowchart of a cloud computing-based enterprise big data analysis and processing method provided in an embodiment of the present application; Figure 2 A schematic diagram of the structure of a cloud computing-based enterprise big data analysis and processing system provided in an embodiment of the present application; Figure 3 A schematic diagram of the structure of a computing device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0021] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application.

[0022] In some of the processes described in the specification and claims of this application and the above-mentioned figures, multiple operations that appear in a specific order are included, but it should be clearly understood that these operations may not be executed in the order in which they appear in this article or executed in parallel. The sequence numbers of the operations, such as 11, 12, etc., are only used to distinguish between different operations, and the sequence numbers themselves do not represent any execution order. In addition, these processes may include more or fewer operations, and these operations may be executed in sequence or in parallel. It should be noted that the descriptions of "first", "second", etc. in this article are used to distinguish different messages, devices, modules, etc., do not represent the order of precedence, and do not limit the "first" and "second" to different types.

[0023] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of this application.

[0024] Figure 1 A flowchart of a cloud computing-based enterprise big data analysis and processing method provided in an embodiment of the present application is shown in FIG. Figure 1 As shown, the method includes: S11. Receive customized data analysis requests submitted by enterprise users through the cloud platform. The customized data analysis requests include basic demand information and advanced demand information. The advanced demand information includes business logic rules specific to the enterprise users.

[0025] The enterprise user can be an internal employee of the enterprise or a customer of the enterprise. The present embodiment does not specifically limit the meaning of the enterprise user. The customized analysis request is used to indicate the customized data analysis requirements of the enterprise user. The customized data analysis request includes basic requirement information and advanced requirement information. The basic requirement information includes the source identification of the data to be analyzed, the purpose of the analysis, and the expected output format. The advanced requirement information includes the business logic rules and privacy protection level requirements specific to the enterprise user.

[0026] It should be understood that the source identifier of the data to be analyzed can be the location or identifier of the data source, such as a database uniform resource locator (URL), a file path, an application programming interface (API) endpoint, etc. The source identifier can also include the time range, version number or other specific filtering conditions of the data to be analyzed. The purpose of analysis refers to the goal that the enterprise user hopes to achieve through data analysis, such as market trend analysis, customer behavior research, product performance evaluation, etc. Optionally, the analysis goal can help determine the required data type and analysis method. The expected output format can refer to the presentation method of the data analysis results, such as reports, charts, model prediction results, etc. The expected output format can also refer to the specific format of the output, such as an Excel table, a portable document format (PDF) document, a hypertext markup language (HTML) page, or a specific data structure.

[0027] It should also be understood that the business logic rules specific to enterprise users are unique rules or standards that enterprise users can follow in business operations, which may affect the selection of data, processing methods, or interpretation of results. For example, in the financial industry, certain transactions may need to comply with specific compliance requirements; in the retail industry, the evaluation of the effectiveness of promotional activities may need to consider seasonal factors. The privacy protection level requirements refer to the specific requirements of enterprise users for data privacy and security, including but not limited to data encryption standards, access control measures, data desensitization methods, etc. These requirements help ensure compliance with relevant laws and regulations during data processing and protect sensitive information of individuals and enterprises from being leaked.

[0028] Optionally, in some actual application scenarios, after receiving a customized data analysis request submitted by an enterprise user through a cloud platform, the embodiment of the present application can parse the customized data analysis request to identify basic demand information and advanced demand information.

[0029] S12. Based on basic demand information and business logic rules, dynamically build the target virtual private cloud environment. This step can ensure that the data to be analyzed meets the user's specified security policy during the entire data processing process and use multi-tenant isolation technology to prevent data leakage.

[0030] S13. Identify and aggregate cross-source data that is scattered in different data sources and is related to basic demand information to obtain a data set. Perform data consistency checks on all cross-source data in the data set to ensure the accuracy and completeness of the cross-source data. After all cross-source data have passed the data consistency check, load the data set into the target virtual private cloud environment.

[0031] Specifically, the embodiment of the present application can use a machine learning algorithm to automatically identify the specific location of the data source based on the basic requirement information in the customized data analysis request (such as the location or identifier of the data source, time range, version number, etc.). According to the specific location of the data source, cross-source data is obtained from the data source, and after the cross-source data passes the data consistency check, it is loaded into the target virtual private cloud environment. In this step, the embodiment of the present application can extract the data to be analyzed from different data sources such as databases, file systems, API endpoints, etc.

[0032] S14. Using an adaptive distributed computing framework in the target virtual private cloud environment to process all cross-source data in the data set.

[0033] Among them, the adaptive distributed computing framework may include Apache Hadoop, Apache Spark, Apache Flink, etc. The adaptive distributed computing framework is a framework that supports adaptive cloud computing resource allocation and optimization, which can ensure the efficient execution of data processing tasks. In addition, the adaptive distributed computing framework can automatically calculate the cloud computing resource allocation results according to the complexity of the data processing task and the data scale, ensuring that the data processing task is completed in the shortest time while maintaining optimal energy efficiency. Specifically, the embodiment of the present application can use time and energy efficiency as influencing factors, and assign corresponding weights to each influencing factor, and then construct an objective function.

[0034] Specifically, this embodiment can allocate initial cloud computing resources according to task complexity and data scale. In the data processing process, cloud computing resources are dynamically adjusted according to actual load conditions. For example, the number of computing nodes is increased or decreased, and the allocation of memory and CPU resources is adjusted. It is ensured that energy efficiency is considered in the resource allocation process to avoid resource waste.

[0035] S15. After the data is processed, the data analysis results are converted into corresponding text descriptions using natural language generation technology, and presented to enterprise users through the cloud platform in combination with graphical display methods. Compared with the data analysis results, the text description is easier to understand. This step supports enterprise users to conduct in-depth data analysis and decision support.

[0036] On the one hand, this embodiment uses natural language generation technology to automatically generate text descriptions based on data analysis results, including explanations of key indicators, trend analysis, anomaly detection, etc. Natural language processing technology is used to perform grammar and semantic checks on the generated text descriptions to improve the quality of the text descriptions.

[0037] Optionally, the natural language generation technology of deep learning introduced in this embodiment can generate more natural and easy-to-understand text descriptions, thereby improving user understanding and acceptance. Extract key features from the data analysis results, such as time series data, classification data, numerical data, etc.; perform data cleaning to ensure data consistency and integrity, and handle missing values ​​and outliers. Select suitable deep learning models, such as Transformer, Long Short Term Memory (LSTM), etc., to generate high-quality text descriptions. Use a large amount of annotated data to train the model to ensure that the model can generate text descriptions that meet user needs. Improve the generalization ability and generation quality of the model through hyperparameter tuning and regularization techniques. Generate text descriptions using a trained deep learning model, including explanations of key indicators, trend analysis, anomaly detection, etc. Optimize the generated text description through post-processing techniques (such as grammar checking and semantic enhancement) to ensure that its logic is clear and easy to understand.

[0038] On the other hand, this embodiment generates various charts based on the data analysis results, such as line charts, bar charts, pie charts, heat maps, etc. The charts can intuitively display key data and trends, helping enterprise users to quickly focus on important information. This embodiment can also provide interactive visualization tools, allowing enterprise users to obtain data details by clicking, dragging, etc., to enhance the user experience.

[0039] Optionally, this embodiment can combine the generated text description with a chart to generate a multimodal report. For example, the text description can be embedded in a chart to provide richer information. The report can contain multiple parts, such as an overview, detailed analysis, conclusions and recommendations. Alternatively, the generated text description can be converted into speech to provide an audio report to meet the customized needs of different enterprise users.

[0040] The generated reports and charts are presented to enterprise users through the cloud platform. Enterprise users can access and view data analysis results through multiple channels such as web pages and mobile applications. Ensure the compatibility and readability of reports and charts on different devices, and support multi-terminal access such as desktop browsers and mobile devices. Personalize the push of data analysis results based on user preferences and needs. For example, enterprise users can choose to receive email notifications, SMS reminders, or in-app messages.

[0041] Furthermore, the introduction of augmented reality and virtual reality technologies can provide a richer and more immersive visualization experience, helping enterprise users to understand data analysis results more deeply. Specifically, convert data analysis results into three-dimensional (3D) models, such as 3D charts, 3D maps, etc. Generate dynamic 3D animations to show the changing trends and relationships of data. Develop augmented reality / virtual reality (AR / VR) applications to support enterprise users to access and interact through devices such as head mounted displays (HMD) or smartphones. Design enterprise user-friendly interactive interfaces to support enterprise users to explore data through natural interaction methods such as gestures and voice. Ensure the compatibility and accessibility of AR / VR applications on different devices, and support multi-terminal access such as desktops, mobile devices, and HMDs. Support multiple enterprise users to access and interact at the same time to promote teamwork and discussion. For example, market trends are displayed in an AR / VR environment, and enterprise users can explore sales data of different time periods through gestures and voice interactions.

[0042] S16. Collect enterprise users’ feedback and suggestions on data analysis results through a continuous learning mechanism, and optimize the data processing flow based on the feedback and suggestions.

[0043] It should be understood that the embodiments of the present application can provide a variety of feedback channels through the cloud platform, such as online questionnaires, user comments, emails, etc., to facilitate enterprise users to submit feedback and suggestion information. Optionally, the embodiments of the present application can collect feedback from enterprise users on satisfaction with data analysis results, improvement suggestions, new needs, and other aspects. At the same time, the embodiments of the present application can use sentiment analysis technology to evaluate the sentiment tendency of enterprise user feedback and understand the overall satisfaction of enterprise users with data analysis results. Further, the embodiments of the present application can identify the main concerns and common problems in the feedback and suggestion information through thematic analysis technology, and provide direction for subsequent improvements.

[0044] It should also be understood that in addition to optimizing the data processing process, the feedback suggestion information can also automatically update the business logic rule base in combination with new needs to improve the response speed and analysis accuracy of subsequent similar needs, and enhance the intelligence level and service personalization of the enterprise big data analysis and processing system. For example, if an enterprise user feedbacks that a seasonal factor is not fully considered, then the embodiment of the present application can add corresponding rules to the business logic rule base to optimize the existing business logic rules and improve the accuracy and applicability of the rules. For example, optimize the anomaly detection algorithm to improve the accuracy of anomaly detection.

[0045] By executing steps S11 to S16, the embodiment of the present application realizes the automation and intelligence of the entire process from data collection, processing to result presentation. Through multi-tenant isolation technology and continuous optimization mechanism, the security of data and the high quality of data analysis results are ensured. Through the adaptive distributed computing framework, the data processing efficiency is improved, bringing a better big data analysis and processing experience to enterprise users.

[0046] In some possible embodiments, how to dynamically build a target virtual private cloud environment is the key to providing more flexible, efficient, and intelligent data processing services. The existing technology can only provide a static private cloud environment. Based on this, the embodiment of the present application provides a detailed description of the above step S12. Exemplarily, S12, based on basic demand information and business logic rules, dynamically builds the target virtual private cloud environment, including: Step 121: Generate a security policy specified by the enterprise user according to the business logic rules and the enterprise user-specific and privacy protection level requirements in the advanced demand information, wherein the security policy includes specific measures such as data encryption, access control, audit log, data backup and recovery, etc.

[0047] Step 122: dynamically create an initial virtual private cloud environment based on the security policy and basic demand information specified by the enterprise user. The initial virtual private cloud environment can be configured according to the security policy specified by the user to ensure the security of the data to be analyzed in the subsequent entire data processing process.

[0048] As a possible implementation, step 122, based on the security policy and basic requirement information specified by the enterprise user, dynamically create an initial virtual private cloud environment, including: Step a1: convert the security policies and basic demand information specified by the enterprise user into structured configuration parameters, generate a personalized infrastructure as a code template based on the configuration parameters, and configure cloud computing resources according to the personalized infrastructure as a code template to generate a virtual private cloud environment to be optimized.

[0049] Step a2: Based on the access control list, combined with the role-based access control policy and the attribute-based access control policy, the resource access rights in the virtual private cloud environment to be optimized are configured and processed, and the validity and security of the resource access rights are verified through the simulation test tool to obtain a preliminary optimized virtual private cloud environment.

[0050] Specifically, step a2, based on the access control list, combined with the role-based access control policy and the attribute-based access control policy, configures and processes the resource access rights in the virtual private cloud environment to be optimized, including: Step a21: Based on the access control list and in combination with the role-based access control policy, the user roles are hierarchically divided to obtain a role permission set. Based on the attribute-based access control policy, the attribute permission configuration result is generated in combination with the user attributes, cloud computing resource attributes and environmental conditions. Through this step, the enterprise big data analysis and processing system can hierarchically divide the user roles and generate a role permission set. This not only enhances the security of the enterprise big data analysis and processing system, but also provides more fine-grained permission control to ensure that each enterprise user can only access resources related to their roles and responsibilities.

[0051] Step a22: Use the weighted average method to merge the role permission set and the attribute permission configuration results to form a comprehensive access control strategy. This step can balance the permission requirements of different dimensions, provide a more reasonable and comprehensive access control mechanism, and avoid the limitations that may be caused by a single strategy.

[0052] Step a23: According to the comprehensive access control strategy, the cloud computing resources are processed by using the access control list application method to obtain the access control list rules applied to the virtual private cloud environment to be optimized. This embodiment can realize the automation of permission configuration, reduce the need for manual intervention, improve efficiency, and ensure the dynamic and adaptability of permission configuration. By tracking the enterprise user behavior and resource access mode, the enterprise big data analysis and processing system can respond to changes in a timely manner, automatically adjust permissions, and improve the flexibility and response speed of the enterprise big data analysis and processing system.

[0053] Step a24: Based on the access control list rules and in combination with dynamic permission adjustment suggestions, a time series analysis model is used to adjust the resource access rights in the virtual private cloud environment to be optimized; the dynamic permission adjustment suggestions are obtained by tracking the enterprise user behavior and resource access patterns.

[0054] By executing steps a21 to a24, the embodiment of the present application achieves more sophisticated and intelligent permission management and dynamic adjustment in the constructed target virtual private cloud environment through multi-level and multi-dimensional access control strategies. Therefore, the present embodiment provides corresponding advantages in terms of security, fine-grained control, automation and dynamic adjustment, and further enhances the flexibility and responsiveness of the enterprise big data analysis and processing system by introducing intelligent fusion and continuous optimization mechanisms.

[0055] Step a3: In combination with future resource demands, the cloud computing resources of the initially optimized virtual private cloud environment are dynamically adjusted using an adaptive expansion algorithm to obtain a secondarily optimized virtual private cloud environment.

[0056] Specifically, the adaptive expansion algorithm includes an expansion algorithm based on queue length. Step a3, combining future resource requirements, dynamically adjusting the cloud computing resources of the initially optimized virtual private cloud environment using the adaptive expansion algorithm to obtain a secondary optimized virtual private cloud environment, including: Step a31: train the autoregressive integral moving average model based on the enterprise user's historical data analysis requests and the historical cloud computing resource allocation results corresponding to the historical data analysis requests, optimize the autoregressive integral moving average model in combination with business logic rules and seasonal factors, obtain a prediction model for predicting future resource requirements, and use the prediction model to output future resource requirements.

[0057] The optimization process of the autoregressive integrated moving average model can be expressed as follows: ; in, is the future resource demand, is an optimization function used to select , and , to minimize the prediction error, represents the autoregressive integrated moving average model, is the time series data of historical data analysis requests, R is the corresponding historical cloud computing resource allocation result, For business logic rules, It is a seasonal factor. , and are the number of autoregressive terms, difference order and moving average terms of the autoregressive integrated moving average model, respectively. is the number of time points, For time series data At the point in time The actual value of is the autoregressive integrated moving average model at time point The predicted value of is a regularization parameter used to control the influence of business logic rules and seasonal factors. Is combined with business logic rules and sexual factors This step improves the accuracy of future resource demand forecasting and ensures that the enterprise big data analysis and processing system can prepare and allocate necessary cloud computing resources in advance to avoid insufficient or excessive resources.

[0058] Step a32: According to future resource requirements, a threshold setting method is used to determine the scope of automatic expansion triggering conditions and define the instance quantity limit.

[0059] Exemplarily, the range of the automatic extension trigger condition can be expressed by the following formula: ; in, is the minimum trigger condition for automatic expansion, For at time point is the time series forecast of future resource demand, is the maximum trigger condition for automatic expansion, is the low threshold parameter set in this embodiment, which is used to determine the minimum trigger condition. is a high threshold parameter set in this embodiment, which is used to determine the maximum trigger condition. This step can provide a flexible expansion strategy that can automatically adjust under different load conditions to ensure the performance and cost-effectiveness of the enterprise big data analysis and processing system.

[0060] Step a33: According to the range of the automatic expansion triggering conditions and the instance quantity limit, an automated tool is used to configure the adaptive expansion group.

[0061] The process of configuring an adaptive scaling group can be expressed as follows: ; Among them, G is the configured adaptive extension group, Is the first in the adaptive expansion group Instances, is the maximum limit on the number of instances, is the number of instances actually configured. This step realizes the automation of resource management, reduces the risk of human error, and improves the response speed and stability of the enterprise big data analysis and processing system.

[0062] Step a34: Select the actually required extension group from the adaptive extension group using an extension algorithm based on queue length.

[0063] The selection process can be obtained by the following formula: ; in, is the actual desired extension group that is ultimately selected, is any one of the configured adaptive extension groups G, where G is a configured adaptive extension group, is the current task queue length, Is an adaptive expansion group This step improves resource utilization efficiency by intelligently selecting expansion groups, ensuring that the enterprise big data analysis and processing system maintains high performance under high load conditions.

[0064] Step a35: Add the actually required extension group to the initially optimized virtual private cloud environment to dynamically adjust the cloud computing resources and obtain a secondary optimized virtual private cloud environment. This step enables the enterprise big data analysis and processing system to respond to changes in real time through dynamic adjustment, ensuring the optimal configuration of cloud computing resources and improving overall performance and user experience.

[0065] By executing steps a31 to a35, the embodiments of the present application provide corresponding advantages in intelligent resource prediction, flexible expansion strategies, automated configuration, intelligent selection of expansion groups and dynamic adjustment, and further enhance the flexibility, responsiveness and resource utilization efficiency of the enterprise big data analysis system by introducing a variety of technologies and methods.

[0066] By executing steps a1 to a3, the embodiment of the present application provides a highly customized, secure and efficient virtual private cloud environment for enterprise users. First, by converting the security policies and basic demand information specified by enterprise users into structured configuration parameters and generating personalized infrastructure-as-code templates, the automatic configuration of cloud computing resources is realized, and a virtual private cloud environment to be optimized is constructed. Then, based on the access control list, combined with the role-based access control policy and the attribute-based access control policy, the resource access rights are finely configured, and their effectiveness and security are verified through simulation test tools, ensuring the security of the enterprise big data analysis and processing system and the accuracy of permission management, thereby obtaining a preliminary optimized virtual private cloud environment. Finally, the adaptive expansion algorithm is used to dynamically adjust cloud computing resources in combination with future resource needs, and the secondary optimization of the preliminary optimized environment is achieved, ensuring that the enterprise big data analysis and processing system can respond to changes in real time, maintain efficient operation and maximize resource utilization.

[0067] In summary, this method not only provides a highly customized and secure infrastructure, but also improves the flexibility, responsiveness and resource utilization efficiency of enterprise big data analysis and processing systems through intelligent prediction, flexible expansion strategies, automated configuration and dynamic adjustment. This enables enterprise users to efficiently process data analysis tasks in a secure and isolated environment, while ensuring the performance optimization and cost-effectiveness of enterprise big data analysis and processing systems.

[0068] Step 123: Introduce a multi-tenant isolation strategy into the initial virtual private cloud environment to obtain a target virtual private cloud environment, wherein the multi-tenant isolation strategy enables the target virtual private cloud environment to achieve isolation between tenants and isolation within tenants.

[0069] It should be understood that since the customized data analysis requests submitted by different enterprise users correspond to different initial virtual private cloud environments, and each initial virtual private cloud environment has an independent virtual network, cloud computing resources, and data storage resources, it ensures that the data and cloud computing resources of different enterprise users are logically and physically isolated from each other, thereby realizing multi-tenant isolation technology between tenants. Exemplarily, both enterprise user A and enterprise user B need to perform data analysis on the same cloud computing platform, but the data processed by both parties are sensitive and different. This embodiment can dynamically build two independent initial virtual private cloud environments according to the needs of enterprise user A and enterprise user B. These technologies ensure that the virtual environment of each enterprise user runs independently and will not be affected by other enterprise users, preventing data leakage and resource conflicts.

[0070] It should also be understood that the present embodiment can also use multi-tenant isolation technology in the same initial virtual private cloud environment to obtain a target virtual private cloud environment, which can implement multi-tenant isolation technology within a tenant. The multi-tenant isolation technology within the tenant can further enhance the security and privacy of the data to be analyzed. Specifically, the present embodiment can use multi-tenant isolation technology to achieve virtual network isolation, security group configuration, cloud computing resource isolation, storage resource isolation, identity and access management, etc. within the tenant.

[0071] Among them, in order to achieve virtual network isolation, this embodiment divides the virtual network in the same virtual private environment into multiple subnets, each subnet is responsible for different functions. Exemplarily, in the initial virtual private cloud environment of user A, the virtual network is divided into multiple subnets. For example, subnet 1 is used to store user A's data, subnet 2 is used to perform data processing tasks, and subnet 3 is used to provide a data access interface. Set firewall rules to limit communication between different subnets and prevent unauthorized access. For example, rules can be set to allow only communication between specific IP addresses and ports to ensure that only authorized traffic can pass. To implement security group configuration, this embodiment can configure a corresponding security group for each virtual machine in the subnet, define allowed inbound and outbound traffic rules, and ensure that only authorized traffic can pass, thereby enhancing security.

[0072] Among them, cloud computing resource isolation and storage resource isolation can ensure that the cloud computing resources (such as virtual machines, containers) of each enterprise user are physically and logically independent of each other, preventing resource competition and data leakage. Ensure that the data storage resources (such as disks, object storage) of each enterprise user are physically and logically independent of each other to prevent cross-data access. Exemplarily, for enterprise user A, independent CPU, memory and network resources, disk space and object storage are allocated. In addition, in terms of identity and access management, this embodiment can use multi-factor authentication (Multi-Factor Authentication, MFA), single sign-on (Single Sign-On, SSO) and other mechanisms to ensure the authenticity of user identity. Define different user roles and permissions to ensure that enterprise users can only access the data and functions required within their responsibilities. Exemplarily, role 1: financial administrator, can access financial data in subnet 1 and subnet 3 and perform financial-related operations. Role 2: data analyst, can access virtual machines in subnet 2, but cannot access subnet 1.

[0073] As a possible implementation method, a multi-tenant isolation strategy is introduced into the initial virtual private cloud environment to obtain a target virtual private cloud environment. The multi-tenant isolation strategy enables the target virtual private cloud environment to achieve isolation between tenants and within tenants, including: Step b1: Define tenant boundaries. Tenant boundaries are used to reflect the resource allocation and service scope of the corresponding tenant. A tenant includes multiple enterprise users. This step clarifies the resource allocation and service scope of each tenant, ensuring that resources and operations between different tenants are completely isolated to prevent data leakage and cross-access.

[0074] Step b2, evaluate the similarity of the business processes of different tenants, and generate corresponding multi-tenant isolation strategies based on the similarity. The multi-tenant isolation strategies include virtual network isolation strategies, cloud computing resource isolation strategies, and storage resource isolation strategies. The higher the similarity, the more similar the tenants are, and the higher the possibility that similar tenants will attempt to access each other. Therefore, when selecting a multi-tenant isolation strategy, you can choose a highly secure multi-tenant isolation strategy for users with high similarity. By evaluating the isolation strategies generated by the similarity of business processes, you can flexibly adjust them according to actual needs to adapt to different business scenarios and changes, thereby improving the adaptability and scalability of the enterprise big data analysis and processing system.

[0075] Among them, the similarity score can be evaluated using the following formula: ; Is a tenant The similarity score of Is a tenant 1 is the business process feature vector, Is a tenant The business process feature vector, is a weight matrix, reflecting the importance of different features, is the dot product of two eigenvectors, and They are and The norm of is the length of the vector. By combining the cosine similarity and the weight matrix, this embodiment can accurately evaluate the similarity of business processes between tenants and ensure the accuracy of the similarity score.

[0076] The multi-tenant isolation strategy can be generated using the following formula: ; in, For tenants The generated multi-tenant isolation strategy, For tenants 1, V is the virtual network isolation strategy matrix, C is the cloud computing resource isolation strategy matrix, and D is the storage resource isolation strategy matrix. are all weight parameters of the corresponding matrices, respectively representing the importance of similarity, virtual network isolation, cloud computing resource isolation, and storage resource isolation. By comprehensively considering the similarity score and other isolation strategy matrices, this embodiment can generate a personalized isolation strategy for each tenant to ensure that the data and operations of each tenant are properly protected. In addition, the weight parameters allow the embodiment of the present application to dynamically adjust the importance of the isolation strategy according to different needs, ensuring the flexibility and adaptability of the isolation strategy.

[0077] Among them, the virtual network isolation strategy creates an independent virtual network for each tenant to ensure complete isolation of network traffic, enhancing the stability and security of the enterprise big data analysis and processing system. The cloud computing resource isolation strategy uses namespace and containerization technologies to ensure the independence of cloud computing resources, avoid mutual influence between different tenants, and improve resource utilization efficiency. The storage resource isolation strategy allocates independent storage volumes to each tenant and enables encryption functions to ensure physical isolation and security of data, supporting high availability and elastic expansion.

[0078] Step b3: Introduce virtual network isolation strategy, cloud computing resource isolation strategy and storage resource isolation strategy into the initial virtual private cloud environment to obtain the target virtual private cloud environment. The multi-tenant isolation strategy enables the target virtual private cloud environment to achieve virtual network isolation, cloud computing resource isolation and storage resource isolation between tenants, as well as virtual network isolation, cloud computing resource isolation and storage resource isolation within a tenant.

[0079] The target virtual private cloud environment can be generated by the following formula: ; in, For the ultimate goal of virtual private cloud environment, For the initial virtual private cloud environment, For the multi-tenant isolation policy generated for each tenant, is the number of multi-tenant isolation policies, It is the sum of all tenant isolation policies. This method ensures that each tenant has an independent and secure operating space in a shared environment by dynamically superimposing the isolation policies of each tenant, greatly improving the overall security and resource management efficiency of the enterprise big data analysis and processing system.

[0080] By executing steps b1 to b3, the embodiment of the present application not only introduces comprehensive multi-tenant isolation technology in the initial virtual private cloud environment, including virtual network isolation, cloud computing resource isolation, and storage resource isolation, but also realizes dual isolation between tenants and within tenants. This method enhances the security and isolation of the enterprise big data analysis and processing system, provides a more sophisticated and flexible resource management mechanism, and improves resource utilization and user experience. By dynamically adjusting and optimizing isolation strategies, the enterprise big data analysis and processing system can better adapt to different business needs and changes, ensuring that enterprise users perform big data analysis and other key business operations in a safe, efficient and compliant environment.

[0081] By executing steps 121 to 123, the embodiment of the present application not only provides a highly secure, flexible and efficient virtual private cloud environment, but also ensures the security and reliability of the entire data processing process through the introduction of customized security policies and multi-tenant isolation policies.

[0082] In some possible embodiments, in an adaptive distributed computing framework, this embodiment can decompose a complex data processing task into multiple subtasks, each of which can be executed in parallel, and the execution order of the subtasks can be determined according to the task priority and resource availability. This embodiment can also monitor the execution status of each subtask in real time, and can also adjust the cloud computing resource allocation results according to the feedback suggestion information. This embodiment can also summarize the results of each subtask to generate the final data analysis results.

[0083] Exemplarily, S14, using an adaptive distributed computing framework in the target virtual private cloud environment to process all cross-source data in the data set, including: Step 141, calculate the complexity of the data processing task. When the complexity is greater than a preset complexity threshold, divide the data processing task into multiple subtasks in combination with business logic rules, and determine the priority of each subtask in combination with the relationship between the subtasks.

[0084] Among them, the complexity of the data processing task can be determined according to the purpose of analysis, data type, algorithm selection, data quality, etc. Exemplarily, the purpose of analysis can affect the complexity of the data processing task. For example, market trend analysis may involve complex statistical methods such as time series analysis and regression models, while simple customer behavior research may only require basic statistical summaries. At the same time, the data type (such as structured data, unstructured data, semi-structured data) affects the complexity of the data processing task. In addition, the complexity of the algorithm used for the data to be analyzed affects the complexity of the data processing task. For example, the training of a deep learning model is usually more complex than a simple linear regression model. Furthermore, the quality of the data is used to determine how much preprocessing and cleaning work is required later, which also affects the complexity of the data processing task.

[0085] Step 142: Determine the execution order of each subtask according to the task priority of each subtask, allocate cloud computing resources according to the resource availability of computing nodes in the adaptive distributed computing framework, the data size of cross-source data required by the subtask, and the execution order of each subtask, and obtain the cloud computing resource allocation result.

[0086] The data scale can be determined based on the amount of cross-source data required by the subtask, the number of data sources, the time range, the data frequency, etc. For example, the size of the data set (such as the number of rows, the number of columns, the file size) directly affects the data scale. If the data comes from multiple data sources, the number of data sources also needs to be considered when determining the data scale. If the data processing process involves historical data over a long period of time, the amount of data increases accordingly. If the data is real-time streaming data, the data scale changes with the real-time inflow of data.

[0087] Step 143: In the target virtual private cloud environment, combined with the cloud computing resource allocation results, an adaptive distributed computing framework is used to process all cross-source data in the data set, and the execution status of each subtask is monitored in real time during the data processing process. When the execution status of any subtask is detected to be abnormal, the cloud computing resource allocation results are adjusted to generate data analysis results within a preset time.

[0088] The embodiments of the present application ensure that complex data processing tasks can be efficiently decomposed and executed through intelligent task division and priority management, thereby reducing the overall processing time. Cloud computing resource allocation based on task priority and real-time resource availability maximizes resource utilization, avoids idle or excessive resource occupation, and reduces costs. Real-time monitoring and adaptive adjustment mechanisms ensure the stability and reliability of task execution, and can quickly recover even in the event of abnormal situations, ensuring the timeliness and accuracy of data analysis results. Through intelligent task management and resource allocation, user operations are simplified, more efficient and reliable services are provided, and user trust and satisfaction are enhanced. Whether it is market trend analysis or simple customer behavior research, the process can be flexibly adjusted according to specific needs, supporting various complex analysis tasks and meeting the diverse business needs of enterprises.

[0089] In summary, steps 141 to 143 not only improve the management efficiency and resource utilization of data processing tasks, but also enhance the reliability and responsiveness of the enterprise big data analysis and processing system through real-time monitoring and adaptive adjustment mechanisms, providing enterprise users with efficient, secure and flexible big data analysis solutions.

[0090] With respect to S16, collecting enterprise users' feedback and suggestion information on data analysis results through a continuous learning mechanism, and optimizing the data processing flow according to the feedback and suggestion information, the embodiment of the present application is described as follows: Specifically, this embodiment can optimize the data processing process according to the feedback suggestion information and the updated business logic rules. For example, the data preprocessing steps can be optimized to reduce the time for data cleaning. The speed and efficiency of data processing can be improved through performance testing and tuning. For example, the resource allocation strategy of the distributed computing framework can be optimized to reduce the task execution time.

[0091] Optionally, the embodiments of the present application can improve the intelligence level of the enterprise big data analysis and processing system through machine learning and artificial intelligence technology. For example, a machine learning model can be used to predict the preferences and actual needs of enterprise users and provide personalized analysis suggestions.

[0092] For example, the embodiments of the present application can introduce reinforcement learning technology to achieve more efficient feedback processing and optimization of enterprise big data analysis and processing systems, and improve the intelligence level and service personalization of enterprise big data analysis and processing systems. Specifically, select suitable reinforcement learning models, such as Q-learning, deep Q-learning network (Q-learning Deep Q-Network, DQN), etc., to optimize the business logic rule base and data processing process. Use the collected user feedback data to train the model to ensure that the model can self-optimize according to user feedback. Deploy the trained model to the production environment to achieve real-time optimization. Update the business logic rule base according to the suggestions of the reinforcement learning model. For example, if the model recommends increasing the weight of a seasonal factor, corresponding adjustments can be made in the rule base. Optimize existing business logic rules to improve the accuracy and applicability of the rules. For example, optimize the anomaly detection algorithm to improve the accuracy of anomaly detection. Optimize the data processing process according to the suggestions of the reinforcement learning model. For example, optimize the data preprocessing steps to reduce the time for data cleaning.

[0093] For another example, the embodiment of the present application can improve the speed and efficiency of data processing through performance testing and tuning. According to the user's historical behavior and feedback suggestion information, a user profile is constructed to understand the preferences and actual needs of enterprise users. This embodiment can use recommendation system technology to recommend personalized analysis tools and templates based on user profiles. For example, relevant analysis tools and templates are recommended for users who often perform market trend analysis.

[0094] Optionally, this embodiment can introduce a federated learning algorithm to achieve collaborative optimization of multiple enterprise users and improve the overall performance and personalized service level of the enterprise big data analysis and processing system. Specifically, each enterprise user trains its own model locally and generates local model parameters. Using a federated learning algorithm, the local model parameters of multiple enterprise users are aggregated to generate global model parameters. The global model parameters are sent to each enterprise user to update the local model of the enterprise user. Through multiple rounds of iterations, the global model is gradually optimized to improve the overall performance of the enterprise big data analysis and processing system. Generate personalized models and provide personalized services based on the feedback and suggestion information and actual needs of each enterprise user. Support collaboration and sharing among multiple enterprise users to promote teamwork and knowledge exchange.

[0095] Optionally, the data to be analyzed and the corresponding feedback suggestion information can be generated locally to ensure the privacy and security of the data. In addition, the present embodiment can use data encryption technology to ensure the security of various data during transmission and storage.

[0096] Therefore, the embodiments of the present application have the following advantages: (1) In the process of dynamically building the target virtual private cloud environment, each enterprise user has an independent virtual private cloud environment to ensure the isolation of data and cloud computing resources. Multi-tenant isolation technology can achieve isolation between tenants and within tenants, further enhancing data security and privacy.

[0097] (2) The data processing stage uses an adaptive distributed computing framework to provide efficient data processing services. The adaptive distributed computing framework can efficiently process large amounts of data, ensuring that data processing tasks are completed in the shortest possible time while maintaining optimal energy efficiency.

[0098] (3) Convert complex data analysis results into easy-to-understand text descriptions and charts, and help business users deeply understand the data and support decision making through multimodal reporting and immersive visualization technology.

[0099] (4) Through continuous learning and optimization, the business logic rule base and data processing flow are automatically adjusted according to feedback and suggestion information, so as to improve the intelligence level and personalized service capabilities of the enterprise big data analysis and processing system, and ensure better user experience and higher analysis accuracy.

[0100] Figure 2 A schematic diagram of the structure of a cloud computing-based enterprise big data analysis and processing system provided in an embodiment of the present application is shown in FIG. Figure 2 As shown, the system includes: The receiving module 21 is used to receive a customized data analysis request submitted by an enterprise user through the cloud platform, wherein the customized data analysis request includes: basic demand information and advanced demand information, wherein the advanced demand information includes business logic rules specific to the enterprise user; A construction module 22, used to dynamically construct a target virtual private cloud environment based on basic demand information and business logic rules; The identification and verification module 23 is used to identify and aggregate cross-source data that is scattered in different data sources and is related to the basic demand information to obtain a data set, and perform data consistency verification on all cross-source data in the data set to ensure the accuracy and integrity of the cross-source data. After all cross-source data pass the data consistency verification, the data set is loaded into the target virtual private cloud environment; The data processing module 24 is used to process all cross-source data in the data set using an adaptive distributed computing framework in the target virtual private cloud environment; The conversion module 25 is used to convert the data analysis results into corresponding text descriptions by using natural language generation technology after the data processing is completed, and present them to enterprise users through the cloud platform in combination with a graphical display method; The collection and optimization module 26 is used to collect feedback and suggestion information from enterprise users on data analysis results through a continuous learning mechanism, and optimize the data processing process based on the feedback and suggestion information.

[0101] Figure 2 The cloud computing-based enterprise big data analysis and processing system can perform Figure 1 The implementation principle and technical effects of the enterprise big data analysis and processing method based on cloud computing described in the illustrated embodiment will not be described in detail. The specific manner in which each module and unit performs operations in the enterprise big data analysis and processing system based on cloud computing in the above embodiment has been described in detail in the embodiment of the method, and will not be elaborated here.

[0102] In one possible design, Figure 2 The enterprise big data analysis and processing system based on cloud computing in the illustrated embodiment can be implemented as a computing device, such as Figure 3 As shown, the computing device may include a storage component 31 and a processing component 32 .

[0103] The storage component 31 stores one or more computer instructions, wherein the one or more computer instructions are called and executed by the processing component 32 .

[0104] The processing component 32 is used to: receive customized data analysis requests submitted by enterprise users through the cloud platform, the customized data analysis requests include: basic demand information and advanced demand information, the advanced demand information includes business logic rules specific to enterprise users; dynamically build a target virtual private cloud environment based on the basic demand information and business logic rules; identify and aggregate cross-source data that are scattered in different data sources and are related to the basic demand information to obtain a data set, perform data consistency checks on all cross-source data in the data set to ensure the accuracy and completeness of the cross-source data, and load the data set into the target virtual private cloud environment after all cross-source data pass the data consistency check; use an adaptive distributed computing framework in the target virtual private cloud environment to process all cross-source data in the data set; after the data processing is completed, use natural language generation technology to convert the data analysis results into corresponding text descriptions, and present them to enterprise users through the cloud platform in combination with graphical display methods; collect feedback and suggestion information on data analysis results from enterprise users through a continuous learning mechanism, and optimize the data processing process based on the feedback and suggestion information.

[0105] The processing component 32 may include one or more processors to execute computer instructions to complete all or part of the steps in the above method. Of course, the processing component may also be implemented by one or more application-specific integrated circuits (ASIC), digital signal processors (DSP), digital signal processing devices (DSPD), programmable logic devices (PLD), field programmable gate arrays (FPGA), controllers, microcontrollers, microprocessors or other electronic components to perform the above method.

[0106] The storage component 31 is configured to store various types of data to support operations at the terminal. The storage component can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as random access memory (RAM), static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0107] Of course, the computing device may also include other components, such as input / output interfaces, display components, communication components, etc.

[0108] The input / output interface provides an interface between the processing component and the peripheral interface module, which may be an output device, an input device, etc.

[0109] The communication component is configured to facilitate, among other things, wired or wireless communications between the computing device and other devices.

[0110] Among them, the computing device can be a physical device or an elastic computing host provided by a cloud computing platform, etc. In this case, the computing device can refer to a cloud server, and the above-mentioned processing components, storage components, etc. can be basic server resources rented or purchased from the cloud computing platform.

[0111] The present application also provides a computer storage medium storing a computer program, wherein the computer program can achieve the above-mentioned Figure 1 The illustrated embodiment is a method for analyzing and processing enterprise big data based on cloud computing.

[0112] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0113] The system embodiment described above is merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment. Those of ordinary skill in the art may understand and implement it without creative work.

[0114] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0115] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for analyzing and processing enterprise big data based on cloud computing, characterized in that: include: Receiving a customized data analysis request submitted by an enterprise user through a cloud platform, wherein the customized data analysis request includes: basic demand information and advanced demand information, wherein the advanced demand information includes business logic rules specific to the enterprise user; Based on the basic demand information and the business logic rules, dynamically construct a target virtual private cloud environment; Identify and aggregate cross-source data that is scattered in different data sources and is related to the basic demand information to obtain a data set, perform data consistency checks on all cross-source data in the data set to ensure the accuracy and integrity of the cross-source data, and load the data set into the target virtual private cloud environment after all cross-source data pass the data consistency check; Using an adaptive distributed computing framework in the target virtual private cloud environment to process all cross-source data in the data set; After the data is processed, natural language generation technology is used to convert the data analysis results into corresponding text descriptions, and then combined with graphical display methods, it is presented to enterprise users through the cloud platform; The continuous learning mechanism is used to collect feedback and suggestion information from enterprise users on the data analysis results, and the data processing flow is optimized based on the feedback and suggestion information.

2. The method according to claim 1, characterized in that The dynamically constructing a target virtual private cloud environment based on the basic demand information and the business logic rules includes: Generate a security policy specified by the enterprise user according to the business logic rules and the enterprise user-specific and privacy protection level requirements in the high-level demand information; Dynamically create an initial virtual private cloud environment based on the security policy specified by the enterprise user and the basic demand information; A multi-tenant isolation strategy is introduced into the initial virtual private cloud environment to obtain a target virtual private cloud environment, and the multi-tenant isolation strategy enables the target virtual private cloud environment to achieve isolation between tenants and isolation within tenants.

3. The method according to claim 2, characterized in that The dynamically creating an initial virtual private cloud environment based on the security policy specified by the enterprise user and the basic requirement information includes: Convert the security policies and basic demand information specified by the enterprise user into structured configuration parameters, generate a personalized infrastructure as code template according to the configuration parameters, and configure cloud computing resources according to the personalized infrastructure as code template to generate a virtual private cloud environment to be optimized; Based on the access control list, combined with the role-based access control strategy and the attribute-based access control strategy, the resource access rights in the virtual private cloud environment to be optimized are configured and processed, and the validity and security of the resource access rights are verified through the simulation test tool to obtain a preliminary optimized virtual private cloud environment; Combined with future resource demands, an adaptive expansion algorithm is used to dynamically adjust the cloud computing resources of the initial optimized virtual private cloud environment to obtain a secondary optimized virtual private cloud environment.

4. The method according to claim 3, characterized in that: The configuration process of resource access rights in the virtual private cloud environment to be optimized based on the access control list and in combination with the role-based access control policy and the attribute-based access control policy includes: Based on the access control list and the role-based access control policy, the user roles are hierarchically divided to obtain the role permission set. Based on the attribute-based access control policy, the attribute permission configuration result is generated by combining the user attributes, cloud computing resource attributes and environmental conditions. The role permission set and the attribute permission configuration result are merged using a weighted average method to form a comprehensive access control strategy; According to the comprehensive access control strategy, the cloud computing resources are processed by using the access control list application method to obtain the access control list rules applied to the virtual private cloud environment to be optimized; Based on the access control list rules and in combination with dynamic permission adjustment suggestions, a time series analysis model is used to adjust the resource access rights in the virtual private cloud environment to be optimized; the dynamic permission adjustment suggestions are obtained by tracking enterprise user behaviors and resource access patterns.

5. The method according to claim 3, characterized in that: The adaptive expansion algorithm includes an expansion algorithm based on queue length. The adaptive expansion algorithm is used to dynamically adjust the cloud computing resources of the initial optimized virtual private cloud environment in combination with future resource requirements to obtain a secondary optimized virtual private cloud environment, including: Based on the historical data analysis requests of enterprise users and the historical cloud computing resource allocation results corresponding to the historical data analysis requests, the autoregressive integral moving average model is trained, and the autoregressive integral moving average model is optimized by combining business logic rules and seasonal factors to obtain a prediction model for predicting future resource demand, and the prediction model is used to output future resource demand; According to the future resource demand, a threshold setting method is used to determine the scope of the automatic expansion trigger condition and define the instance quantity limit; Use automated tools to configure adaptive scaling groups based on the range of automatic scaling trigger conditions and instance quantity limits; An expansion algorithm based on queue length is used to select the actually required expansion group from the adaptive expansion group; The actually required extension groups are added to the initially optimized virtual private cloud environment to achieve dynamic adjustment of cloud computing resources and obtain a secondary optimized virtual private cloud environment.

6. The method according to claim 2, characterized in that Introducing a multi-tenant isolation strategy into the initial virtual private cloud environment to obtain a target virtual private cloud environment, wherein the multi-tenant isolation strategy enables the target virtual private cloud environment to achieve inter-tenant isolation and intra-tenant isolation, including: Define tenant boundaries, where the tenant boundaries are used to reflect resource allocation and service scope of the corresponding tenant. One tenant includes multiple enterprise users. Evaluate the similarity of business processes of different tenants, and generate corresponding multi-tenant isolation strategies according to the similarity, wherein the multi-tenant isolation strategies include virtual network isolation strategies, cloud computing resource isolation strategies, and storage resource isolation strategies; A virtual network isolation strategy, a cloud computing resource isolation strategy and a storage resource isolation strategy are introduced into the initial virtual private cloud environment to obtain a target virtual private cloud environment. The multi-tenant isolation strategy enables the target virtual private cloud environment to achieve virtual network isolation, cloud computing resource isolation and storage resource isolation between tenants, as well as virtual network isolation, cloud computing resource isolation and storage resource isolation within a tenant.

7. The method according to claim 1, characterized in that The adopting an adaptive distributed computing framework in the target virtual private cloud environment to process all cross-source data in the data set includes: Calculate the complexity of the data processing task, and if the complexity is greater than a preset complexity threshold, divide the data processing task into multiple subtasks in combination with business logic rules, and determine the priority of each subtask in combination with the association relationship between the subtasks; According to the task priority of each subtask, the execution order of each subtask is determined, and cloud computing resources are allocated according to the resource availability of computing nodes in the adaptive distributed computing framework, the data size of cross-source data required by the subtask and the execution order of each subtask, and the cloud computing resource allocation result is obtained; In the target virtual private cloud environment, combined with the cloud computing resource allocation results, an adaptive distributed computing framework is used to process all cross-source data in the data set, and the execution status of each subtask is monitored in real time during the data processing process. When the execution status of any subtask is detected to be abnormal, the cloud computing resource allocation results are adjusted to generate data analysis results within a preset time.

8. A cloud computing-based enterprise big data analysis and processing system, characterized in that: include: A receiving module, configured to receive a customized data analysis request submitted by an enterprise user through a cloud platform, wherein the customized data analysis request includes: basic demand information and advanced demand information, wherein the advanced demand information includes business logic rules specific to the enterprise user; A construction module, used to dynamically construct a target virtual private cloud environment based on the basic demand information and the business logic rules; An identification and verification module is used to identify and aggregate cross-source data that is scattered in different data sources and is related to the basic demand information to obtain a data set, and perform data consistency verification on all cross-source data in the data set to ensure the accuracy and integrity of the cross-source data. After all cross-source data pass the data consistency verification, the data set is loaded into the target virtual private cloud environment; A data processing module, used for processing all cross-source data in the data set by using an adaptive distributed computing framework in a target virtual private cloud environment; The conversion module is used to convert the data analysis results into corresponding text descriptions using natural language generation technology after the data processing is completed, and present them to enterprise users through the cloud platform in combination with graphical display methods; The collection and optimization module is used to collect feedback and suggestion information from enterprise users on the data analysis results through a continuous learning mechanism, and optimize the data processing flow according to the feedback and suggestion information.

9. A computing device, characterized in that It comprises a processing component and a storage component; the storage component stores one or more computer instructions; the one or more computer instructions are used to be called and executed by the processing component to implement a cloud computing-based enterprise big data analysis and processing method as described in any one of claims 1 to 7.

10. A computer storage medium, characterized in that: A computer program is stored, and when the computer program is executed by a computer, an enterprise big data analysis and processing method based on cloud computing as described in any one of claims 1 to 7 is implemented.

Images