A Data Management System for Simulation Process
By combining edge computing, cloud computing and blockchain technology, the problems of poor data management and task scheduling dependence on manual in the existing simulation process management system are solved, efficient processing and secure sharing of simulation data are realized, and the simulation process is improved Efficiency and safety.
Patent Information
- Application Number
- CN202410377272.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-03-29
- Publication Date
- 2024-11-15
- Estimated Expiration
- 2044-03-29
AI Technical Summary
The existing simulation process management system lacks an effective data management mechanism, which leads to the failure to fully realize the potential of simulation data in the design and optimization process. Data security and permission control are difficult to achieve in a multi-user and multi-organization environment. Traditional task scheduling relies on manual Experience, resource utilization is low and simulation cycle is long.
Adopt edge computing, cloud computing, blockchain technology, data directory management, advanced data retrieval and analysis, simulation design and optimization tools, task and progress monitoring, user portal and permission management and other technologies to achieve efficient processing and secure sharing of simulation data , intelligent retrieval and analysis, dynamic task scheduling and fine-grained permission control.
It improves the overall efficiency and security of the simulation process, realizes the effective utilization and reuse of simulation data, optimizes resource allocation and task execution, and enhances user experience and system security.
Smart Images

Figure CN118278188B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of computer simulation, and in particular to a data management system for a simulation process. Background Art
[0002] With the widespread application of simulation technology, a large amount of simulation data has been generated in the fields of engineering design, scientific research, product development, etc. The effective management, processing and utilization of these data have become the key to improving the efficiency and quality of simulation processes; however, existing simulation process management systems often lack effective data management mechanisms and cannot fully tap the potential of simulation data in the design and optimization process; at the same time, data security and authority control also face challenges in a multi-user and multi-organizational collaborative environment; in addition, traditional simulation task scheduling and optimization mainly rely on manual experience, lack flexibility and adaptability, resulting in low resource utilization and long simulation cycles; therefore, there is an urgent need for a simulation process data management system that can comprehensively solve the above problems to support the efficient execution of complex simulation tasks and the effective use of simulation data. Summary of the invention
[0003] In view of the shortcomings of the prior art, the present invention proposes an integrated system solution, which aims to achieve efficient processing, secure sharing, intelligent retrieval and analysis, dynamic task scheduling and fine-grained permission control of simulation data through edge computing, cloud computing, blockchain technology, data directory management, advanced data retrieval and analysis, simulation design and optimization tools, task and progress monitoring, and user portal and permission management technologies, thereby improving the overall efficiency and security of the simulation process.
[0004] The system of the present invention is composed as follows:
[0005] Edge Data Processing Unit (EDPU): Located at the forefront of the system, it is responsible for preliminary data processing at the source of data generation, including data collection, preprocessing and preliminary analysis. By performing these operations on the edge side, it can effectively reduce the burden on the cloud data processing center, reduce the delay and cost of data transmission, and ensure the timeliness and feasibility of data.
[0006] Cloud Data Processing Center (CDPC): As the core of the system, it is responsible for receiving and storing processed data from the edge data processing unit, conducting in-depth analysis, storing simulation results, and generating optimization strategies; processing and analyzing large-scale simulation data, thereby providing a scientific basis for simulation design and optimization.
[0007] Blockchain Data Sharing Network (BDSN): Utilizes blockchain technology to achieve secure sharing of simulation data and manages data sharing rules and permissions through smart contracts; this mechanism ensures the immutability, security and traceability of data, and is particularly suitable for cross-institutional or cross-field simulation project collaboration.
[0008] Data Catalog and Metadata Management System (DCMS): responsible for managing simulation data and its metadata, supporting data classification, retrieval and version control; by effectively managing metadata, it improves the discoverability and availability of data, and facilitates the execution of simulation tasks and the reuse of simulation results.
[0009] Advanced Search and Analysis Engine (ASAE): It implements advanced search functions based on metadata in the data catalog and metadata management system, and performs data mining tasks to provide in-depth data support for simulation design and optimization; it intelligently analyzes simulation data to reveal patterns and trends behind the data and assist in decision making.
[0010] Simulation Design and Optimization Tool (SDOT): provides a user-friendly interface that supports the design, editing, and optimization of simulation models. Based on the analysis results and optimization suggestions provided by the cloud data processing center, it can automatically or semi-automatically adjust simulation parameters to improve the accuracy and efficiency of simulation.
[0011] Task and Progress Monitoring Center (TPMS): manages the entire life cycle of simulation tasks, from task submission to execution, monitoring and notification; realizes dynamic scheduling and real-time monitoring of simulation tasks, optimizes the task management process, and improves the efficiency and response speed of task execution.
[0012] User Portal and Permission Management Center (UPPM): Provides the system's user interface and implements the management of user permissions and roles; improves user experience and system security through a role-based dynamic permission allocation mechanism.
[0013] Furthermore, the working principle and scope of the edge data processing unit are as follows:
[0014] Perform preliminary data processing at the edge of the data generation source (such as sensors, devices, etc.); this includes data cleaning and standardization, etc., in order to reduce the burden on the central server, reduce the amount of data transmission, and improve the real-time and effectiveness of data processing. The operations involved include but are not limited to:
[0015] A. Data Collection
[0016] Collect data in real time from connected sensors and devices.
[0017] B. Data Cleansing
[0018] An adaptive data cleaning algorithm is applied to identify outliers and clean data. Its innovation lies in proposing a dynamic outlier identification method based on the long short-term memory network (LSTM); LSTM can capture long-term dependencies in time series data, and dynamically predict the expected value and its confidence interval at each time point by learning the normal patterns of historical data, thereby realizing dynamic identification of outliers.
[0019] Specifically, the method is as follows:
[0020] Data preprocessing: Standardize the original time series data to eliminate the dimension effect and reduce the impact of extreme values.
[0021] LSTM model training: The preprocessed historical time series data is used as the model input to predict the value at the next time point, and minimizing the difference between the predicted value and the actual value is used as the model optimization goal.
[0022] Outlier identification: For each time point t, use the trained LSTM model to predict its value and confidence interval, if the actual observed value If it is not within the predicted confidence interval, then it is considered It is an outlier and needs to be removed.
[0023] Confidence interval calculation: using the standard deviation of the model predictions and a preset confidence level , calculate the confidence interval ,in , .
[0024] Parameter adjustment: According to the accuracy and recall of outlier detection in actual applications, adjust the structure (such as the number of layers and neurons) and training parameters (such as learning rate and batch size) of the LSTM model.
[0025] C. Missing value handling
[0026] In order to improve the efficiency and accuracy of the edge data processing unit in processing missing values, we adopted an adaptive method to fill the missing values, and different missing value filling methods need to be selected for different types of data.
[0027] Specifically, the missing value filling strategy is as follows:
[0028] Time series data: If the data shows a clear linear trend, use linear interpolation; if the data shows a nonlinear trend, polynomial interpolation can be used.
[0029] Spatial data: Apply the K-nearest neighbor (KNN) algorithm to fill in missing values based on the values of neighboring data points, with weights adjusted based on distance.
[0030] D. Data upload
[0031] The processed data is securely uploaded to the cloud data processing center for processing.
[0032] Furthermore, the working principle and scope of the cloud data processing center are as follows:
[0033] Receive processed data from the edge data processing unit, conduct further in-depth analysis, store simulation results, and generate simulation optimization strategies based on the analysis results. The operations involved include but are not limited to:
[0034] A. Receiving data
[0035] Receive processed data sent by the edge data processing unit in real time through a high-speed network connection.
[0036] B. Data Storage
[0037] In order to meet the large-scale simulation data storage needs from different edge data processing units, the cloud data processing center adopts a distributed database system, such as Hadoop's HDFS or other distributed file systems, to achieve efficient and reliable data storage, support data redundant backup and fault recovery, and ensure the long-term security of data.
[0038] C. Data Analysis
[0039] Use big data processing technologies, such as Apache Hadoop or Apache Spark, to achieve distributed processing of stored data. These frameworks support parallel computing of large-scale data sets and significantly improve the speed and efficiency of data processing by distributing computing tasks to multiple computing nodes for execution. According to specific analysis needs, implement algorithms such as time series analysis, cluster analysis, and association rule learning.
[0040] D. Simulation Optimization
[0041] Generally speaking, we can consider the relationship between simulation parameters and simulation effects as an optimization problem:
[0042] Assume that our goal is to minimize the error of a certain performance indicator in the simulation. This performance indicator can be any quantitative simulation result such as structural strength, energy consumption, response time, etc.; define the objective function is the square of the difference between the expected value of the simulation performance index and the simulation experimental value, that is, the sum of squared errors (SSE). Our goal is to adjust the simulation input parameters X to minimize the objective function :
[0043]
[0044] in:
[0045] is the expected value of the performance index in the ith simulation experiment;
[0046] is the simulated value of the performance index when the input parameter X is given in the i-th simulation experiment;
[0047] is the total number of simulation experiments.
[0048] In order to find the minimum objective function The parameter X can be optimized by gradient descent:
[0049]
[0050] Where:
[0051] , They are the values before and after the parameter update;
[0052] is the learning rate, which controls the step size of parameter update;
[0053] is the objective function J with respect to X in The gradient at indicates the direction in which J grows fastest;
[0054] In each iteration, the gradient of the current parameters is calculated , and then update the parameters according to the gradient direction until J(X) converges to the minimum value or reaches the preset number of iterations.
[0055] E. Parameter Adjustment Suggestion Generation
[0056] By analyzing the current simulation data and historical simulation results, the influence of each parameter on the simulation results is analyzed, and optimization suggestions for simulation parameters are automatically put forward to optimize the subsequent simulation process.
[0057] Furthermore, the working principle and scope of the blockchain data sharing network are as follows:
[0058] Through the distributed ledger based on blockchain technology, the security, transparency and immutability of simulation data are ensured; each data transaction (including data upload, access and modification) is recorded as a block and connected into a chain through encryption algorithms to form a traceable data operation history. The operations involved include but are not limited to:
[0059] A. Data upload and encryption
[0060] The data provider uploads the simulation data to the blockchain data sharing network and encrypts it using the public key of the data requester. The encrypted simulation data is uploaded to the blockchain network along with the transaction information; the transaction information includes the data metadata, upload timestamp, data provider and expected data requester identification.
[0061] B. Block Creation
[0062] The system packages multiple data upload transactions collected within a certain period of time and creates a new block. In addition to multiple data upload transactions, each block also contains a hash value pointing to the previous block to ensure the continuity of the blockchain. The newly created block is hashed to generate a unique identifier (hash value) for the block. The hash calculation includes encryption of the block content to ensure that the block content cannot be tampered with.
[0063] C. Consensus Mechanism Execution
[0064] After receiving the new block, the nodes in the network begin to verify the validity of the block. The verification process includes checking the validity of the block hash value, the legitimacy of the transaction, and the consistency of the blockchain; using a specific consensus algorithm (such as proof of work PoW, proof of stake PoS, etc.) to ensure that most nodes in the network agree to the addition of the new block; once a consensus is reached, the new block is added to the blockchain and becomes part of the consensus of the entire network.
[0065] D. Smart Contract Management Permissions
[0066] Define and deploy smart contracts on the blockchain. Smart contracts contain data access rules and logic, such as data access rights, data usage fees, etc. When a data requester initiates a data access request, the smart contract automatically executes to verify the requester's identity and access rights. Only requests that pass the verification are allowed to access the corresponding data.
[0067] E. Data Access and Decryption
[0068] F. After verification by the smart contract, the data requester is authorized to access specific encrypted data; the data requester uses his own private key to decrypt the encrypted data; the decrypted data can be used for subsequent simulation analysis and processing by the requester; the smart contract can also record the access and use of data, providing traceability and transparency for the use of data.
[0069] Furthermore, the working principle and scope of the data directory and metadata management system are as follows:
[0070] By establishing a unified metadata framework, a structured description and indexing method is provided for simulation data.
[0071] Metadata, that is, data about data, covers key information such as the source, format, creation time, last modification time, version information, etc. A basic metadata model may contain attributes such as data ID, data type, creation time, description, version number, etc. In addition, the model is designed to be extensible and can adapt to the specific needs of different fields by adding custom attributes.
[0072] The data directory and metadata management system also implements a version control mechanism. Whenever the data changes, the system records the new version of the data. The version information becomes an important part of the metadata. Version control uses serial numbers or time-based naming strategies (such as v1.0, 2021-03-01-01) to ensure that historical changes in data can be accurately tracked and managed. The operations involved include but are not limited to:
[0073] A. Metadata Creation
[0074] Whenever new simulation data is generated, the system automatically or requires manual intervention by the user to create corresponding metadata records. These metadata records describe in detail key information such as the source, type, creation time, and last modification time of the simulation data.
[0075] B. Data Storage
[0076] The simulation data and its corresponding metadata are securely stored in the data warehouse. To ensure the integrity and consistency of the data, the storage solution may use a relational database or a NoSQL database, depending on the structural complexity of the metadata and the query requirements. At the same time, the system will create indexes for commonly used query attributes, such as data ID and creation time, to improve retrieval efficiency.
[0077] C. Data Retrieval
[0078] Users can retrieve the required simulation data by querying metadata, supporting flexible queries based on multi-attribute combinations, such as searching by data type and creation time period.
[0079] D. Version management
[0080] When the simulation data is modified, the data directory and metadata management system will record the new version information while retaining the old version information, supporting comparison and rollback operations between versions; to save storage space, the version control mechanism uses differential storage and only saves the difference data between versions; concurrency control strategies, such as optimistic locking or pessimistic locking, are adopted to manage concurrent modifications to data and avoid version conflicts.
[0081] E. Data Sharing and Rights Management
[0082] Based on the permission information contained in the metadata, the data directory and metadata management system manages data access and sharing, and implements dynamic permission verification for data access based on blockchain technologies such as smart contracts, ensuring that only authorized users can access specific data sets.
[0083] Furthermore, the working principle and scope of the advanced search and analysis engine are as follows:
[0084] Based on data mining and machine learning technology, it intelligently retrieves and analyzes simulation data stored in data directories and metadata management systems; it uses the rich information described by metadata to achieve advanced retrieval of simulation data, and through data mining technology, it extracts patterns, trends and associations from large amounts of simulation data, providing a scientific basis for simulation design and optimization.
[0085] Specifically, the advanced search function can be expressed as a multi-dimensional query problem, where the query condition C can contain constraints in multiple dimensions, such as time, simulation type, parameter range, etc.:
[0086]
[0087] Where:
[0088] D represents the simulation data entity;
[0089] C represents the multidimensional query condition specified by the user.
[0090] Step 5.1 User query construction
[0091] Users enter query conditions through the graphical user interface (GUI) or command line interface (CLI) provided by the advanced search and analysis engine; the interface should support flexible query condition combinations, including but not limited to data type, creator, specific time period, etc.; the system parses the query conditions entered by the user and converts them into query expressions that can be understood by the system.
[0092] The operations involved include but are not limited to:
[0093] A. Perform an Advanced Search
[0094] The advanced search and analysis engine accesses the metadata stored in the data directory and metadata management system to perform efficient searches based on the query conditions constructed by the user; this step may involve complex database query operations such as joins and filters; to improve search efficiency, the system uses pre-established indexes to quickly locate relevant simulation data sets, and the index structure should be optimized for commonly used query conditions.
[0095] B. Data Mining Task Selection
[0096] Based on the user's simulation design and optimization goals, determine the appropriate data mining task type. For example, in order to discover the intrinsic associations between data sets, you can choose association rule learning; in order to classify data, you can choose a classification algorithm; according to the determined task type, select one or more applicable data mining algorithms from the algorithm library for subsequent processing.
[0097] Taking association rule learning as an example, the goal is to discover the association rules between simulation parameters and simulation results:
[0098]
[0099] Where:
[0100] X and Y represent the simulation parameter combination and simulation result set, respectively;
[0101] s represents support, the frequency of X and Y occurring simultaneously;
[0102] c represents the confidence level, which is the probability of Y occurring under the condition that X occurs.
[0103] C. Data Preprocessing
[0104] The retrieved data is preprocessed, including data cleaning (removing or correcting erroneous and missing data records, including filling missing values, deleting outliers, etc.), feature extraction (extracting features from raw data that are helpful for data mining tasks, including statistical calculations, text analysis, etc.), and standardization (standardizing feature data, such as normalization and standard deviation standardization, to eliminate the impact of different dimensions).
[0105] D. Perform data mining and analysis
[0106] Use the selected data mining algorithm to analyze the preprocessed data, including the model training and verification process, to ensure the accuracy and reliability of the analysis results. For large-scale data sets, use distributed computing or parallel computing technology to improve data processing speed.
[0107] E. Result interpretation and application
[0108] Convert the results of data mining and analysis into intuitive and easy-to-understand formats, such as charts, reports, etc., and generate specific simulation parameter adjustment suggestions or design optimization solutions based on the analysis results.
[0109] Further, the working principle and scope of the simulation design and optimization tool are as follows:
[0110] Combining the functions of simulation modeling, parameter optimization and result analysis, it supports users to complete the full life cycle management of simulation models in a single environment. It automatically receives the analysis results and optimization suggestions of simulation data through deep integration with the cloud data processing center, making the simulation process more intelligent and automated. The operations involved include but are not limited to:
[0111] A. Model Design
[0112] Users use graphical tools to build simulation models and define preliminary simulation parameters.
[0113] B. Simulation Execution
[0114] Execute simulation tasks locally or in the cloud and collect simulation results.
[0115] C. Send the simulation results to the cloud data processing center for in-depth analysis to obtain performance indicators and potential optimization directions.
[0116] D. Parameter Optimization
[0117] Automatically or manually adjust simulation parameters based on feedback and optimization suggestions from the cloud data processing center.
[0118] In general, it is assumed that the performance indicators of the simulation model can be quantified by the objective function J(X), where is the parameter vector of the simulation model. The goal is to find a set of parameters , so that the objective function J(X) is minimized:
[0119]
[0120] In the formula Indicates the search for The set of parameters to minimize , and then optimize the target through algorithms such as gradient descent or genetic algorithm.
[0121] Regarding the sensitivity analysis of parameters, we can Slight changes , observe the change of objective function J(X) , to assess the sensitivity or importance of parameters:
[0122]
[0123] Where:
[0124] Indicates sensitivity;
[0125] Represents the objective function J with respect to the parameter The partial derivative of The rate of change of J when a small change occurs.
[0126] E. Optimization Iteration
[0127] The simulation is repeated until the performance target is met or a preset number of iterations is reached.
[0128] F. Result Feedback and Application
[0129] Display the final simulation results, support the export and visualization of the results, and adjust the simulation model design according to the optimization results.
[0130] Furthermore, the working principle and scope of the task and progress monitoring center are as follows:
[0131] The core functions of the system are realized by integrating the Resource Allocation Optimization Model (RAOM), Dynamic Task Scheduling Algorithm (DTSA), real-time monitoring and notification mechanisms to achieve the goal of optimizing simulation task management. RAOM is responsible for macro-level resource allocation decisions, while DTSA handles the specific task execution order and parallelism to improve efficiency and reduce waiting time. The real-time monitoring function ensures transparency of task execution and resource usage, while the notification function sends updates to users and administrators when key events occur, ensuring timely feedback and adjustments. The operations involved include but are not limited to:
[0132] A. Requirements Analysis and Resource Assessment
[0133] Analyze simulation task requirements and evaluate current system resource status, including available CPU, memory, and storage resources.
[0134] B. Execute RAOM
[0135] Based on the demand and resource evaluation results, run the RAOM algorithm to determine the optimal resource allocation solution:
[0136]
[0137] Where:
[0138] n represents the number of tasks;
[0139] represents the weight of task i, reflecting its priority;
[0140] represents the estimated execution time of task i;
[0141] The amount of resources actually allocated to task i;
[0142] The optimization goal is to minimize the sum of the ratios of weighted task execution time and resource quantity to achieve optimal resource allocation.
[0143] C. Application of DTSA
[0144] Based on the resource allocation results of RAOM, the DTSA algorithm adjusts the execution order and resource allocation of tasks to maximize resource utilization efficiency and reduce task execution time.
[0145] Specifically, DTSA is used to determine the optimal execution order of tasks:
[0146]
[0147] Where:
[0148] Indicates the order in which tasks are executed;
[0149] Indicates the priority of the i-th task in the arrangement;
[0150] Indicates the waiting time of the task in the current queue;
[0151] The optimization goal is to find a task execution order that minimizes the waiting time considering the priority weights.
[0152] D. Simulation execution and real-time monitoring
[0153] The tasks are executed according to the scheduling plan, and TPMS starts real-time monitoring to track task progress and resource usage.
[0154] E. Performance Analysis and Optimization Suggestions
[0155] After the task is completed, performance analysis is performed based on the monitoring data to identify bottlenecks and inefficient links and generate optimization suggestions.
[0156] G. Result Feedback and Notification
[0157] The simulation results, performance analysis reports and optimization suggestions are fed back to users through email, SMS or internal system notifications.
[0158] Furthermore, the working principle and scope of the user portal and the rights management center are as follows:
[0159] The graphical user interface allows users to submit simulation tasks, configure simulation parameters, and view progress and results in the most direct way; fine-grained permission control strategies are used to ensure data security and user privacy, including user authentication, role definition, and access control lists (ACLs), to ensure that each user operates within the scope of their authorization, effectively preventing unauthorized data access and potential leakage risks. The operations involved include but are not limited to:
[0160] A. User Login and Authentication
[0161] Users log in to the system through the user portal and the rights management center. The system verifies the user's identity through an authentication mechanism to ensure secure access.
[0162] B. Role definition and authority allocation
[0163] System administrators define different user roles, each with a set of predefined permissions that determine the scope of resources that users can access and operate. Based on the user's work requirements, administrators assign users to appropriate roles, and users inherit the permissions of the role, thus completing the permission setting.
[0164] C. Simulation Task Management
[0165] Users can create new simulation tasks according to their own permissions, including filling in task descriptions, setting simulation parameters, and uploading necessary files. Users can view the status of tasks they submit, including progress, estimated completion time, and result summary.
[0166] D. Data access and management
[0167] Users access simulation data based on assigned permissions. The system uses access control lists (ACLs) to ensure that users can only access authorized data and resources. Users can view, download, and modify data based on their permissions. All operations will be recorded by the system for auditing and backtracking.
[0168] E. Result Feedback and Notification
[0169] After the simulation is completed, users can view the simulation results, including visual data display and performance reports. For key task status changes or completions, the system will automatically send notifications to users via email, SMS or internal system notifications to ensure that users can obtain simulation results in a timely manner.
[0170] F. Security and Auditing
[0171] The system records the operation history of all users, including login information, task operations, and data access, to support security audits and problem diagnosis. The system conducts regular security checks and updates to respond to new security threats and vulnerabilities and ensure the overall security of the system.
[0172] Beneficial effects of the present invention:
[0173] 1. Improve the efficiency and quality of the simulation process: By integrating advanced technologies such as edge computing, cloud computing, and blockchain technology, the present invention greatly improves the processing speed of simulation data and the overall efficiency of the simulation process. At the same time, through precise data management and optimization strategies, the accuracy and reliability of the simulation results are improved.
[0174] 2. Realize secure sharing of simulation data: The data sharing network established using blockchain technology ensures the security and immutability of simulation data, and provides a solid data security guarantee for cross-institutional or cross-field simulation project cooperation.
[0175] 3. Intelligent retrieval and analysis support decision-making: Through advanced retrieval and analysis engines, this system can intelligently retrieve simulation data based on rich metadata, and reveal the patterns and trends behind the data through data mining, providing powerful data support for simulation design and optimization.
[0176] 4. Optimize resource allocation and task scheduling: By adopting the resource allocation optimization model (RAOM) and the dynamic task scheduling algorithm (DTSA), the system can achieve optimal resource allocation and efficient scheduling of task execution, significantly reducing the waiting time and execution cycle of simulation tasks.
[0177] 5. Enhanced user experience and system security: Through the fine-grained permission control strategy and user-friendly interface design of the user portal and permission management center, data access security is guaranteed while improving user experience.
[0178] 6. Dynamic simulation parameter optimization: Through simulation design and optimization tools, combined with the analysis results and optimization suggestions of the cloud data processing center, the system can automatically or semi-automatically adjust simulation parameters, optimize simulation design, and improve the accuracy and effectiveness of simulation.
[0179] 7. Data immediacy and validity: The preliminary data processing performed by the edge data processing unit (EDPU) at the source of data generation ensures the immediacy and validity of the data, reduces the burden on the cloud data processing center, and reduces the delay and cost of data transmission. BRIEF DESCRIPTION OF THE DRAWINGS
[0180] Figure 1 The system architecture diagram of the present invention is shown;
[0181] Figure 2 A workflow diagram of the present invention is shown. DETAILED DESCRIPTION
[0182] Exemplary embodiments of the present invention are described in detail below with reference to the accompanying drawings.
[0183] Combination Figure 1 The present invention provides a data management system for a simulation process, and the system composition is as follows:
[0184] Edge Data Processing Unit (EDPU): Located at the forefront of the system, it is responsible for preliminary data processing at the source of data generation, including data collection, preprocessing and preliminary analysis. By performing these operations on the edge side, it can effectively reduce the burden on the cloud data processing center, reduce the delay and cost of data transmission, and ensure the timeliness and feasibility of data.
[0185] Cloud Data Processing Center (CDPC): As the core of the system, it is responsible for receiving and storing processed data from the edge data processing unit, conducting in-depth analysis, storing simulation results, and generating optimization strategies; processing and analyzing large-scale simulation data, thereby providing a scientific basis for simulation design and optimization.
[0186] Blockchain Data Sharing Network (BDSN): Utilizes blockchain technology to achieve secure sharing of simulation data and manages data sharing rules and permissions through smart contracts; this mechanism ensures the immutability, security and traceability of data, and is particularly suitable for cross-institutional or cross-field simulation project collaboration.
[0187] Data Catalog and Metadata Management System (DCMS): responsible for managing simulation data and its metadata, supporting data classification, retrieval and version control; by effectively managing metadata, it improves the discoverability and availability of data, and facilitates the execution of simulation tasks and the reuse of simulation results.
[0188] Advanced Search and Analysis Engine (ASAE): It implements advanced search functions based on metadata in the data catalog and metadata management system, and performs data mining tasks to provide in-depth data support for simulation design and optimization; it intelligently analyzes simulation data to reveal patterns and trends behind the data and assist in decision making.
[0189] Simulation Design and Optimization Tool (SDOT): provides a user-friendly interface that supports the design, editing, and optimization of simulation models. Based on the analysis results and optimization suggestions provided by the cloud data processing center, it can automatically or semi-automatically adjust simulation parameters to improve the accuracy and efficiency of simulation.
[0190] Task and Progress Monitoring Center (TPMS): manages the entire life cycle of simulation tasks, from task submission to execution, monitoring and notification; realizes dynamic scheduling and real-time monitoring of simulation tasks, optimizes the task management process, and improves the efficiency and response speed of task execution.
[0191] User Portal and Permission Management Center (UPPM): Provides the system's user interface and implements the management of user permissions and roles; improves user experience and system security through a role-based dynamic permission allocation mechanism.
[0192] Combination Figure 2 , the workflow of the present invention is as follows:
[0193] S1. Data collection and preprocessing:
[0194] The system collects real-time environmental data, user behavior data, etc. through a network of sensors installed at key locations. For example, in a smart city traffic flow optimization project, sensors collect information such as vehicle volume and speed; in an agricultural irrigation system, they collect soil moisture and meteorological data.
[0195] Apply dynamic outlier identification algorithms to process data. For example, long short-term memory networks (LSTMs) are used to analyze time series data, identify and remove abnormal data points, and adaptive polynomial interpolation methods are used to fill missing values to ensure data integrity and consistency.
[0196] S2. Data upload:
[0197] After preliminary processing, the data is securely uploaded to the cloud data processing center (CDPC) through encryption protocols (such as TLS). Encryption technology and hash algorithms (such as SHA-256) are used to ensure the security of data during transmission.
[0198] S3. Cloud data processing and analysis:
[0199] In the cloud, high-performance computing resources are used to perform in-depth data analysis. Machine learning algorithms such as support vector machines (SVMs) and deep learning are used to further analyze the data and extract valuable information to provide a basis for simulation model design.
[0200] During the analysis process, the system dynamically adjusts computing resources according to project requirements to ensure efficient execution of analysis tasks. For example, during peak demand periods, the system automatically adds computing resources to shorten analysis time.
[0201] S4. Simulation model design and implementation:
[0202] Based on the cloud analysis results, the project team used the Simulation Design and Optimization Tool (SDOT) to build a simulation model. The model design takes into account various practical constraints and goals to ensure the practicality and accuracy of the simulation results.
[0203] When executing the simulation model, the system automatically allocates computing resources according to the model requirements. During the simulation execution process, the running status of the simulation task is dynamically monitored, and resource allocation is adjusted in time to ensure the efficient completion of the simulation task.
[0204] S5. Result evaluation and optimization:
[0205] After the simulation is completed, the system automatically summarizes the simulation results and displays them to the user through visualization tools. The project team evaluates the simulation results and identifies the optimal solution and its potential room for improvement.
[0206] The resource allocation optimization model (RAOM) is used to analyze resource usage during the simulation process, optimize resource allocation strategy, reduce resource waste, and improve simulation efficiency.
[0207] S6. Parameter adjustment and iteration:
[0208] Based on the evaluation results, the project team and the system adjust the simulation parameters and optimize the simulation model according to the dynamic task scheduling algorithm (DTSA). Through iterative simulation, the optimal solution is gradually approached until the predetermined optimization goal is met or the upper limit of the number of iterations is reached.
[0209] S7. Data Sharing and Access Control:
[0210] Through fine-grained permission management and blockchain technology, simulation results can be shared securely. Smart contracts are used to define data access rules to ensure that only authorized users can access specific simulation results and analysis reports. All access behaviors are recorded to improve the transparency and traceability of data sharing.
[0211] In order to further illustrate the present invention, two specific embodiments will be described in detail below:
[0212] Example 1: Simulation Analysis of Smart City Traffic Flow Optimization
[0213] Simulation implementation background:
[0214] This simulation project is applied to the optimization of smart city traffic flow, with the goal of improving the traffic congestion in the city center during peak hours in the morning and evening. The specific scenario covers the city center, and the key data include:
[0215] Daily traffic volume is about 10,000 vehicles;
[0216] Average driving speed: 30 km / h (morning peak), 50 km / h (evening peak);
[0217] Average parking time: 3 minutes (morning rush hour), 2 minutes (evening rush hour).
[0218] Simulation implementation process:
[0219] S1. Data collection and preprocessing:
[0220] The urban traffic monitoring system is used to collect real-time data on vehicle flow, vehicle speed, and parking time. The data is processed using a dynamic outlier identification method based on a long short-term memory network (LSTM) to remove outliers, and the missing values are filled using a polynomial interpolation method.
[0221] S2. Data upload:
[0222] The pre-processed data is securely uploaded to the cloud data processing center (CDPC) through an encryption protocol, and the data is hashed using the SHA-256 algorithm to ensure transmission security.
[0223] S3. Cloud data processing and analysis:
[0224] In CDPC, cluster analysis (K-means algorithm) is applied to identify high congestion periods and areas based on traffic volume and parking time, and big data technology is used to predict traffic flow trends in the next week to provide a basis for the simulation model.
[0225] S4. Simulation model design and implementation:
[0226] Use the Simulation Design and Optimization Tool (SDOT) to build a traffic flow simulation model, simulate different traffic control strategies, such as adjusting the signal light cycle, conduct multiple rounds of simulation experiments, and record the results for analysis.
[0227] S5. Result evaluation and optimization:
[0228] The results of different simulation strategies are analyzed to evaluate their effects on increasing average driving speed and reducing parking time, and the optimal strategy is selected to reduce traffic congestion time by more than 10%.
[0229] S6. Parameter adjustment and iteration:
[0230] According to the feedback from the simulation results, the simulation parameters are adjusted for iterative optimization. The signal light cycle parameters are fine-tuned using the gradient descent method until the expected traffic flow optimization effect is achieved or the iteration limit is reached.
[0231] S7. Data Sharing and Access Control:
[0232] The optimized simulation results and traffic control strategies are securely shared with urban planning and traffic management departments through the constructed blockchain network. Smart contracts manage data sharing rules and access rights to ensure that only authorized personnel have access.
[0233] This embodiment integrates edge computing and blockchain technology to not only realize the instant processing and secure sharing of traffic data, improve processing efficiency and security, but also ensure the accuracy and integrity of the data, providing reliable input for the simulation model. Through in-depth analysis of traffic flow through cluster analysis and big data technology, and simulation experiments using the Simulation Design and Optimization Tool (SDOT), this embodiment effectively improves simulation accuracy and efficiency, and supports scientific decision-making in urban traffic management. Finally, with the help of the Advanced Data Retrieval and Analysis Engine (ASAE) and the Task and Progress Monitoring Center (TPMS), in-depth analysis and real-time monitoring of simulation results are achieved, enhancing project management and scientific decision-making.
[0234] Example 2: Simulation Optimization of Agricultural Irrigation System
[0235] Simulation implementation background:
[0236] This example is applied to a 100-hectare farm, with the goal of optimizing the irrigation system through intelligent analysis technology to adapt to the water requirements of crops at different growth stages and changing environmental conditions. Key monitoring data include soil moisture (40%-60% updated every hour), crop growth stage (sowing period, growth period, maturity period), daily evaporation (5mm to 15mm) and precipitation (0mm to 20mm, seasonal changes).
[0237] Simulation implementation process:
[0238] S1. Data collection and preprocessing:
[0239] Key environmental data is collected using soil moisture sensors and weather stations. Abnormal soil moisture readings are dynamically identified and removed through long short-term memory networks (LSTM) to ensure data accuracy. For missing data, polynomial interpolation is used for adaptive filling to maintain the integrity of the data set.
[0240] S2. Data upload:
[0241] The preprocessed data is securely uploaded to the cloud data processing center (CDPC) through the TLS encryption protocol and hashed using the SHA-256 algorithm to ensure data security and integrity.
[0242] S3. Cloud data processing and analysis:
[0243] In CDPC, support vector machines (SVM) are used to analyze the relationship between soil moisture and crop growth stages to predict irrigation needs. Meteorological data (rainfall and evaporation) are combined to optimize irrigation strategies and provide scientific irrigation recommendations.
[0244] S4. Irrigation model design and implementation:
[0245] Based on the cloud analysis results, the resource allocation optimization model (RAOM) and dynamic task scheduling algorithm (DTSA) are used to calculate the optimal irrigation quota and adjust the order and time of irrigation task execution. The irrigation system automatically adjusts the working parameters based on these outputs and executes a customized irrigation plan.
[0246] S5. Result evaluation and optimization:
[0247] By monitoring crop growth and soil moisture changes, the effectiveness of irrigation strategies can be evaluated. Feedback data can be used to adjust forecast models and continuously optimize irrigation strategies to cope with climate change and water resource fluctuations.
[0248] S6. Parameter adjustment and iteration:
[0249] During the actual operation and monitoring of the irrigation system, the system uses a dynamic task scheduling algorithm (DTSA) to make detailed parameter adjustments to the irrigation plan based on real-time feedback on crop growth, soil moisture changes, and climate conditions. This includes optimizing irrigation time, adjusting irrigation volume, and changing irrigation frequency to ensure that each irrigation can meet the actual needs of crops while taking into account the efficient use of water resources.
[0250] Resource Allocation Optimization Model (RAOM) is used to re-evaluate and allocate existing resources, especially in water-stressed or multi-crop situations. The optimization model can ensure that high-priority crops receive adequate irrigation while maximizing water utilization across the farm.
[0251] The system will continuously iterate and optimize the irrigation strategy based on the evaluation of the results after each round of irrigation, combined with the in-depth analysis and suggestions of the cloud data processing center. This process is cyclical to adapt to various changes in the crop growth cycle and unpredictable environmental factors, ensuring that the irrigation strategy always remains in the optimal state.
[0252] S7. Secure data sharing and decision support:
[0253] The simulation data and irrigation strategies are securely shared through blockchain technology, ensuring the data is tamper-proof and transparent. Smart contracts ensure that only authorized users (such as farm managers and decision makers) can access relevant data.
[0254] Enable decision makers to formulate or adjust irrigation plans based on real-time and accurate data, optimize resource allocation, and support the sustainable development of agricultural production.
[0255] By comprehensively using RAOM and DTSA, this embodiment demonstrates an advanced agricultural irrigation system simulation optimization solution. The system can not only provide accurate irrigation strategies based on intelligent analysis, but also dynamically adjust irrigation plans to adapt to environmental changes and changes in crop water requirements. Through the intelligent and automated irrigation process, the utilization efficiency of water resources is significantly improved, while ensuring the healthy growth of crops, reflecting the application value and technological innovation of the invention in the field of agricultural irrigation.
[0256] In addition, the system's secure data sharing mechanism and advanced data retrieval capabilities provide farm managers and decision makers with powerful support tools to help them better understand and manage farm irrigation needs and support efficient and sustainable agricultural production.
[0257] In summary, the present invention proposes an efficient simulation process data management system, which aims to achieve efficient processing, secure sharing, intelligent retrieval and analysis, dynamic task scheduling and fine-grained permission control of simulation data through the comprehensive application of advanced technologies such as edge computing, cloud computing, blockchain technology, data directory and metadata management, advanced data retrieval and analysis, simulation design and optimization tools, task and progress monitoring, and user portal and permission management. This system solution not only improves the overall efficiency and security of the simulation process, but also provides a solid technical foundation for the efficient execution of complex simulation tasks and the effective use of simulation data.
[0258] It is worth noting that although this document shows the specific application of the present invention through specific embodiments, these embodiments are only for illustrative purposes and are not intended to limit the scope of protection of the present invention. The scope of protection of the present invention is defined by the claims, and any appropriate modification, equivalent substitution or improvement based on the principles of the present invention should be deemed to fall within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be given the broadest interpretation to cover all such modifications, equivalent structures and functions.
Claims
1. A data management system for a simulation process, the features of which include: Edge Data Processing Unit (EDPU): Located at the forefront of the system, it is responsible for preliminary data processing at the source of data generation, including data collection, preprocessing, and preliminary analysis. Cloud Data Processing Center (CDPC): As the core of the system, it is responsible for receiving and storing processed data from edge data processing units, conducting in-depth analysis, storing simulation results, and generating optimization strategies; it processes and analyzes large-scale simulation data, thereby providing a scientific basis for simulation design and optimization; Blockchain Data Sharing Network BDSN: Use blockchain technology to achieve secure sharing of simulation data, manage data sharing rules and permissions through smart contracts, and ensure the immutability, security and traceability of data; Data Catalog and Metadata Management System DCMS: responsible for managing simulation data and its metadata, supporting data classification, retrieval and version control; by effectively managing metadata, it improves the discoverability and availability of data, and facilitates the execution of simulation tasks and the reuse of simulation results; Advanced Search and Analysis Engine (ASAE): It implements advanced search functions based on metadata in the data catalog and metadata management system, and performs data mining tasks to provide in-depth data support for simulation design and optimization. Through intelligent analysis of simulation data, the patterns and trends behind the data are revealed to assist decision making; Simulation Design and Optimization Tool SDOT: provides a user-friendly interface to support the design, editing and optimization of simulation models. Based on the analysis results and optimization suggestions provided by the cloud data processing center, it can automatically or semi-automatically adjust simulation parameters to improve the accuracy and efficiency of simulation. Task and Progress Monitoring Center TPMS: manages the entire life cycle of simulation tasks, from task submission to execution, monitoring and notification, realizes dynamic scheduling and real-time monitoring of simulation tasks, optimizes the task management process, and improves the efficiency and response speed of task execution; User Portal and Rights Management Center UPPM: provides the system user interface, implements the management of user rights and roles, and improves the user experience and system security through a role-based dynamic rights allocation mechanism; The edge data processing unit EDPU specifically includes: A. Data collection: Collect data from connected sensors and devices in real time; B. Data cleaning: Adaptive data cleaning algorithms are used to identify outliers and clean data, and a dynamic outlier identification method based on the long short-term memory network (LSTM) is used. LSTM can capture long-term dependencies in time series data, learn the normal patterns of historical data, and dynamically predict the expected value and its confidence interval at each time point, thereby achieving dynamic identification of outliers. Data cleaning specifically includes: Data preprocessing: standardizing the original time series data to eliminate the impact of dimensions and reduce the impact of extreme values; LSTM model training: Use the preprocessed historical time series data as model input, predict the value at the next time point, and minimize the difference between the predicted value and the actual value as the model optimization goal; Outlier identification: For each time point t, use the trained LSTM model to predict its value And the confidence interval, if the actual observation value X t If it is not within the predicted confidence interval, then X t It is an outlier and needs to be removed; Confidence interval calculation: using the standard deviation σ of the model prediction t and a preset confidence level Z, calculate the confidence interval [L t ,U t ], where the left endpoint of the confidence interval Right endpoint of the confidence interval Parameter adjustment: adjust the structure and training parameters of the LSTM model according to the accuracy and recall rate of outlier detection in actual applications; C. Missing value processing: Time series data: If the data shows a clear linear trend, use linear interpolation; if the data shows a nonlinear trend, use polynomial interpolation; Spatial data: Apply the K nearest neighbor (KNN) algorithm to fill missing values based on the values of adjacent data points, with weights adjusted based on distance; D. Data upload: The processed data is securely uploaded to the cloud data processing center for processing; The cloud data processing center CDPC specifically includes: Receive processed data from the edge data processing unit, perform further in-depth analysis, store simulation results, and generate simulation optimization strategies based on the analysis results; A1. Receive data: Receive processed data sent by the edge data processing unit in real time through a high-speed network connection; B1. Data storage: In order to cope with the large-scale simulation data storage requirements from different edge data processing units, the cloud data processing center adopts a distributed database system; C1. Data analysis: Use big data processing technology to achieve distributed processing of stored data; D1.Simulation optimization: The objective function J(X) is defined as the square of the difference between the expected value of the simulation performance index and the simulation experimental value, that is, the sum of squared errors SSE. The goal is to adjust the simulation input parameter X to minimize the objective function J(X): Where: Y expected,i is the expected value of the performance index in the ith simulation experiment; Y sim,i is the simulated value of the performance index when the input parameter X is given in the ith simulation experiment; n is the total number of simulation experiments; In order to find the parameter X that minimizes the objective function J(X), the gradient descent method is used to optimize the objective: Where: X new , X old They are the values before and after the parameter update; γ is the learning rate, which controls the step size of the parameter update; is the objective function J with respect to X in X old The gradient at indicates the direction in which J grows fastest; In each iteration, the gradient of the current parameters is calculated Then update the parameters according to the gradient direction until J(X) converges to the minimum value or reaches the preset number of iterations; E1. Generation of parameter adjustment suggestions: By analyzing the current simulation data and historical simulation results, the influence of each parameter on the simulation results is analyzed, and optimization suggestions for simulation parameters are automatically put forward to optimize the subsequent simulation process; The simulation design and optimization tool SDOT specifically includes: Combining the functions of simulation modeling, parameter optimization and result analysis, it supports users to complete the full life cycle management of simulation models in a single environment. Through deep integration with the cloud data processing center, it automatically receives the analysis results and optimization suggestions of simulation data, making the simulation process more intelligent and automated. A2. Model design: Users use graphical tools to build simulation models and define preliminary simulation parameters; B2. Simulation execution: Execute simulation tasks locally or in the cloud and collect simulation results; C2. Send the simulation results to the cloud data processing center for in-depth analysis to obtain performance indicators and potential optimization directions; D2. Parameter optimization: Automatically or manually adjust simulation parameters based on feedback and optimization suggestions from the cloud data processing center; The performance of the simulation model is quantified by the objective function J(X), where X = {x1, x2, ... x n } is the parameter vector of the simulation model. The goal is to find a set of parameters X* that minimizes the objective function J(X): X * =argminxJ(x) Where argmin x It means finding the parameter set X that minimizes J(x), and then optimizing the target through gradient descent or genetic algorithm; E2. Optimization iteration: Repeat the simulation until the performance target is met or the preset number of iterations is reached; F2. Result feedback and application: Display the final simulation results, support the export and visualization of the results, and adjust the simulation model design according to the optimization results; The task and progress monitoring center TPMS specifically includes: applying the resource allocation optimization model RAOM, the dynamic task scheduling algorithm DTSA, and the real-time monitoring and notification mechanism to realize its core functions, so as to achieve the goal of optimizing simulation task management. RAOM is responsible for macro-level resource allocation decisions, while DTSA handles the specific task execution order and parallelism to improve efficiency and reduce waiting time; A3. Requirements Analysis and Resource Assessment: Analyze simulation task requirements and assess current system resource status, including available CPU, memory, and storage resources; B3. Execute RAOM: Based on the demand and resource evaluation results, run the RAOM algorithm to determine the optimal resource allocation plan: Where: n represents the number of tasks; ωi represents the weight of task i, reflecting its priority; T i represents the estimated execution time of task i; Ra i The amount of resources actually allocated to task i; The optimization goal is to minimize the sum of the weighted task execution time and resource quantity ratio to achieve optimal resource allocation; C3. Apply DTSA: Based on the resource allocation results of RAOM, the DTSA algorithm adjusts the execution order and resource allocation of tasks to maximize resource utilization efficiency and reduce task execution time: Where: σ represents the order of task execution; P(σ i ) represents the priority of the i-th task in the arrangement; T ω (σ i ) indicates the waiting time of the task in the current queue; The optimization goal is to find a task execution order that minimizes the waiting time considering the priority weights; D3. Simulation execution and real-time monitoring: The task is executed according to the scheduling plan, and the TPMS starts real-time monitoring to track the task progress and resource usage; E3. Performance analysis and optimization suggestions: After the task is completed, performance analysis is performed based on the monitoring data to identify bottlenecks and inefficient links and generate optimization suggestions; F3. Result feedback and notification: The simulation results, performance analysis reports and optimization suggestions will be fed back to the user via email, SMS or internal system notification.
2. The data management system for the simulation process according to claim 1, wherein the blockchain data sharing network BDSN specifically includes: The distributed ledger based on blockchain technology ensures the security, transparency and immutability of simulation data. Each data transaction is recorded as a block and connected into a chain through encryption algorithms to form a traceable data operation history. A4. Data upload and encryption: Data providers upload simulation data to the blockchain data sharing network and encrypt it using the public key of the data requester. The encrypted simulation data is uploaded to the blockchain network along with the transaction information; the transaction information includes the data metadata, upload timestamp, and the identifiers of the data provider and the expected data requester. B4. Block creation: The system packages multiple data upload transactions collected within a certain period of time to create a new block. In addition to multiple data upload transactions, each block also contains a hash value pointing to the previous block to ensure the continuity of the blockchain. The newly created block is hashed to generate a unique identifier for the block, namely the hash value. The hash calculation includes encryption of the block content to ensure that the block content cannot be tampered with. C4. Consensus mechanism execution: After receiving a new block, the nodes in the network begin to verify the validity of the block. The verification process includes checking the validity of the block hash value, the legitimacy of the transaction, and the consistency of the blockchain. A specific consensus algorithm is used to ensure that most nodes in the network agree to the addition of the new block. Once a consensus is reached, the new block is added to the blockchain and becomes part of the consensus of the entire network. D4. Smart contract management authority: define and deploy smart contracts on the blockchain. Smart contracts contain the rules and logic for data access. When a data requester initiates a data access request, the smart contract automatically executes to verify the requester's identity and access rights. Only requests that pass the verification are allowed to access the corresponding data. E4. Data access and decryption: After verification by the smart contract, the data requester is authorized to access specific encrypted data; the data requester uses his own private key to decrypt the encrypted data; The decrypted data can be used for subsequent simulation analysis and processing by the requester; the smart contract also records the access and use of the data, providing traceability and transparency for the use of the data.
3. According to the data management system of the simulation process described in claim 2, the data directory and metadata management system DCMS and the advanced search and analysis engine ASAE work together to realize the advanced search function and data mining tasks of simulation data based on metadata.
4. The data management system for the simulation process according to claim 1, wherein the user portal and the permission management system UPPM provides fine-grained permission control strategies, including user authentication, role definition and access control list ACL, to ensure data security and user privacy.
Citation Information
Patent Citations
Nonlinear adaptive closed-loop control system for blockchain Internet of Things management
CN111193630A
Vehicle-mounted network application data transmission and charging optimization method based on mobile edge computing and block chain
CN113344255A
Networked vehicle edge computing and video analysis resource allocation method based on blockchain
CN113795012A
Simulation test platform based on block chain
CN116319387A
Production logistics management platform based on block chain and digital twinning and implementation method
CN117172641A
