Big data simulation method, simulation platform and device
By constructing an operator component library and a graphical interface, and combining product ID data association and parallel computing, the problem of big data simulation computing in existing technologies has been solved, achieving efficient simulation and ease of use across multiple data sources, and improving simulation processing efficiency and result accuracy.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- SHANGHAI SHUKEYI TECHNOLOGY CO LTD
- Filing Date
- 2026-02-09
- Publication Date
- 2026-05-08
AI Technical Summary
Existing industrial simulation software cannot directly connect to large databases to perform long-term big data simulation calculations on a large number of individual products, and big data computing platforms lack physical simulation model integration and multi-science solution scheduling support, making it difficult to meet the business needs of enterprises.
A reusable operator component library is built, a simulation task flow is constructed through a graphical interface, multiple data sources are associated and loaded and output uniformly based on product ID, parallel task scheduling and distributed computing execution are implemented, and the calculation results are aggregated and post-processed.
It enables the association loading and output of data from multiple data sources, improves simulation processing efficiency, supports the efficient execution of large-scale simulation tasks, and enhances ease of use and accuracy of simulation results.
Smart Images
Figure CN121997612A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of simulation computing technology, and in particular to a big data simulation method, simulation platform and device. Background Technology
[0002] In fields such as intelligent manufacturing and digital twins, industrial simulation software is a core tool for product design and performance evaluation. However, current industrial simulation software can only support simulation calculations based on a segment of running data for a single product. It cannot directly connect to large databases to simultaneously execute large-scale big data simulation calculations for a large number of individual products with long running times.
[0003] Existing big data computing platforms are mostly used for tasks such as data statistical analysis and machine learning, and lack support for services such as physical simulation model integration and multi-science solution scheduling, making it difficult to meet business needs.
[0004] Product big data encompasses two categories: product operational big data, which includes the large volume of operational signal responses of products, such as voltage and current signals; and product characteristic big data, which includes the large volume of product feature parameters, such as diameter, hardness, and preload. In enterprises, product big data is typically stored in multiple database tables. Therefore, tools for performing big data simulation must first be able to filter or sample data from multiple sources to obtain subsets, and then perform data association matching, unified modeling, and simulation calculations based on the product IDs of the data within these subsets. Therefore, there is an urgent need for a big data simulation method, platform, and device that can solve the above problems and is easy for users to operate. Summary of the Invention
[0005] To achieve the above-mentioned objectives, the technical solution adopted by this invention is as follows:
[0006] A big data simulation method includes the following steps: Step 1: Construct a reusable operator component library, including a data loading operator component for loading data, a data generation operator component for generating data, a data matching operator component for associating data from multiple data sources, a data calculation operator component for performing data calculations, a result storage operator component for storing results, and a result display operator component for displaying results in charts;
[0007] Step 2: Construct the simulation task flow through a graphical interface. The graphical interface receives drag-and-drop and connection operations of multiple operators in the operator component library from the user to construct the simulation task flow. The connection operation is used to define the data flow between operators.
[0008] Step 3: Based on product ID, realize product-individualized data association of multiple data sources, edit the simulation task flow, and load or generate multiple data operators. Each operator is an independent data source. Establish product ID matching rules between data sources through data matching operators to realize the association loading and unified output of data from multiple data sources.
[0009] Step 4: Parallel task scheduling and distributed computing execution. According to the preset parallel computing strategy, the simulation task process is divided into multiple sub-tasks, and the sub-tasks are distributed to multiple computing nodes for computing.
[0010] Step 5: Aggregate the calculation results and perform post-processing. Collect and process the results of each calculation node and perform post-processing.
[0011] As an improvement, the data loading operator component in step 1 includes: a product operation big data operator for loading signal data and parameter data from the database, a product characteristic big data operator, and a local data upload operator for reading local data.
[0012] As an improvement, the data loading operator component in step 1 can filter or sample the data; the filtering or sampling content includes product ID, signal, parameter and data date.
[0013] As an improvement, the data generation operator component in step 1 supports customizing the number of product IDs, product parameter names, and the statistical distribution type and basic parameter values for each parameter. The system first generates a value equal to the number of product IDs for each product parameter based on the statistical distribution type and corresponding basic parameter values; then, according to preset combination rules, it associates these parameter values with the product IDs to generate the final dataset.
[0014] As an improvement, the preset combination rules include: random matching between parameters, matching by size order, and matching of specified data pairs; the statistical distribution types include: normal distribution, log-normal distribution, Weibull distribution, exponential distribution, and uniform distribution.
[0015] As an improvement, the product ID matching rules established by the data matching operator in step 3 between various data sources include: consistent product ID matching, random matching between product IDs, sequential matching between product IDs, or user-specified product ID pair matching.
[0016] As an improvement, the parallel computing strategy in step 4 includes: splitting by product ID, splitting by time period of product running data, or splitting by operators in the simulation task flow.
[0017] This application also provides a big data simulation platform, including:
[0018] The data loading module is used to filter and sample product IDs and product characteristic big data associated with product IDs in the database or local storage to generate data loading operators.
[0019] The data generation module is used to define the number of product IDs, product parameter names, and the statistical distribution type and basic parameter values of each parameter in order to generate data generation operators.
[0020] The model development module is used to upload computational models and data processing models, and to write programs in various languages to form data computation operators.
[0021] The integrated modeling module provides a graphical interface for connecting, arranging, and associating data IDs between data sources with data loading operators, data generation operators, data calculation operators, and data matching operators to construct the simulation task flow.
[0022] The computation settings module is used to configure computational resources and set parallel computing methods for simulation task processes.
[0023] Post-processing module: Used for filtering, statistical analysis and visualization of simulation results.
[0024] As an improvement, the computing resources configured in the computing settings module include a central processing unit, a graphics processing unit, and memory, and the parallel computing methods include parallelism by product ID, parallelism by time period, or parallelism by operator, and combinations thereof.
[0025] This application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the above method steps.
[0026] The beneficial effects of this invention are as follows: by setting up a graphical, drag-and-drop user interface, complex data loading, data generation, data ID matching, data processing and calculation, data storage, data display, and process control can be encapsulated into standardized operator components, which improves the overall ease of use and makes it convenient for different groups of people to use.
[0027] By generating a series of individual products with different parameter values through data generation operators, the impact of manufacturing variance in product parts can be simulated; by using preset combination rules between parameters, random and regular assembly between product parts can be simulated.
[0028] By setting data matching specifications based on product ID, it is possible to load and output data from multiple data sources, enabling multiple differentiated simulations of products with different parameters and signals in a single simulation task. This can realistically reflect the impact of the manufacturing and assembly process and differentiated use on product performance and function.
[0029] By setting up parallel computing strategies for simulation tasks, large-scale simulation tasks can be broken down and scheduled to distributed computing resources, thereby improving the overall simulation processing efficiency.
[0030] By setting up a simulation platform, industrial simulation and big data parallel computing architecture can be fully integrated, achieving a high degree of synergy between data, models, and computing power, which facilitates the import and computation of different data. Attached Figure Description
[0031] Figure 1 This is a schematic diagram of a big data simulation method according to the present invention. Detailed Implementation
[0032] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of the embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.
[0033] It should be noted that the terms "first" and "second" in this application are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this application, "multiple" means at least two, such as two, three, etc., unless otherwise explicitly specified. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or apparatuses.
[0034] like Figure 1As shown, a big data simulation method includes the following steps: Step 1: Construct a reusable operator component library, including a data loading operator component for loading data, a data generation operator component for generating data, a data matching operator component for associating data from multiple data sources, a data calculation operator component for performing data calculations, a result storage operator component for storing results, and a result display operator component for displaying results in charts; Step 2: Receive drag-and-drop and connection operations from the user on multiple operators in the operator component library through a graphical interface to construct a simulation task flow, wherein the connection operation is used to define the data flow direction between operators; Step 3: Edit the simulation task flow, for multiple data loading operators or data generation operators, each operator is an independent data source, and product ID matching rules are established between data sources through data matching operators to achieve associative loading and unified output of multiple data sources; Step 4: According to a preset parallel computing strategy, the simulation task flow is divided into multiple subtasks, and the subtasks are distributed to multiple computing nodes for calculation; Step 5: Collect and process the results of each computing node, and perform post-processing. The data loading operator component is used to obtain initial data from heterogeneous data sources, including a product operation big data operator and a product characteristic big data operator. The product operation big data operator extracts time-series signal data bound to the product ID from a database or file, such as voltage, rotation speed, and temperature. The product characteristic big data operator obtains static attribute and characteristic parameter data bound to the product ID from a database or file, such as geometric dimensions, material properties, and processing and assembly parameters.
[0035] The data loading operator component in step 1 includes: a product operation big data operator for loading signal data and parameter data from the database, a product characteristic big data operator, and a local data upload operator for reading local data. The data loading operator component in step 1 can filter or sample data; the filtering or sampling content includes product ID, signal, parameter, and data date. The data generation operator component in step 1 supports customizing the number of product IDs, product parameter names, and the statistical distribution type and basic parameter values for each parameter. The system first generates a value equal to the number of product IDs for each product parameter based on the statistical distribution type and the corresponding basic parameter values; then, according to preset combination rules, it associates these parameter values with the product IDs to generate the final dataset. The preset combination rules include: random matching between parameters, size-ordered matching, and specified data column matching; the statistical distribution types include: normal distribution, log-normal distribution, Weibull distribution, exponential distribution, and uniform distribution. The product ID matching rules established by the data matching operator in step 3 between various data sources include: consistent product ID matching, random matching between product IDs, sequential matching between product IDs, or user-specified product ID pair matching. The parallel computing strategy in step 4 includes: splitting by product ID, splitting by the time period of product running data, or splitting by the operator in the simulation task flow. Before distributing parallel computing, step 4 also includes a compliance check on the constructed simulation task flow. After each computing node completes its computation, it outputs the results to a unified storage system. The post-processing module automatically collects and aggregates the result data from all nodes and processes and analyzes the results according to user needs, such as calculating the statistical distribution of all product performance indicators; filtering out a list of product IDs that meet specific conditions; and generating visualization charts.
[0036] When in use, users can access the platform's graphical modeling interface through a computer terminal, and drag and drop the required operators from the operator library onto the canvas. Then, they can connect the input or output ports of each operator using the mouse to define the data flow and complete a full simulation task.
[0037] A big data simulation platform, comprising:
[0038] The system comprises several modules: a data loading module for filtering and sampling product ID-related big data and product characteristic big data associated with product IDs in a database or local storage to generate data loading operators; a data generation module for defining the number of product IDs, product parameter names, statistical distribution types of each parameter, and basic parameter values for each distribution to generate data generation operators; a model development module for uploading computational models, data processing models, and writing programs in various languages to form data computation operators; an integrated modeling module providing a graphical interface for connecting, arranging, and associating data IDs between data sources using data loading, data generation, data computation, and data matching operators to construct the simulation task flow; a computation setting module for configuring computational resources and setting parallel computation modes for the simulation task flow; and a post-processing module for filtering, statistically analyzing, and visualizing the simulation computation results. The computational resources configured in the computation setting module include a central processing unit (CPU), a graphics processing unit (GPU), and memory. The parallel computation modes include parallelism by product ID, parallelism by time period, parallelism by operator, and combinations thereof. The computational settings configuration allows for manual control of the computational power allocation at each stage, enabling efficient application of computational power.
[0039] An electronic device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the method steps described above.
[0040] Example:
[0041] The specific steps for using a big data simulation method, platform, and device include:
[0042] Step 1: Generate big data operators for product operation in the data loading module;
[0043] Specifically, a number of product IDs are sampled by combining product model category, product component configuration, product performance parameter range, product activation time, and product registration location. For example, 100 products are sampled, and the data of the time period is extracted by defining the running time period. The signal column of the time period is filtered by signal type, and finally the running big data of the signal in the time period of the ID is obtained. The product running big data operator is generated and named LoadProfiles100.
[0044] Step 2: Generate big data operators for product characteristics in the data generation module;
[0045] Specifically, a number of product characteristic parameter values can be defined according to a normal distribution, a certain number of parameter values can be randomly generated, and the parameters can be matched according to a preset combination rule, such as random matching, to generate a parameter characteristic table of 100 corresponding digital product IDs, that is, a product characteristic big data operator, named DM100.
[0046] Step 3: Develop various data calculation operators in the model development module;
[0047] Specifically, computational programs or models can be written or uploaded using languages such as Python and FMU, while defining the input parameters, output parameters, input attributes, input signals, and output signals of operators. This module will automatically parse them into standard input / output operators, completing the development of various data computation operators. For multidisciplinary applications, users can develop mechanistic models or AI proxy models for multiple disciplines such as force, heat, light, sound, electricity, magnetism, and fluid dynamics based on physical equations or various simulation and modeling tools.
[0048] Step 4: In the integrated modeling module, the various operators are integrated in series and parallel. The data matching operator is used to complete the data association between the two data sources through product ID matching, and a task flow from loading data, calculation and arrangement, and result storage is built.
[0049] Specifically, users drag and drop the required product operation big data operator LoadProfiles100, product feature big data operator DM100, data matching operator, various data calculation operators, and result storage operator from the operator component library. In the data matching operator, a random matching method is selected to define data matching based on ID for the product operation big data operator and product feature big data; this is used to integrate the above operators in a connected manner, enabling the construction of the task flow and data storage by product ID. The product ID can originate from either the product operation big data operator or the product feature big data operator.
[0050] Step 5: In the computing settings module, set the computing resource requirements and parallel computing mode;
[0051] Specifically, for example, if the CPU is set to 10 cores and the parallel computing method is set according to product ID, the parsing program will divide the 100 running big data and 100 product feature big data into 10 parts according to product ID, and then send the data and the complete task process to different 10 CPUs for parallel solving. Each CPU gets the result of 10 product IDs, and a total of 100 product IDs are obtained.
[0052] Step 6: In the result post-processing module, post-process the results to obtain key indicators and display the indicators;
[0053] Specifically, the parsing program will process the data according to the post-processing method defined by the user on the front end. For example, if the user chooses to perform result distribution ratio analysis by product ID, the corresponding result distribution interval - product quantity distribution chart will be obtained.
[0054] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.
Claims
1. A big data simulation method, characterized in that, Includes the following steps: Step 1: Build a reusable operator component library, including a data loading operator component for loading data, a data generation operator component for generating data, a data matching operator component for associating data from multiple data sources, a data calculation operator component for performing data calculations, a result storage operator component for storing results, and a result display operator component for charting and displaying results. Step 2: Construct the simulation task flow through a graphical interface. The graphical interface receives drag-and-drop and connection operations of multiple operators in the operator component library from the user to construct the simulation task flow. The connection operation is used to define the data flow between operators. Step 3: Based on product ID, realize product-individualized data association of multiple data sources, edit the simulation task flow, and load or generate multiple data operators. Each operator is an independent data source. Establish product ID matching rules between data sources through data matching operators to realize the association loading and unified output of data from multiple data sources. Step 4: Parallel task scheduling and distributed computing execution. According to the preset parallel computing strategy, the simulation task process is divided into multiple sub-tasks, and the sub-tasks are distributed to multiple computing nodes for computing. Step 5: Aggregate the calculation results and perform post-processing. Collect and process the results of each calculation node and perform post-processing.
2. The big data simulation method according to claim 1, characterized in that, The data loading operator component in step 1 includes: a product operation big data operator and a product characteristic big data operator for loading signal data and parameter data from the database, and a local data upload operator for reading local data.
3. The big data simulation method according to claim 1, characterized in that, In step 1, the data loading operator component can filter or sample the data; the filtering or sampling content includes product ID, signal, parameter and data date.
4. The big data simulation method according to claim 1, characterized in that, In step 1, the data generation operator component supports customizing the number of product IDs, product parameter names, and the statistical distribution type and basic parameter values for each parameter. The system first generates a value equal to the number of product IDs for each product parameter based on the statistical distribution type and corresponding basic parameter values. Then, according to preset combination rules, these parameter values are associated with the product IDs to generate the final dataset.
5. The big data simulation method according to claim 4, characterized in that, The preset combination rules include: random matching between parameters, matching by size order, and matching of specified data pairs; the statistical distribution types include: normal distribution, log-normal distribution, Weibull distribution, exponential distribution, and uniform distribution.
6. The big data simulation method according to claim 1, characterized in that, The product ID matching rules established by the data matching operator in step 3 between various data sources include: consistent product ID matching, random matching between product IDs, sequential matching between product IDs, or user-specified product ID pair matching.
7. The big data simulation method according to claim 1, characterized in that, The parallel computing strategies in step 4 include: splitting by product ID, splitting by time period of product running data, or splitting by operators in the simulation task process, and their combinations.
8. A big data simulation platform, characterized in that, include: The data loading module is used to filter and sample product IDs and product characteristic big data associated with product IDs in the database or local storage to generate data loading operators. The data generation module is used to define the number of product IDs, product parameter names, and the statistical distribution type and basic parameter values of each parameter in order to generate data generation operators. The model development module is used to upload computational models and data processing models, and to write programs in various languages to form data computation operators. The integrated modeling module provides a graphical interface for connecting, arranging, and associating data IDs between data sources with data loading operators, data generation operators, data calculation operators, and data matching operators to construct the simulation task flow. The computation settings module is used to configure computational resources and set parallel computing methods for simulation task processes. Post-processing module: Used for filtering, statistical analysis and visualization of simulation results.
9. A big data simulation platform according to claim 8, characterized in that, The computing resources configured in the computing settings module include a central processing unit, a graphics processing unit, and memory. The parallel computing methods include parallelism by product ID, parallelism by time period, parallelism by operator, and combinations thereof.
10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the big data simulation method as described in any one of claims 1 to 9.