Modeling simulation cloud service data acquisition method and system

Abstract

The invention relates to a modeling simulation cloud service data acquisition method and system, and the method comprises the steps: S1, obtaining the total flow and data acquisition channel number of each resource group, and constructing a dynamically expanded resource pool; s2, obtaining a current distribution request of the simulation system, wherein the current distribution request comprises the required data acquisition channel number and the required flow of each simulation system; and S3, obtaining the current available flow and the available data acquisition channel of each resource group so as to allocate the data acquisition channel for each simulation system in the resource pool according to the allocation request, and returning to the step S2. The problem that in the prior art, the three core requirements of unified expression of multi-source heterogeneous data, security isolation of multi-user data and low-delay acquisition-transmission-distribution cannot be met at the same time under the cloud architecture is solved.

Description

Technical Field

[0001] This invention relates to the field of cloud service technology, and in particular to a method and system for modeling and simulating cloud service data acquisition. Background Technology

[0002] With the rapid development of cloud service technology, the modeling and simulation industry is evolving from the traditional model of single physics field, single user, and offline computing to a new paradigm of "multi-physics field coupling—cloud-edge-device collaboration—intelligent decision-making." Specifically, this manifests in: ① The demand for multi-physics field coupling is becoming increasingly urgent; the same model needs to solve multiple physical fields such as structure, heat, electromagnetics, and fluidity simultaneously, and the coupling data between fields must be iteratively exchanged within milliseconds; ② Cloud architecture is becoming mainstream; major international simulation platforms have containerized solvers, pre- and post-processing, and visualization services, supporting remote access by multiple users via Web / VR terminals and relying on edge nodes for real-time rendering and interaction; ③ Intelligent upgrades are accelerating; technologies such as AI mesh generation, AI reduced-order models (ROM), and AI parameter calibration are being introduced into the modeling process, placing higher demands on the real-time performance and consistency of training data, inference data, and simulation data.

[0003] However, in cloud-based multi-user environments, traditional simulation data acquisition and distribution architectures are generally based on a "single-point acquisition - local caching - file-level transfer" model, which has the following drawbacks: 1. Multi-source heterogeneous data silos: Data formats, sampling frequencies, and spatiotemporal resolutions of structural grids, electromagnetic field quantities, fluid particles, thermodynamic signals, etc., vary significantly. Existing solutions lack a unified semantic description, which leads to repeated format conversions in subsequent collaborative simulations, increasing preprocessing time by 30% to 50%.

[0004] 2. Insufficient user-level data isolation: Existing cloud simulation platforms mostly adopt a coarse-grained permission model of "project-folder". Process data, intermediate files and boundary conditions of different tenants in the same physical cluster are mixed through shared file systems or object storage buckets. Isolation is only achieved by operating system layer permission bits or bucket policies, which cannot prevent cross-container side-channel attacks and illegal memory mapping.

[0005] 3. High latency in the acquisition-transmission-distribution link: The classic "solver → local I / O → post-processing → visualization" pipeline uses a batch file-to-disk method, and the round-trip time (RTT) of a single frame of data is generally in the hundreds of milliseconds to seconds, resulting in interaction delays and reduced simulation efficiency.

[0006] Therefore, how to improve the existing technology's inability to simultaneously meet the three core requirements of "unified expression of multi-source heterogeneous data, secure isolation of multi-user data, and low-latency acquisition-transmission-distribution" under cloud architecture is a technical problem that urgently needs to be solved in this field. Summary of the Invention

[0007] Based on this, the purpose of this application is to provide a data acquisition method and system for modeling and simulation cloud services to solve at least one of the technical problems mentioned in the background art.

[0008] Firstly, this application provides a data acquisition method for modeling and simulation cloud services, including: S1: Obtain the total traffic and number of data acquisition channels in each resource group, and build a dynamically expanding resource pool; S2: Obtain the current allocation request of the simulation system, including the number of data acquisition channels and the required traffic for each simulation system; S3: Obtain the current available traffic and available data acquisition channels for each resource group, and allocate data acquisition channels for each simulation system in the resource pool according to the allocation request, then return to step S2.

[0009] Further steps in building a dynamically expanding resource pool include: Configure N resource groups and determine the total traffic and number of data acquisition channels for each resource group; Assign a unique identifier to each resource group and data acquisition channel, and determine the mapping relationship between each resource group and data acquisition channel; By adding or deleting resource groups, a dynamically expanding resource pool can be built.

[0010] Furthermore, the step of obtaining the current available traffic and available data acquisition channels for each resource group, and allocating data acquisition channels for each simulation system in the resource pool according to the allocation request, includes: Calculate the total traffic and total number of channels required by each simulation system; obtain the current available traffic and available number of channels for each resource group to calculate the current total available traffic and total available number of channels for the resource pool; Determine whether the current total available traffic and total available channels of the resource pool are greater than the total traffic and total number of channels required by each simulation system; If not, add a resource group; If so, then among the currently available data acquisition channels, find the data acquisition channel that meets the required traffic, and configure the corresponding number and traffic of data acquisition channels for each simulation system.

[0011] Furthermore, among the currently available data acquisition channels, the steps of finding data acquisition channels that meet the required traffic volume and configuring the corresponding number and traffic volume of data acquisition channels for each simulation system include: Determine whether each resource group has available traffic and available data acquisition channels. If not, remove the corresponding resource group; if so, retain it. Determine whether the number of data acquisition channels required by each simulation system is greater than 1. If not, leave it unchanged. If so, split the corresponding simulation system into several subsystems until the number of channels required by each simulation system is 1. Based on the available traffic of each resource group and the estimated traffic of each simulation system, the resource groups and simulation systems are sorted to obtain the sorted resource groups and simulation systems. Matching is performed sequentially based on the order of resource groups and simulation systems, and corresponding data acquisition channels are allocated to each simulation system according to the matching results; data acquisition channels of resource groups with large available traffic are allocated to simulation systems with large required traffic.

[0012] Furthermore, the process of matching resources and simulation systems sequentially, and then allocating corresponding data acquisition channels to each simulation system based on the matching results, includes: Determine if the number of resource groups is greater than that of the simulation system; If so, then match them sequentially according to the order of resource groups and simulation systems, and allocate the corresponding data acquisition channels to each simulation system in the corresponding resource group; If not, group the simulation systems according to their estimated traffic and / or required number of channels to obtain several simulation groups, so that the estimated traffic and / or required number of channels in each group are balanced; sort the simulation groups to obtain sorted simulation groups; match the resource groups and simulation groups in sequence, and allocate the corresponding data acquisition channels to the simulation systems in each simulation group in the corresponding resource group.

[0013] Furthermore, the grouping step based on the estimated traffic and / or required number of channels for each simulation system includes: The simulation systems are grouped according to their estimated traffic and required number of channels to obtain several simulation groups, so that the estimated traffic and required number of channels are balanced in each group. Alternatively, the estimated traffic of each simulation system can be grouped to obtain several simulation groups, so that the estimated traffic of each group is balanced. Alternatively, the simulation systems can be grouped according to their required number of channels to obtain several simulation groups, so that the required number of channels in each group is balanced.

[0014] Furthermore, after obtaining the current available traffic and available data acquisition channels for each resource group, and allocating data acquisition channels for each simulation system in the resource pool according to the allocation request, the process also includes: The current traffic required by each simulation system is collected in real time, the total traffic within a set time period is obtained, and the product of the total traffic and a set coefficient is obtained as the number of buffered data queues required. Get the initial number of cached data queues and determine if it is greater than the required number of cached data queues. If it is, keep it unchanged; otherwise, get the remaining amount of middleware cluster resources. Determine if the remaining capacity is sufficient to create new cached data queues. If so, expand the capacity to obtain several new cached data queues. If not, stop data collection.

[0015] Secondly, this application also provides a modeling and simulation cloud service data acquisition system, including: The cloud-based data acquisition microservice cluster module is used to obtain the total traffic and number of data acquisition channels contained in each resource group and build a dynamically expanding resource pool. The data acquisition configuration management module is used to obtain the current allocation request of the simulation system, including the number of data acquisition channels and the required traffic for each simulation system, and allocate data acquisition channels for each simulation system in the resource pool according to the allocation request and the current available traffic and available data acquisition channels of each resource group. The data acquisition scheduling and management module is used to obtain the available traffic and available data acquisition channels of each resource group in real time and send them to the data acquisition configuration management module.

[0016] The simulation terminal module is used to acquire simulation data of the corresponding simulation system according to the allocated data acquisition channels, perform simulation operations, and obtain the simulation results of each simulation system. The cloud-based data distribution module is used to send the simulation results of each simulation system to the corresponding simulation data user terminal; The simulation data user end is used to access the corresponding simulation results.

[0017] Furthermore, the data acquisition configuration management module includes a statistics unit, a judgment unit, and an expansion unit and an allocation unit connected in sequence to the output of the judgment unit: The statistics unit is used to calculate the total traffic and total number of channels required by each simulation system; and to obtain the current available traffic and available data acquisition channels for each resource group in order to calculate the current total available traffic and total available channels of the resource pool. The judgment unit is used to determine whether the current available total traffic and available total number of channels in the resource pool are greater than the total traffic and total number of channels required by each simulation system. If not, the expansion unit is called; if so, the allocation unit is called. Expansion units are used to add resource groups; The allocation unit is used to find data acquisition channels that meet the required traffic volume among the currently available data acquisition channels, and to configure the corresponding number and traffic volume of data acquisition channels for each simulation system.

[0018] Furthermore, the allocation unit includes a first judgment component and a second judgment component arranged in parallel, and a sorting component and an allocation component connected sequentially, with the input connected to the first judgment component and the second judgment component: The first judgment component is used to determine whether each resource group has available traffic and available data acquisition channels. If not, the corresponding resource group is removed; if so, it is retained. The second judgment component is used to determine whether the number of data acquisition channels required by each simulation system is greater than 1. If not, it remains unchanged; if so, the corresponding simulation system is split into several subsystems until the number of channels required by each simulation system is 1. The sorting component is used to sort each resource group and each simulation system according to the available traffic of each resource group and the estimated traffic of each simulation system, so as to obtain the sorted resource groups and simulation systems. The allocation component is used to match the resource groups and simulation systems sequentially according to their order, and to allocate the corresponding data acquisition channels to each simulation system based on the matching results; the data acquisition channels of resource groups with large available traffic are allocated to simulation systems with large required traffic.

[0019] This invention provides a data acquisition method and system for modeling and simulation cloud services. Through S1: acquiring the total traffic and number of data acquisition channels in each resource group to construct a dynamically expanding resource pool; automatically expanding / shrinking the pool based on changes in current load or demand to improve system flexibility and scalability; then through S2: acquiring the current allocation requests of the simulation systems, including the required number of data acquisition channels and traffic for each simulation system; taking the actual needs of the simulation systems as the starting point, ensuring "on-demand" resource allocation, avoiding over-allocation or resource idleness, ensuring that each simulation system receives sufficient resources to support its operation while avoiding resource waste and improving overall resource utilization. This ensures stable system operation even under high load, achieving load balancing. Then, in step S3, the current available traffic and data acquisition channels for each resource group are obtained. Based on the allocation request, data acquisition channels are allocated to each simulation system in the resource pool, returning to step S2. Combining the current allocation request and remaining resource status, the system can dynamically adjust resource allocation, ensuring that resources are always prioritized for the simulation systems that need them most. This improves system response speed and flexibility, forming a continuously optimized closed loop. It can continuously adjust resource allocation strategies based on real-time data and changing demands, ensuring the system is always in optimal operating condition, avoiding incorrect allocation due to information lag, and improving allocation accuracy. This solves the problem that existing technologies cannot simultaneously meet the three core requirements of "unified representation of multi-source heterogeneous data, secure isolation of multi-user data, and low-latency acquisition-transmission-distribution" in a cloud architecture. Attached Figure Description

[0020] Figure 1 This is a flowchart of the modeling and simulation cloud service data acquisition method according to an embodiment of the present invention; Figure 2 This is a schematic diagram of the structure of the modeling and simulation cloud service data acquisition system according to an embodiment of the present invention; Figure 3 This is a schematic diagram of another structure of the modeling and simulation cloud service data acquisition system according to an embodiment of the present invention. Detailed Implementation

[0021] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0022] It should be noted that if the embodiments of the present invention involve directional indications, such as up, down, left, right, front, back, etc., these directional indications are only used to explain the relative positional relationships and movement of the components in a specific posture. If the specific posture changes, the directional indications will also change accordingly. Furthermore, if the embodiments of the present invention involve descriptions such as "first," "second," "S1," "S2," "step one," "step two," etc., these descriptions are for descriptive purposes only and should not be construed as indicating or implying their relative importance, or implicitly indicating the number of technical features indicated or the order of method execution. Those skilled in the art will understand that anything that does not violate the inventive concept and is within the scope of the present invention should be included in the protection scope of the present invention.

[0023] like Figure 1 As shown, this invention provides a method for data acquisition in cloud services for modeling and simulation: S1: Obtain the total traffic and number of data acquisition channels in each resource group, and build a dynamically expanding resource pool; Specifically, it is optional, but not limited to, acquiring several resource groups and their total traffic and data acquisition channels, and constructing a dynamically expandable resource pool with several resource groups. When the total traffic and data acquisition channels of the resource pool cannot meet the consumption, the number of resource groups can be increased to complete the expansion. When there is a lot of idle traffic and data acquisition channels, the number of resource groups can be reduced to complete resource recycling, reduce operating costs, and realize the dynamic expansion of the resource pool.

[0024] Preferably, the steps for constructing a dynamically expandable resource pool include: S11: Configure N resource groups and determine the total traffic and number of data acquisition channels for each resource group; S12: Assign a unique identifier to each resource group and data acquisition channel, and determine the mapping relationship between each resource group and data acquisition channel; S13: Build a dynamically expanding resource pool by adding or deleting resource groups.

[0025] Specifically, N resource groups can be configured, and the total traffic and number of data acquisition channels for each resource group can be determined. A unique identifier is set for each resource group and data acquisition channel, and the mapping relationship between each resource group and data acquisition channel is determined. This allows for the allocation of data acquisition channels that meet the traffic requirements of each simulation system based on the unique identifier and mapping relationship. At the same time, the resource pool can be dynamically expanded according to the total traffic and data acquisition channel usage in the resource pool. When the idle traffic and data acquisition channels in the resource pool cannot meet the practical needs of the simulation system, resource groups can be added to expand the resource pool as needed. When the usage demand of the simulation system is low, resulting in the traffic and data acquisition channels in the resource pool being idle for a long time, the number of resource groups can be reduced to reduce the usage cost.

[0026] For example, a data acquisition microservice cluster can be constructed using a dynamically scalable resource pool to provide cloud services to various simulation systems. The resource pool consists of several resource groups, each containing several data acquisition channels. A unique identifier is then assigned to each resource group and each data acquisition channel to facilitate the allocation of corresponding data acquisition channels to each simulation system in subsequent steps. Since the data acquisition channels in each resource group share the total traffic of the corresponding resource group, a mapping relationship between each resource group and each data acquisition channel also needs to be set to avoid the traffic occupied by the data acquisition channels in each resource group exceeding the total traffic of the corresponding resource group when allocating corresponding data acquisition channels to each simulation system, thus preventing simulation data loss.

[0027] S2: Obtain the current allocation request of the simulation system, including the number of data acquisition channels and the required traffic for each simulation system; S3: Obtain the current available traffic and available data acquisition channels for each resource group, and allocate data acquisition channels for each simulation system in the resource pool according to the allocation request, then return to step S2.

[0028] Specifically, optional but not limited to obtaining the current available traffic and available data acquisition channels for each resource group, based on the current allocation request of the simulation system, including the required number of data acquisition channels and required traffic for each simulation system, searching for data acquisition channels in each resource group in the resource pool that meet the traffic requirements of each simulation system, thereby allocating data acquisition channels for each simulation system, and finally returning to step S2, continuously iterating. When allocating data acquisition channels, the corresponding resource group must meet the requirement that the current available traffic is greater than the required traffic of the corresponding simulation system, to avoid data loss in the corresponding simulation system due to insufficient available traffic in the resource group corresponding to the data acquisition channels allocated to each simulation system. In addition, the number of available data acquisition channels must be greater than 0 to ensure that there are available data acquisition channels to provide cloud services for the simulation system, and to avoid the situation where there is available traffic in the resource group but all data acquisition channels are occupied, resulting in the inability to input data into the simulation system.

[0029] Preferably, the step of obtaining the current available traffic and available data acquisition channels for each resource group, and allocating data acquisition channels for each simulation system in the resource pool according to the allocation request, includes: S31: Calculate the total traffic and total number of channels required by each simulation system; obtain the current available traffic and available number of channels for each resource group to calculate the current available total traffic and available total number of channels of the resource pool; S32: Determine whether the current total available traffic and total available channels of the resource pool are greater than the total traffic and total number of channels required by each simulation system; S33: If not, add a resource group; S34: If so, then among the currently available data acquisition channels, find the data acquisition channel that meets the required flow rate, and configure the corresponding number and flow rate of data acquisition channels for each simulation system.

[0030] Specifically, since the total available traffic and the total number of data acquisition channels in the resource pool are limited, when the required traffic and the required number of data acquisition channels of a simulation system exceed the total available traffic and the total number of data acquisition channels, it is known that no matter how the allocation is done, some simulation systems will inevitably be unable to be allocated data acquisition channels, or even if they are allocated data acquisition channels, data loss will occur due to insufficient available traffic in the resource pool, preventing data upload. Therefore, before allocating data acquisition channels to each simulation system, it is advisable to statistically analyze the total required traffic and the total number of channels for each simulation system, as well as the current total available traffic and the total number of channels in the resource pool, and determine the current available traffic and the total number of channels in the resource pool. If the total traffic and number of channels exceed the total traffic and number of channels required by each simulation system, it indicates a resource shortage that cannot provide services to all simulation systems. In this case, a resource group needs to be added to expand the resource pool to avoid situations where simulation systems cannot be allocated data acquisition channels or data loss occurs during data transmission. If the total available traffic and number of data acquisition channels in the current resource pool are sufficient to meet the needs of each simulation system, there is no need to add a resource group to expand the resource pool. Instead, channels that meet the traffic requirements can be found directly among the available data acquisition channels in the current resource pool, and the corresponding number and traffic of data acquisition channels can be configured for each simulation system.

[0031] Preferably, the steps of finding data acquisition channels that meet the required traffic volume among the currently available data acquisition channels and configuring a corresponding number and traffic volume of data acquisition channels for each simulation system include: S341: Determine whether each resource group has available traffic and available data acquisition channels. If not, remove the corresponding resource group; if yes, retain it. Specifically, the process can be optional, but not limited to, determining whether each resource group has available traffic and available data acquisition channels. If not, it means that there are no available data acquisition channels and / or traffic in the corresponding resource group, so the corresponding resource group needs to be removed to ensure that all considered resource groups have available channels and traffic, reduce the complexity of subsequent processing, speed up allocation, avoid invalid matching, and improve allocation efficiency. If yes, it means that the corresponding resource group still has at least one allocable data acquisition channel and some available traffic for the corresponding simulation system, which needs to be retained to improve resource utilization efficiency.

[0032] S342: Determine whether the number of data acquisition channels required by each simulation system is greater than 1. If not, leave it unchanged. If yes, then split the corresponding simulation system into several subsystems until the number of channels required by each simulation system is 1. Specifically, the number of data acquisition channels required by each simulation system varies. However, when the number of channels required by a simulation system is greater than 1, it indicates that the simulation system is composed of several subsystems. For example, a power simulation system may be composed of simulation subsystems such as thermal, electrical, and control. The data of each simulation subsystem is incompatible with each other, and each data acquisition channel corresponds to an independent physical or logical simulation system. Furthermore, the data acquisition channels cannot be merged. Therefore, it is possible to determine whether the number of channels required by each simulation system is greater than 1. If so, the corresponding simulation system is split into several independent simulation subsystems until the number of channels required by each simulation system is 1. This facilitates the allocation of data acquisition channels for each simulation system that requires a separate data acquisition channel in subsequent steps.

[0033] S343: Based on the available traffic of each resource group and the estimated traffic of each simulation system, sort each resource group and each simulation system to obtain the sorted resource groups and simulation systems. S344: Match the resource groups and simulation systems sequentially according to their order, and allocate corresponding data acquisition channels to each simulation system based on the matching results; allocate the data acquisition channels of resource groups with large available traffic to simulation systems with large required traffic.

[0034] Specifically, optional but not limited to sorting each resource group and each simulation system according to the same set order based on the available traffic of each resource group and the estimated traffic of each simulation system, the sorted resource groups and simulation systems are obtained. The sorting mechanism makes resource allocation more orderly, prioritizing simulation systems with high traffic demand, ensuring fairer resource allocation, avoiding system bottlenecks caused by uneven resource allocation, and improving overall system performance. Then, matching is performed according to the sorting of resource groups and simulation systems to allocate corresponding data acquisition channels to each simulation system based on the matching results. This ensures that each simulation system can obtain resources that match its needs, improves resource utilization, and can flexibly adjust the allocation strategy according to the sorting results to adapt to different simulation system needs and resource conditions. In this way, data acquisition channels of resource groups with high available traffic are allocated to simulation systems with high traffic demand to achieve load balancing.

[0035] Preferably, the steps of matching resource groups and simulation systems sequentially according to their order, and allocating corresponding data acquisition channels to each simulation system based on the matching results, include: S3441: Determine if the number of resource groups is greater than that of the simulation system; S3442: If so, then match according to the order of resource groups and simulation systems, and allocate corresponding data acquisition channels to each simulation system in the corresponding resource group; S3443: If not, group the simulation systems according to their estimated traffic and / or required number of channels to obtain several simulation groups, so that the estimated traffic and / or required number of channels in each group are balanced; sort the simulation groups to obtain sorted simulation groups; match the resource groups and simulation groups in sequence, and allocate the corresponding data acquisition channels to the simulation systems in each simulation group in the corresponding resource group.

[0036] Specifically, the method can be optional, but not limited to, determining whether the number of resource groups exceeds the number of simulation systems. If so, resources are sufficient, and a fine-grained matching process ("one-to-one" or "many-to-one") can be used. Matching is performed sequentially based on the order of resource groups and simulation systems. Based on the matching results, corresponding data acquisition channels are allocated to each simulation system, distributing them across resource groups to achieve load balancing. If not, resources are scarce, requiring aggregation and grouping of simulation systems before resource allocation to prevent local overload and achieve load balancing. This involves first grouping simulation systems based on their estimated traffic and / or required number of channels, resulting in several simulation groups with balanced estimated traffic and / or required channel counts. These simulation groups are then sorted. Matching is performed sequentially based on the order of resource groups and simulation groups, allocating corresponding data acquisition channels to the simulation systems within each resource group. This achieves the intelligent scheduling goal of ensuring performance when resources are sufficient and ensuring fairness and efficiency when resources are scarce.

[0037] For example, suppose there are three resource groups, each with 1000Mb, 800Mb, and 900Mb of traffic, and five data acquisition channels. Three simulation systems send allocation requests. Assume these three simulation systems do not have subsystems, so each simulation system corresponds to only one data acquisition channel, with estimated traffic of 500Mb, 600Mb, and 800Mb respectively. The resource groups and simulation systems are sorted, resulting in resource groups of 1000Mb, 900Mb, and 800Mb; and simulation systems of 800Mb, 600Mb, and 500Mb. Based on the sorting results, data acquisition channels are allocated to each simulation system within each resource group. Ultimately, the available traffic for each resource group is 200Mb, 300Mb, and 30Mb respectively, thus achieving load balancing as much as possible while preventing data overflow.

[0038] For example, suppose there are three resource groups, each with 1000Mb of traffic and 5 data acquisition channels. Five simulation systems send allocation requests. Assume that none of these five simulation systems have subsystems, so each simulation system corresponds to only one data acquisition channel. The estimated traffic is 300Mb, 400Mb, 800Mb, 600Mb, and 300Mb, respectively. Optionally, the five simulation systems can be divided into three groups: 600Mb+300Mb, 300Mb+400Mb, and 800Mb. Then, the three groups of simulation systems are allocated to the resource groups one by one. Finally, the available traffic for each resource group is 100Mb, 300Mb, and 200Mb, respectively, thereby achieving load balancing as much as possible while preventing data overflow.

[0039] This embodiment presents a data acquisition method for modeling and simulation cloud services according to the present invention. S1: Obtain the total traffic and number of data acquisition channels in each resource group to construct a dynamically expandable resource pool. Automatic scaling up / down based on changes in current load or demand enhances the system's flexibility and scalability. Then, S2: Obtain the current allocation request of the simulation system, including the required number of data acquisition channels and traffic for each simulation system. Based on the actual needs of the simulation system, this method ensures "on-demand" resource allocation, avoiding over-allocation or resource idleness. It ensures that each simulation system receives sufficient resources to support its operation while preventing resource waste and improving overall resource efficiency. Utilization is optimized to ensure stable system operation under high load, achieving load balancing. Then, step S3 retrieves the current available traffic and data acquisition channels for each resource group. Based on allocation requests, data acquisition channels are allocated to each simulation system in the resource pool, returning to step S2. Combining the current allocation requests and remaining resource status, the system can dynamically adjust resource allocation, ensuring resources are always prioritized for the simulation systems that need them most. This improves system response speed and flexibility, forming a continuous optimization closed loop. It continuously adjusts resource allocation strategies based on real-time data and changing demands, ensuring the system is always in optimal operating condition, avoiding incorrect allocation due to information lag, and improving allocation accuracy. This solves the problem that existing technologies cannot simultaneously meet the three core requirements of "unified representation of multi-source heterogeneous data, secure isolation of multi-user data, and low-latency acquisition-transmission-distribution" in a cloud architecture.

[0040] Preferably, during the data acquisition process, because the simulation system may suddenly write a large amount of traffic, causing a surge in system access in a short period of time, exceeding the normal load capacity and causing traffic shocks, caching needs to be set during the data acquisition process to avoid data loss. Therefore, the data acquisition steps may optionally include: S35: Real-time acquisition of the current required traffic of each simulation system, acquisition of the total traffic within a set time period, and acquisition of the product of the total traffic and a set coefficient, which is the required number of cached data queues; Specifically, it is optional, but not limited to, real-time collection of the current required traffic of each simulation system to obtain the total traffic within a set time period, and to obtain the product of the total traffic and a set coefficient as the required number of cached data queues to cope with traffic fluctuations and sudden peaks.

[0041] Preferably, the time period is set to 1 second, and the coefficient is set to 60, which means that one minute of data can be cached.

[0042] S36: Get the initial number of cached data queues and determine whether it is greater than the required number of cached data queues. If yes, keep it unchanged; otherwise, get the remaining amount of middleware cluster resources. Specifically, the intermediate cluster resources can be selected from memory-based message queues, which are also resource pools that can be deployed in a cluster. They support dynamic addition of nodes, and the entire cluster can be dynamically scheduled and allocated. Before the system runs, the intermediate cluster resources will create several initial cached data queues to cope with traffic fluctuations and sudden peaks. Then, based on the number of cached data queues required in step S41, it is determined whether the number of initial cached data queues is greater than the number of required cached data queues. If so, it means that the allocated initial cached data queues are sufficient to cope with sudden traffic fluctuations and no further processing is required. If not, it means that the number of initial cached data queues is insufficient and it is necessary to obtain the remaining middleware cluster resources, i.e., the remaining available traffic, to provide a data basis for the subsequent steps of creating new cached data queues for expansion.

[0043] S37: Determine whether the remaining capacity is sufficient to create new cached data queues. If so, expand the capacity to obtain several new cached data queues. If not, stop data collection.

[0044] Specifically, since the resources contained in the middleware cluster are pre-defined, before scaling up, it is necessary to obtain the remaining resources of the middleware cluster and determine whether the remaining resources are sufficient to create new cache data queues. If so, scaling up can be carried out to obtain several new cache data queues to cope with traffic fluctuations and sudden peaks and avoid data loss. If not, it means that the overall capacity of the cluster has been used up and cannot be newly allocated. Data collection needs to be stopped to avoid further data loss.

[0045] Preferably, when the middleware cluster resources are all used up, the number of cluster nodes can be manually increased according to the actual situation to achieve the purpose of overall expansion of the resource pool.

[0046] like Figure 2 , Figure 3 As shown, the present invention also provides a modeling and simulation cloud service data acquisition system, comprising: The cloud-based data acquisition microservice cluster module 201 is used to obtain the total traffic and number of data acquisition channels contained in each resource group and build a dynamically expanding resource pool. The data acquisition configuration management module 202 is used to obtain the current allocation request of the simulation system, including the number of data acquisition channels and the required traffic for each simulation system, and allocate data acquisition channels for each simulation system in the resource pool according to the allocation request and the current available traffic and available data acquisition channels of each resource group. The data acquisition scheduling and management module 203 is used to obtain the available traffic and available data acquisition channels of each resource group in real time and send them to the data acquisition configuration management module. The simulation terminal module 204 is used to acquire simulation data of the corresponding simulation system according to the allocated data acquisition channel, perform simulation operations, and obtain the simulation results of each simulation system. The cloud data distribution module 205 is used to send the simulation results of each simulation system to the corresponding simulation data user terminal. Simulation data user terminal 206 is used to access the corresponding simulation results.

[0047] Specifically, the total traffic and number of data acquisition channels in each resource group can be obtained through the cloud-based data acquisition microservice cluster module to build a dynamically expandable resource pool. When the total traffic and data acquisition channels in the resource pool cannot meet the consumption, the number of resource groups can be increased to achieve expansion. When there is a lot of idle traffic and data acquisition channels, the number of resource groups can be reduced to achieve resource reclamation, reduce operating costs, and achieve dynamic expansion of the resource pool. Then, the current allocation request of the simulation system is obtained through the data acquisition configuration management module, including the required number of data acquisition channels and the required traffic for each simulation system. Based on the allocation request and the current available traffic and available data acquisition channels of each resource group, the resource pool is allocated to each... Each simulation system is allocated a data acquisition channel. The process involves obtaining the current allocation requests from the simulation systems, including the required number of data acquisition channels and the required bandwidth for each system. Based on these requests and the current available bandwidth and data acquisition channels in each resource group, a data acquisition channel that meets the bandwidth requirements of each simulation system is searched within the resource pool. This allocation ensures that the available bandwidth of the corresponding resource group is greater than the bandwidth required by the corresponding simulation system. This prevents data loss due to insufficient available bandwidth in the resource group corresponding to the allocated data acquisition channels. Furthermore, the number of available data acquisition channels must be greater than zero to guarantee the availability of usable data. The data acquisition channels provide cloud services to the simulation system, preventing situations where resource groups have available traffic but all data acquisition channels are occupied, thus preventing data input to the simulation system. The data acquisition scheduling and management module obtains the available traffic and available data acquisition channels for each resource group in real time and sends this information to the data acquisition configuration management module. The module also collects the available data acquisition channels for each resource group in real time and monitors the traffic occupied by each channel. Based on the total traffic of each resource group, the available traffic for each resource group is calculated, thus obtaining the available traffic and available data acquisition channels for each resource group. This information is then sent to the data acquisition configuration management module, providing a data foundation for subsequent iterative steps. Finally, the simulation terminal module processes the allocated data... The data acquisition channel acquires simulation data from the corresponding simulation system and performs simulation operations to obtain the simulation results of each system. By completing the simulation operations in the cloud, the computational resource consumption and hardware costs of local simulation at the simulation data user end are reduced. Optionally, more computational resources can be provided through cloud services to improve simulation efficiency. Then, the simulation results of each simulation system are sent to the corresponding simulation data user end through the cloud data distribution module, and the simulation data user end can directly use the corresponding simulation results. This completes the data acquisition for modeling and simulation cloud services, forming a continuously optimized closed loop. It can continuously adjust the resource allocation strategy according to real-time data and changes in demand to ensure that the system is always in the best operating state.

[0048] Preferably, the data acquisition configuration management module includes a statistics unit, a judgment unit, and an expansion unit and an allocation unit connected in sequence to the output of the judgment unit: The statistics unit is used to calculate the total traffic and total number of channels required by each simulation system; and to obtain the current available traffic and available data acquisition channels for each resource group in order to calculate the current total available traffic and total available channels of the resource pool. The judgment unit is used to determine whether the current available total traffic and available total number of channels in the resource pool are greater than the total traffic and total number of channels required by each simulation system. If not, the expansion unit is called; if so, the allocation unit is called. Expansion units are used to add resource groups; The allocation unit is used to find data acquisition channels that meet the required traffic volume among the currently available data acquisition channels, and to configure the corresponding number and traffic volume of data acquisition channels for each simulation system.

[0049] Specifically, since the total traffic and total number of data acquisition channels in the resource pool are limited, when the required traffic and number of data acquisition channels of a simulation system exceed the available total traffic and the available total number of data acquisition channels, it is known that no matter how the allocation is carried out, some simulation systems will inevitably be unable to be allocated data acquisition channels, or even if they are allocated data acquisition channels, data loss will occur due to insufficient available traffic in the resource pool, preventing data upload. Therefore, before allocating data acquisition channels to each simulation system, it is advisable to use a statistical unit to count the total traffic and total number of channels required by each simulation system; obtain the current available traffic and available data acquisition channels for each resource group to count the current available total traffic and available total number of channels in the resource pool, and then... The judgment unit determines whether the current total traffic and total number of channels in the resource pool are greater than the total traffic and total number of channels required by each simulation system. If not, it indicates that resources are scarce and cannot provide services to all simulation systems. In this case, the expansion unit needs to be called to add resource groups to expand the resource pool, so as to avoid situations where simulation systems cannot be allocated data acquisition channels or data loss occurs during data transmission. If so, it indicates that the current total traffic and number of data acquisition channels in the resource pool are sufficient to meet the current needs of each simulation system. There is no need to add resource groups to expand the resource pool. The allocation unit can be directly called to find channels that meet the traffic requirements among the available data acquisition channels in the current resource pool and configure the corresponding number and traffic of data acquisition channels for each simulation system.

[0050] Preferably, the allocation unit includes a first judgment component and a second judgment component arranged in parallel, and a sorting allocation component and an allocation component connected sequentially, with the input connected to the first judgment component and the second judgment component: The first judgment component is used to determine whether each resource group has available traffic and available data acquisition channels. If not, the corresponding resource group is removed; if so, it is retained. The second judgment component is used to determine whether the number of data acquisition channels required by each simulation system is greater than 1. If not, it remains unchanged; if so, the corresponding simulation system is split into several subsystems until the number of channels required by each simulation system is 1. The sorting component is used to sort each resource group and each simulation system according to the available traffic of each resource group and the estimated traffic of each simulation system, so as to obtain the sorted resource groups and simulation systems. The allocation component is used to match the resource groups and simulation systems sequentially according to their order, and to allocate the corresponding data acquisition channels to each simulation system based on the matching results; the data acquisition channels of resource groups with large available traffic are allocated to simulation systems with large required traffic.

[0051] Specifically, the first judgment component can be used to determine whether each resource group has available traffic and available data acquisition channels. If not, it means that there are no available data acquisition channels and / or traffic in the corresponding resource group, so the corresponding resource group needs to be removed to reduce the complexity of subsequent processing, ensure that all considered resource groups have available channels and traffic, speed up the allocation, avoid invalid matching, and improve allocation efficiency. If yes, it means that the corresponding resource group still has at least one allocable data acquisition channel and some available traffic to the corresponding simulation system, which needs to be retained. Since the number of channels required by each simulation system is different, when the number of channels required by the simulation system is greater than 1, it means that the simulation system is composed of several subsystems. For example, a power simulation system may be composed of thermal, electrical, control and other simulation subsystems. The data of each simulation subsystem is incompatible with each other, and each data acquisition channel corresponds to an independent physical or logical simulation system. The data acquisition channels cannot be merged. Therefore, the second judgment component can be used to determine whether the number of channels required by each simulation system is greater than 1. If yes, the corresponding simulation system can be removed. The system is divided into several independent simulation subsystems until each simulation system requires only one channel. This facilitates the allocation of data acquisition channels to each simulation system requiring a separate channel in subsequent steps. A sorting component then sequentially sorts each resource group and simulation system according to the available traffic of each resource group and the estimated traffic of each simulation system, resulting in sorted resource groups and simulation systems. This sorting mechanism ensures more orderly resource allocation, prioritizing simulation systems with high traffic demands, ensuring fairer resource allocation, avoiding system bottlenecks caused by uneven resource distribution, and improving overall system performance. Finally, an allocation component matches the resource groups and simulation systems according to their sorting, allocating corresponding data acquisition channels to each simulation system based on the matching results. This ensures that each simulation system receives resources matching its needs, improving resource utilization. The allocation strategy can be flexibly adjusted based on the sorting results to adapt to different simulation system needs and resource conditions, thereby allocating data acquisition channels from resource groups with high available traffic to simulation systems with high traffic demands, achieving load balancing.

[0052] Preferably, the allocation component includes a first allocation block and a second allocation block arranged side-by-side and connected to the input and decision blocks: The decision block is used to determine whether the number of resource groups is greater than that of the simulation system, and selectively call the first allocation component and the second allocation component based on the decision result. The first allocation block is used to match the resource groups and simulation systems sequentially, and allocate corresponding data acquisition channels to each simulation system in the corresponding resource group. The second allocation block is used to group simulation systems according to their estimated traffic and / or required number of channels, resulting in several simulation groups, so that the estimated traffic and / or required number of channels in each group are balanced; sorting each simulation group to obtain sorted simulation groups; matching the resource group and simulation group in sequence, and allocating the corresponding data acquisition channels to the simulation systems in each simulation group in the corresponding resource group.

[0053] Specifically, the system can optionally, but is not limited to, using a judgment block to determine whether the number of resource groups exceeds the number of simulation systems. Based on the judgment result, it can selectively call the first and second allocation components to adopt different strategies when resources are sufficient or scarce, based on the comparison between resources and demand. This prevents over-allocation or allocation failure, dynamically selects the optimal allocation strategy, and improves system adaptability. When resources are sufficient, the first allocation block can use a fine-grained "one-to-one" or "many-to-one" matching method to match resource groups and simulation systems sequentially according to their order. Based on the matching results, it can allocate corresponding data acquisition channels to each simulation system and distribute the simulation systems to the various resource groups. In the process, load balancing is achieved. When resources are scarce, the simulation systems need to be aggregated and grouped first through the second allocation block, and then resources are allocated to prevent local overload and achieve load balancing. First, the simulation systems are grouped according to their estimated traffic and / or required number of channels to obtain several simulation groups, so that the estimated traffic and / or required number of channels in each group are balanced. The simulation groups are then sorted to obtain sorted simulation groups. According to the sorting of resource groups and simulation groups, matching is performed in sequence, and corresponding data acquisition channels are allocated to the simulation systems in each simulation group in the corresponding resource group. This achieves the intelligent scheduling goal of ensuring performance when resources are sufficient and ensuring fairness and efficiency when resources are scarce.

[0054] This embodiment presents a modeling and simulation cloud service data acquisition system of the present invention. It obtains the total traffic and number of data acquisition channels in each resource group through a cloud-based data acquisition microservice cluster module, constructing a dynamically expandable resource pool. When the total traffic and data acquisition channels in the resource pool cannot meet the consumption, the number of resource groups can be increased to achieve expansion. When there is a large amount of idle traffic and data acquisition channels, the number of resource groups can be reduced to achieve resource reclamation, reducing operating costs and realizing dynamic expansion of the resource pool. Then, the data acquisition configuration management module obtains the current allocation request of the simulation system, including the required number of data acquisition channels and the required traffic for each simulation system, and adjusts the allocation according to the allocation request and the current availability of each resource group. Using traffic and available data acquisition channels, data acquisition channels are allocated to each simulation system in the resource pool. By obtaining the current allocation requests from the simulation systems, including the required number of data acquisition channels and traffic for each system, data acquisition channels that meet the traffic requirements of each simulation system are searched within each resource group in the resource pool based on the allocation requests and the current available traffic and available data acquisition channels of each resource group. This ensures that data acquisition channels are allocated to each simulation system, and that the corresponding resource group's current available traffic is greater than the required traffic of the corresponding simulation system. This prevents data loss in the corresponding simulation system due to insufficient available traffic in the resource group corresponding to the allocated data acquisition channels. Furthermore, the number of available data acquisition channels... A value greater than 0 ensures the existence of available data acquisition channels to provide cloud services to the simulation system. This avoids situations where resource groups have available traffic, but all data acquisition channels are occupied, preventing data input to the simulation system. The data acquisition scheduling management module then obtains the available traffic and available data acquisition channels for each resource group in real time and sends this information to the data acquisition configuration management module. The module also collects the available data acquisition channels for each resource group in real time and tracks the traffic occupied by each channel. Based on the total traffic of each resource group, the available traffic for each resource group is calculated, thus obtaining the available traffic and available data acquisition channels for each resource group. This information is then sent to the data acquisition configuration management module, providing a data foundation for subsequent iterative steps. Finally, the simulation terminal module... The module acquires simulation data from the corresponding simulation system based on the allocated data acquisition channels, performs simulation operations, and obtains the simulation results of each system. By completing the simulation operations in the cloud, the computational resource consumption and hardware costs of local simulation at the simulation data user end are reduced. Optional cloud services can provide more computational resources to improve simulation efficiency. Then, the simulation results of each system are sent to the corresponding simulation data user end via the cloud data distribution module, allowing the simulation data user end to directly use the corresponding simulation results. This completes the data acquisition for modeling and simulation cloud services, forming a continuously optimized closed loop. It can continuously adjust resource allocation strategies based on real-time data and changing demands, ensuring the system is always in optimal operating condition. This solves the problem that existing technologies cannot simultaneously meet the three core requirements of "unified representation of multi-source heterogeneous data, secure isolation of multi-user data, and low-latency acquisition-transmission-distribution" under a cloud architecture.

[0055] The embodiments described above are merely illustrative of several implementations of the present invention, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these all fall within the protection scope of the present invention. Therefore, the protection scope of this invention patent should be determined by the appended claims.

Claims

1. A method for data acquisition in cloud services for modeling and simulation, characterized in that, include: S1: Obtain the total traffic and number of data acquisition channels in each resource group, and build a dynamically expanding resource pool; S2: Obtain the current allocation request of the simulation system, including the number of data acquisition channels and the required traffic for each simulation system; S3: Obtain the current available traffic and available data acquisition channels for each resource group, and allocate data acquisition channels for each simulation system in the resource pool according to the allocation request, then return to step S2.

2. The method according to claim 1, characterized in that, The steps for building a dynamically expanding resource pool include: Configure N resource groups and determine the total traffic and number of data acquisition channels for each resource group; Assign a unique identifier to each resource group and data acquisition channel, and determine the mapping relationship between each resource group and data acquisition channel; By adding or deleting resource groups, a dynamically expanding resource pool can be built.

3. The method according to claim 2, characterized in that, The steps for obtaining the current available traffic and available data acquisition channels for each resource group, and allocating data acquisition channels for each simulation system in the resource pool according to the allocation request, include: Calculate the total traffic and total number of channels required by each simulation system; obtain the current available traffic and available number of channels for each resource group to calculate the current total available traffic and total available number of channels for the resource pool; Determine whether the current total available traffic and total available channels of the resource pool are greater than the total traffic and total number of channels required by each simulation system; If not, add a resource group; If so, then among the currently available data acquisition channels, find the data acquisition channel that meets the required traffic, and configure the corresponding number and traffic of data acquisition channels for each simulation system.

4. The method according to claim 3, characterized in that, The steps for identifying data acquisition channels that meet the required throughput from the currently available data acquisition channels and configuring the corresponding number and throughput data acquisition channels for each simulation system include: Determine whether each resource group has available traffic and available data acquisition channels. If not, remove the corresponding resource group; if so, retain it. Determine whether the number of data acquisition channels required by each simulation system is greater than 1. If not, leave it unchanged. If so, split the corresponding simulation system into several subsystems until the number of channels required by each simulation system is 1. Based on the available traffic of each resource group and the estimated traffic of each simulation system, the resource groups and simulation systems are sorted to obtain the sorted resource groups and simulation systems. Matching is performed sequentially based on the order of resource groups and simulation systems, and corresponding data acquisition channels are allocated to each simulation system according to the matching results; data acquisition channels of resource groups with large available traffic are allocated to simulation systems with large required traffic.

5. The method according to claim 4, characterized in that, The steps of matching resources and simulation systems sequentially, and allocating corresponding data acquisition channels to each simulation system based on the matching results, include: Determine if the number of resource groups is greater than that of the simulation system; If so, then match them sequentially according to the order of resource groups and simulation systems, and allocate the corresponding data acquisition channels to each simulation system in the corresponding resource group; If not, group the simulation systems according to their estimated traffic and / or required number of channels to obtain several simulation groups, so that the estimated traffic and / or required number of channels in each group are balanced; sort the simulation groups to obtain sorted simulation groups; match the resource groups and simulation groups in sequence, and allocate the corresponding data acquisition channels to the simulation systems in each simulation group in the corresponding resource group.

6. The method according to claim 5, characterized in that, The steps for grouping based on the estimated traffic and / or required number of channels for each simulation system include: The simulation systems are grouped according to their estimated traffic and required number of channels to obtain several simulation groups, so that the estimated traffic and required number of channels are balanced in each group. Alternatively, the estimated traffic of each simulation system can be grouped to obtain several simulation groups, so that the estimated traffic of each group is balanced. Alternatively, the simulation systems can be grouped according to their required number of channels to obtain several simulation groups, so that the required number of channels in each group is balanced.

7. The method according to claim 1, characterized in that, After obtaining the current available traffic and available data acquisition channels for each resource group, and allocating data acquisition channels for each simulation system in the resource pool according to the allocation request, the process also includes: The current traffic required by each simulation system is collected in real time, the total traffic within a set time period is obtained, and the product of the total traffic and a set coefficient is obtained as the number of buffered data queues required. Get the initial number of cached data queues and determine if it is greater than the required number of cached data queues. If it is, keep it unchanged; otherwise, get the remaining amount of middleware cluster resources. Determine if the remaining capacity is sufficient to create new cached data queues. If so, expand the capacity to obtain several new cached data queues. If not, stop data collection.

8. A modeling and simulation cloud service data acquisition system, characterized in that, include: The cloud-based data acquisition microservice cluster module is used to obtain the total traffic and number of data acquisition channels contained in each resource group and build a dynamically expanding resource pool. The data acquisition configuration management module is used to obtain the current allocation request of the simulation system, including the number of data acquisition channels and the required traffic for each simulation system, and allocate data acquisition channels for each simulation system in the resource pool according to the allocation request and the current available traffic and available data acquisition channels of each resource group. The data acquisition scheduling and management module is used to obtain the available traffic and available data acquisition channels of each resource group in real time and send them to the data acquisition configuration management module. The simulation terminal module is used to acquire simulation data of the corresponding simulation system according to the allocated data acquisition channels, perform simulation operations, and obtain the simulation results of each simulation system. The cloud-based data distribution module is used to send the simulation results of each simulation system to the corresponding simulation data user terminal; The simulation data user end is used to access the corresponding simulation results.

9. The system according to claim 8, characterized in that, The data acquisition configuration management module includes a statistics unit, a judgment unit, and an expansion unit and an allocation unit connected in sequence to the output of the judgment unit: The statistics unit is used to calculate the total traffic and total number of channels required by each simulation system; and to obtain the current available traffic and available data acquisition channels for each resource group in order to calculate the current total available traffic and total available channels of the resource pool. The judgment unit is used to determine whether the current available total traffic and available total number of channels in the resource pool are greater than the total traffic and total number of channels required by each simulation system. If not, the expansion unit is called; if so, the allocation unit is called. Expansion units are used to add resource groups; The allocation unit is used to find data acquisition channels that meet the required traffic volume among the currently available data acquisition channels, and to configure the corresponding number and traffic volume of data acquisition channels for each simulation system.

10. The system according to claim 8, characterized in that, The allocation unit includes a first judgment component and a second judgment component arranged in parallel, and a sorting component and an allocation component connected sequentially, with the input connected to the first judgment component and the second judgment component: The first judgment component is used to determine whether each resource group has available traffic and available data acquisition channels. If not, the corresponding resource group is removed; if so, it is retained. The second judgment component is used to determine whether the number of data acquisition channels required by each simulation system is greater than 1. If not, it remains unchanged; if so, the corresponding simulation system is split into several subsystems until the number of channels required by each simulation system is 1. The sorting component is used to sort each resource group and each simulation system according to the available traffic of each resource group and the estimated traffic of each simulation system, so as to obtain the sorted resource groups and simulation systems. The allocation component is used to match the resource groups and simulation systems sequentially according to their order, and to allocate the corresponding data acquisition channels to each simulation system based on the matching results. Allocate the data acquisition channels of resource groups with high available bandwidth to simulation systems that require high bandwidth.

Images