Data processing system and method for multi-source heterogeneous water environment data

Abstract

The application provides a data processing system and method for multi-source heterogeneous water environment data, which is applied to the technical field of water environment data processing. The system comprises a data adaptation layer, a model library layer, a model adaptation layer and a function entry layer. The data adaptation layer adopts an adaptation module to convert multi-source data into a standard format, realizing decoupling of data access and subsequent model calculation. The model library layer stores calculation models with different functions. The model adaptation layer encapsulates each calculation model into a unified adaptation function, so that the calculation model is loaded on demand only when the function is actually called. The function allocation layer maps the adaptation function into a subcommand, and an external host system can trigger the processing of a water conservancy calculation request by inputting the subcommand name in the form of a sub-process. In this way, each water conservancy calculation request only loads the calculation model required for this calculation, and the calculation model that is not called does not occupy the running memory of the data processing system at all times, thereby greatly saving the memory space.

Description

Technical Field

[0001] This application relates to the field of water environment data processing technology, and more specifically, to a data processing system and method for multi-source heterogeneous water environment data in the field of water environment data processing technology. Background Technology

[0002] In complex scientific computing and engineering applications such as environmental impact assessment, hydrological and hydrodynamic simulation, water pollution diffusion prediction, and ecological carrying capacity assessment, data processing systems typically need to process water environment data from multiple sources and in multiple formats to process water conservancy calculation requests. However, different water conservancy calculation requests usually require different calculation models. Therefore, data processing systems need to be configured with multiple calculation models.

[0003] When using a data processing system to process water conservancy calculation requests based on water environment data, even if a single target calculation model can handle the target water conservancy calculation request, the data processing system still needs to load all calculation models at startup and use the target calculation model to process the water conservancy calculation request. This results in a long loading time for the data processing system to load the calculation models and requires a large amount of running memory to store all the calculation models. Summary of the Invention

[0004] This application provides a data processing system and method for multi-source heterogeneous water environment data. The system can load the calculation model for processing water conservancy calculation requests only after calling the target adaptation function, which can reduce the loading time of the calculation model and save memory space during system operation.

[0005] Firstly, a data processing system for multi-source heterogeneous water environment data is provided, including: The data adaptation layer is used to convert water environment data from different sources into a standard data format. The data adaptation layer includes multiple data adaptation modules. Each data adaptation module is used to convert the corresponding water environment data into a standard data format. The data adaptation modules correspond one-to-one with the water environment data. The model library layer stores computational models with different functions, and each computational model is used to process data in a standard data format. The model adaptation layer is used to encapsulate the various computational model calls in the model library layer into corresponding adaptation functions. Each adaptation function has a corresponding function signature. The function allocation layer maps the function signature of each adapter function to the corresponding first sub-command, so that the host system can call the target adapter function corresponding to the second sub-command from each adapter function based on the second sub-command and the first sub-command generated by the hydraulic calculation request, in order to load the calculation model and data adaptation module corresponding to the target adapter function, and process the hydraulic calculation request based on the calculation model, data adaptation module and water environment data corresponding to the target adapter function.

[0006] In one possible implementation, the model adaptation layer stores the third-party dependency libraries for each computational model. When the target adaptation function is called, the model adaptation layer loads the third-party dependency libraries of the computational model corresponding to the target adaptation function from the model adaptation layer.

[0007] In one possible implementation, the standard data format includes at least one of the following: geographic information exchange format, character-separated value table format, and multidimensional numerical array format; the water environment data includes at least one of the following: computer graphics files, hydrodynamic model result data, network meteorological data, geographic information vector or raster files, network structured data, and network semi-structured data.

[0008] In one possible implementation, when new water environment data is detected in the system, the data adaptation layer is also used to add a data adaptation module corresponding to the new water environment data. When a new computing model is identified in the system, the model library layer is used to add the new computing model, the model adaptation layer is used to add the adaptation function and function signature corresponding to the new computing model, and the function allocation layer is used to map the function signature corresponding to the new computing model to a sub-command.

[0009] In a second aspect, a data processing method is provided, applied to the system in any embodiment of the first aspect, comprising the following steps: In response to the input second sub-command, the control function allocation layer determines the target adaptation function corresponding to the second sub-command based on the obtained second sub-command and the first sub-command of each adaptation function. The second sub-command is generated by the host system based on the water conservancy calculation request, and the first sub-command is generated by the function allocation layer based on the function signature of the adaptation function. The control function allocation layer calls the target adaptation function to load the computational model corresponding to the target adaptation function from the model library layer, and loads the data adaptation module corresponding to the target adaptation function from the data adaptation layer. The control model adaptation layer and the data adaptation layer process the hydraulic calculation request based on the calculation model corresponding to the target adaptation function, the data adaptation module, and the water environment data to obtain the processing result.

[0010] In one possible implementation, the control model adaptation layer and the data adaptation layer process the hydraulic calculation request based on the computational model corresponding to the target adaptation function, the data adaptation module corresponding to the target adaptation function, and the water environment data to obtain the processing result, including: The control data adaptation layer uses a data adaptation module to transform water environment data to obtain target data; The control model adaptation layer processes the water conservancy calculation request based on the target data and the calculation model corresponding to the target adaptation function to obtain the processing result.

[0011] In one possible implementation, the control model adaptation layer processes the hydraulic calculation request based on the target data and the computational model corresponding to the target adaptation function to obtain the processing result, including: When there are N target adaptation functions, the control model adaptation layer calls the N target adaptation functions in sequence according to the preset function order, so that the calculation model corresponding to the called target adaptation function processes the water conservancy calculation request according to the target data. The processing result of the nth calculation model is used as the input data of the (n+1)th calculation model to obtain the processing result corresponding to the Nth target adaptation function, where n is an integer greater than 0 and less than N.

[0012] In one possible implementation, the data processing method further includes: Load the third-party dependency library of the computational model corresponding to the target adaptation function from the model adaptation layer; The control model adaptation layer and the data adaptation layer process the water conservancy calculation request based on the calculation model corresponding to the target adaptation function, the third-party dependency library, the data adaptation module and the water environment data to obtain the processing result.

[0013] In one possible implementation, the data processing method further includes: The control function allocation layer uses a command dispatch component to map the function signature of each adapter function to the corresponding first sub-command.

[0014] This application provides a data processing system and method for multi-source heterogeneous water environment data. The system includes a data adaptation layer, a model library layer, and a function entry layer. The data adaptation layer consists of multiple data adaptation modules, each corresponding to a specific water environment data point, used to uniformly convert multi-source heterogeneous water environment data into a standard data format, decoupling data access from subsequent model calculation input. The model library layer stores computational models with different functions. The model adaptation layer encapsulates the calls to each computational model in the model library layer into corresponding adaptation functions, each with a corresponding function signature. The function allocation layer maps the function signature of each adaptation function to a corresponding first sub-command. During the data processing system startup phase, no computational models or data adaptation modules need to be loaded. Only when an actual computation request reaches an adaptation function does the adaptation function load the computational model and data adaptation modules for the current computation according to the second sub-command. Compared with the existing technology that loads all computing models at startup, this application fundamentally avoids the loading overhead of irrelevant computing models, significantly shortens system response time, greatly reduces memory usage, and enables heterogeneous computing models with incompatible dependencies to coexist in isolation within the same data processing system. Attached Figure Description

[0015] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0016] Figure 1 A schematic diagram of a data processing system architecture for multi-source heterogeneous water environment data is provided in this application embodiment; Figure 2 A flowchart illustrating a data processing method for multi-source heterogeneous water environment data provided in this application embodiment. Figure 1 ; Figure 3 A flowchart illustrating a data processing method for multi-source heterogeneous water environment data provided in this application embodiment. Figure 2 ; Figure 4 A flowchart illustrating a data processing method for multi-source heterogeneous water environment data provided in this application embodiment. Figure 3 ; Figure 5 A schematic diagram of the structure of a data processing device for multi-source heterogeneous water environment data provided in an embodiment of this application; Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application.

[0017] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to particular embodiments. Detailed Implementation

[0018] The technical solutions in this application will be clearly and thoroughly described below with reference to the accompanying drawings. In the description of the embodiments of this application, unless otherwise stated, " / " means "or," for example, A / B can mean A or B. "And / or" in the text is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Furthermore, in the description of the embodiments of this application, "multiple" refers to two or more than two.

[0019] Hereinafter, the terms "first" and "second" are used for descriptive purposes only and should not be construed as implying or suggesting relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature.

[0020] Before introducing the embodiments of this application, the application background of the embodiments of this application will be explained first: In complex scientific computing and engineering applications such as environmental impact assessment, hydrological and hydrodynamic simulation, water pollution diffusion prediction, and ecological carrying capacity assessment, data processing systems typically need to process water environment data from multiple sources and in multiple formats to process water conservancy calculation requests. However, different water conservancy calculation requests usually require different calculation models. Therefore, data processing systems need to be configured with multiple calculation models.

[0021] The data processing system can handle graphic files from hydraulic auxiliary software, proprietary time series and grid result files from hydrodynamic software, meteorological reanalysis and remote sensing raster data from public scientific data services, vector and raster files from geographic information systems, and structured and semi-structured data that needs to be crawled from public websites. These different sources of water environment data have different formats, inconsistencies in coordinate systems, and significant differences in data organization structures. Hard-coding the logic for parsing water environment data from different sources into the source code of the data processing system would result in highly coupled source code, poor reusability, and maintenance difficulties.

[0022] Data processing systems need to integrate a wide variety of computational models, including analytical solution models based on closed mathematical expressions, large-scale distributed hydrological and hydrodynamic models, atmospheric diffusion models based on Gaussian plumes, and machine learning and natural language processing models based on neural networks. These computational models differ greatly in their input data formats, output formats, computational parameters, model operation methods, and third-party dependency libraries. In some cases, conflicts even exist between different computational models or dependency libraries. For example, certain geospatial libraries, hydrodynamic interface libraries, and deep learning data processing systems have incompatible requirements for the versions of underlying dynamic link libraries or dependency libraries.

[0023] In view of the characteristics of the above-mentioned data processing systems, the existing data processing systems mainly have the following shortcomings: First, importing all computational models and their third-party dependencies in a centralized manner during the startup phase of the data processing system leads to high startup overhead and high memory usage. Any computation process involving only a single model must also load all computational models and their dependencies. Second, when the versions of third-party libraries that different computational models depend on conflict with each other, centralized import will directly cause the process to fail to load, making it difficult for different computational models to coexist in the same system.

[0024] Secondly, there is a lack of unified calling conventions among the various computing models. Therefore, the source code of the data processing system needs to be written with special adaptation logic for each computing model. The front-end or back-end host system needs to be aware of the specific type, parameter order and result structure of each computing model. This results in high integration costs and difficulties in expansion for each computing model in the data processing system. Each new computing model requires modifications to the existing source code in multiple places, which increases the maintenance cost of the source code.

[0025] Third, the host system often uses a different technology stack than the computing model. For example, the host system may use C language to write the source code, while the computing model may use Python language. Therefore, when configuring the computing model to the host system, the Python language source code of the computing model needs to be integrated into the C language source code of the host system. However, data processing systems lack low-cost cross-language integration methods and usually need to build network services or perform complex inter-language bindings for each computing model, which increases the maintenance cost of the source code.

[0026] Fourth, the data processing system lacks isolation for concurrent computing scenarios of different computing models and standardized management of result files. Result files from different computing models processing different requests are prone to overwriting each other and are difficult to trace.

[0027] In summary, the main problem with existing data processing systems is that when processing water conservancy calculation requests based on water environment data, even if a single target calculation model can handle the request, the system still needs to load all calculation models and use the target model to process the request. This results in a long loading time for the data processing system and requires a large amount of RAM to store all the calculation models.

[0028] To address the technical problems existing in related technologies, the inventors of this application propose the following solution: For the technical problem of long startup time and excessive memory consumption caused by loading all computational models in a data processing system, this application first sets up a data adaptation layer, composed of multiple data adaptation modules corresponding one-to-one with water environment data. This layer is used to uniformly convert multi-source heterogeneous water environment data into a standard data format, achieving decoupling between data access and subsequent model computation input. A model library layer is set up to store computational models with different functions. A model adaptation layer is set up to encapsulate the calls of each computational model in the model library layer into corresponding adaptation functions, each with a corresponding function signature. A function allocation layer is set up to map the function signature of each adaptation function to a corresponding first sub-command. During system startup, no computational models or data adaptation modules need to be loaded. Only when an actual computation request reaches the adaptation function does the adaptation function load the computational model and data adaptation modules for this computation as needed, based on the second sub-command. Compared with the existing technology that loads all computing models at startup, this application fundamentally avoids the loading overhead of irrelevant computing models, significantly shortens system response time, greatly reduces memory usage, and enables heterogeneous computing models with incompatible dependencies to coexist in isolation within the same data processing system.

[0029] Figure 1 This is a schematic diagram of a data processing system architecture for multi-source heterogeneous water environment data provided in an embodiment of this application, such as... Figure 1 As shown, the data processing system includes a data adaptation layer, a model library layer, a model adaptation layer, and a function allocation layer.

[0030] The data adaptation layer is used to convert water environment data from different sources into a standard data format. The data adaptation layer includes multiple data adaptation modules, each of which is used to convert the corresponding water environment data into a standard data format. Each data adaptation module corresponds one-to-one with the water environment data.

[0031] The data adaptation layer, located at the bottom of the data processing system architecture, is responsible for converting water environment data from different sources into a standardized data format. In its implementation, the data adaptation layer is configured as an independent data transformation set, internally divided into multiple data adaptation modules based on the type of data source being processed.

[0032] Each data adaptation module corresponds to a specific type of water environment data format and is specifically responsible for reading, parsing, and converting the format of that type of water environment data. Each data adaptation module is independent of the others, with no code-level dependencies. Each module encapsulates its own parsing logic and conversion rules for its specific water environment data format. Any data adaptation module can be developed, tested, deployed, and upgraded independently without affecting the normal operation of other data adaptation modules in the data adaptation layer.

[0033] Specifically, a standard data format is a unified data representation specification agreed upon within a data processing system. The output of all data adaptation modules and the input of all computational models follow the standard data format. Standard data formats include at least one or more of the following types: geographic information exchange format for expressing spatial vector data, character-delimited value table format for expressing structured tabular data, and multidimensional numerical array format for expressing gridded or spatiotemporal sequence data.

[0034] For example, standard data formats include at least one of the following: geographic information exchange format, character-delimited value table format, and multidimensional numerical array format; water environment data includes at least one of the following: computer graphics files, hydrodynamic model result data, network meteorological data, geographic information vector or raster files, network structured data, and network semi-structured data. Through the unified agreement of the above standard data formats, water environment data from different sources and in different original formats are transformed by the data adaptation layer and presented in a consistent data form within the data processing system architecture, thus achieving data format standardization.

[0035] The model library layer stores computational models with different functions, and each computational model is used to process data in a standard data format.

[0036] The model library layer, located above the data adaptation layer, stores computational models with different functions. In its implementation, the model library layer is written as a model storage subpackage, which is divided into multiple independent computational model modules according to their functions.

[0037] In one embodiment, each computational model is encapsulated as an independent computational model module. Each module contains the core computational logic, parameter configuration, and input / output interface definitions for that model. These modules are independent at the code level, with no direct function calls or variable references, and each maintains its own required computational logic and dependencies.

[0038] For example, computational models include, but are not limited to: analytical solution models encapsulating analytical formulas for water environment impacts, models encapsulating distributed urban stormwater models, models encapsulating one-dimensional hydrodynamic models, models encapsulating two-dimensional hydrodynamic models, models encapsulating atmospheric Gaussian diffusion models, and models encapsulating machine learning sequence prediction and text mining. Each computational model is independent of the others and requires different third-party dependencies.

[0039] In the model library layer, the input interfaces of each computational model are configured to accept data in a standard data format. Specifically, before performing a computation, each computational model reads data that has been converted to a standard data format from the input data source (e.g., the output of the data adaptation module), without needing to handle the parsing and conversion of the original data format itself. The output of the computation results from each computational model is also in a standard data format, facilitating unified output processing by subsequent computational modules or passing it to other computational models for coupled computation.

[0040] The model adaptation layer is used to encapsulate the calls of each computational model in the model library layer into corresponding adaptation functions, and each adaptation function has a corresponding function signature.

[0041] The model adaptation layer, located above the model library layer, encapsulates the calls to various computational models in the model library layer into corresponding adaptation functions. In its implementation, the model adaptation layer is configured as a set of adaptation functions, internally defining at least one set of adaptation functions, each corresponding one-to-one with a computational model in the model library layer. Each adaptation function has a corresponding function signature. The function signature includes the adaptation function name and a parameter list. The parameter list includes fields such as the current call identifier, file storage path, and parameters of the computational model.

[0042] For example, the adaptation functions include, but are not limited to, adaptation functions for one-dimensional river water quality calculation, adaptation functions for two-dimensional river water quality calculation, adaptation functions for urban stormwater model calculation, and adaptation functions for atmospheric Gaussian diffusion calculation, and all adaptation functions have the same function signature format.

[0043] After the adaptor function is invoked, the corresponding computational model processes the hydraulic calculation request and returns the processing result. This result includes at least an identifier field, a coordinate field, and a data field. The identifier field is used to backfill the call identifier for this call, allowing the host system to associate the call request and response. The coordinate field describes the spatial or temporal dimension of the processing result, such as longitudinal distance, lateral distance, or time sequence. The data field carries the physical quantities obtained by the computational model from the hydraulic calculation request, such as pollutant concentration distribution. For computational models that produce complex output results such as two-dimensional concentration fields or concentration fields evolving over time, the adaptor function, along with the returned result object, also writes result files such as contour maps, animations, or tables to the file storage path.

[0044] Therefore, although the analytical solution model outputs the values ​​of closed expressions on the grid, the distributed model outputs the time series of nodes and pipe segments, and the atmospheric model outputs the concentration on the polar coordinate grid, the model adaptation layer will standardize the structure of the output processing results to obtain result objects. This allows the host system to receive result objects with the same structure without needing to be aware of the specific type of the computation model.

[0045] For example, part of the source code for the adapter function encapsulation is shown below: def cal_c_river_1d_plot(appid, sdir, title, k, Ex, u, B, A, Cp, Qp, Ch, Qh, x1, x2, dx): # Key point: The computational model is loaded lazily on demand within the function body, rather than being imported when the data processing system loads it. from psyp.modelbase.weia import cal_c_jxj # Call the model library layer to complete the calculation res = cal_c_jxj.cal_c_river_1d(k, Ex, u, B, A, Cp, Qp, Ch, Qh, x1,x2, dx) # Map the processing results to a unified structured result object, and write the result file named with appid to sdir ... return {"id": appid, "x": x.tolist(), "data": data} The function allocation layer maps the function signature of each adapter function to the corresponding first sub-command, so that the host system can call the target adapter function corresponding to the second sub-command from each adapter function based on the second sub-command and the first sub-command generated by the hydraulic calculation request, in order to load the calculation model and data adaptation module corresponding to the target adapter function, and process the hydraulic calculation request based on the calculation model, data adaptation module and water environment data corresponding to the target adapter function.

[0046] The function allocation layer is located above the model adaptation layer, which is the top layer of the data processing system architecture. It maps the function signatures of each adaptation function in the model adaptation layer to the corresponding first sub-command and establishes a mapping table between each first sub-command and its corresponding entry function. The entry function is the call entry point exposed by the data processing system to the external host system. Each entry function corresponds to one adaptation function and is used to call the corresponding adaptation function after receiving a call request.

[0047] For example, the following is a portion of the code called by the external host system as a child process: def cal_c_river_1d_plot(title, k, Ex, u, B, A, Cp, Qp, Ch, Qh, x1,x2, dx): from psyp.scenario.modeladpter import mcore suid = str(uuid.uuid1()) # Generates the identifier for each call. res = mcore.cal_c_river_1d_plot( suid, 'D: / ... / ClientScenarioes', # Identifier for this call and file storage path title, k, Ex, u, B, A, Cp, Qp, Ch, Qh, x1, x2, dx) print(json.dumps(res)) # Print the result object to the standard output stream if __name__ == '__main__': fire.Fire() # Reflective command line, i.e., selecting the entry function. In one embodiment, the external host system calls the function allocation layer as a subprocess, passing in the subcommand name and computational model parameters corresponding to the second subcommand. Upon receiving the second subcommand, the function allocation layer searches for the first subcommand matching the second subcommand in the mapping table based on the subcommand name, determines the entry function corresponding to the first subcommand, and passes the computational model parameters to the entry function. After the entry function is called, it first generates a call identifier for this call, then passes the call identifier, the preset file storage path, and the computational model parameters to the corresponding field of the function signature of the corresponding adapter function. Upon receiving the function signature including the above parameters, the target adapter function, within its function body, delays loading the corresponding computational model and data adaptation module according to the needs of the water conservancy calculation request. The computational model and data adaptation module execute the request processing according to the computational model parameters in the function signature. The model adaptation layer maps the processing result into a unified structured result object, stores it in a folder under the file storage path, and returns the unified structured result object to the entry function. Upon receiving the result object returned by the adapter function, the entry function serializes the result object into a string and prints it via the standard output stream for the external host system to read.

[0048] Since each call to the entry function generates a unique call identifier, and this identifier is used as the naming basis for the folders written to the file storage path, even if multiple users or multiple business scenarios initiate calculations simultaneously under the file storage path, the result files generated by each calculation will not overwrite each other due to their different names, thereby achieving isolation of the result objects of concurrent calculations and traceable management of the result files.

[0049] In summary, the data processing system provided in this application maps the function signatures of each adaptor function to corresponding subcommands through a function allocation layer. This allows the host system to input only the subcommand, and the data processing system can then locate the target computational model based on the subcommand, processing the hydraulic calculation request without needing to know the specific type and parameter details of the target computational model. Furthermore, through a dynamic loading mechanism within the adaptor function, the corresponding computational model and data adaptation module are loaded only when the target adaptor function is called, avoiding a full loading of the computational model during system startup. This significantly reduces the startup latency and memory usage of the data processing system.

[0050] Furthermore, the model adaptation layer stores the third-party dependency libraries for each computational model. When the target adaptation function is called, the model adaptation layer loads the third-party dependency library corresponding to the computational model from this layer. These third-party dependency libraries provide highly optimized numerical computation functions, which the computational model can directly call to complete core calculations, significantly reducing the development difficulty of the computational model. In addition, some third-party dependency libraries have cross-language interface encapsulation capabilities. For example, some numerical computation libraries are implemented in C or C++. Encapsulating these libraries into third-party dependency libraries and providing Python interfaces allows Python-implemented computational models to directly call the low-level computational code of high-performance third-party dependency libraries, balancing development efficiency and runtime performance.

[0051] In one embodiment, the data processing system for multi-source heterogeneous water environment data provided in this application adopts an on-demand delayed loading mechanism. That is, in the prior art, data processing systems typically use a full, centralized loading of all computational models. Specifically, during the data processing system startup phase, all computational models and third-party dependency libraries required by the data processing are centrally imported into the top layer of the first-level code (e.g., the first few lines of code in the function entry layer). For example, the hydrodynamic interface library and the geospatial library have different version requirements for the underlying dynamic link libraries, and deep learning data processing systems also have specific dependencies on the runtime environment. If all these dependencies are centrally imported during data processing system startup, version conflicts or environment incompatibility will directly cause process loading failure, making it impossible for heterogeneous models to coexist within the same system.

[0052] Therefore, this application does not use the direct import method at the top level of the first layer of code in the data processing system in each adapter function of the model adaptation layer and each entry function of the function allocation layer. Instead, all import statements are placed inside the function body. Specifically, each adapter function and entry function only declares its function signature during definition. The computational model, data adaptation module, and related third-party dependency libraries referenced within their function bodies are loaded on demand within the function body through local import statements. The above import operation is only triggered when the adapter function is actually called and the import statement is executed, rather than being executed uniformly when the system starts or a certain layer is loaded.

[0053] Based on the above-mentioned on-demand lazy loading mechanism, the data processing system provided in this application achieves the following two technical effects: On the one hand, in any hydraulic calculation request, the data processing system only loads the calculation model actually involved in that request and its corresponding third-party dependency library. Other calculation models not called within the data processing system and their potentially conflicting dependency libraries are never loaded. Since conflicting dependency libraries are not loaded simultaneously in the same process and in the same run, originally incompatible third-party dependency libraries can coexist peacefully within the same data processing system.

[0054] On the other hand, during the startup process of a data processing system for multi-source heterogeneous water environment data, regardless of how many models are integrated into the system, the startup phase only completes the initialization of the system itself and the registration of commands, without performing any import operations of computational models or third-party dependency libraries, thereby significantly reducing the startup latency and memory usage of the data processing system.

[0055] Furthermore, upon detecting newly added water environment data in the system, the data adaptation layer also adds a data adaptation module corresponding to the newly added water environment data, specifically including: When a new computing model is identified in the system, the model library layer is used to add the new computing model, the model adaptation layer is used to add the adaptation function and function signature corresponding to the new computing model, and the function allocation layer is used to map the function signature corresponding to the new computing model to a sub-command.

[0056] When a user wants to add a new type of data source, the user only needs to add a new data adaptation module in the data adaptation layer, and the data processing system can realize the conversion from the data source format to the standard data format without modifying the model library layer, the model adaptation layer, or the existing data adaptation module.

[0057] When a user wants to add a new type of computational model, the user first needs to add a new computational model encapsulating the computational logic of that model to the model library layer of the data processing system. Then, the data processing system adds an adaptation function and function signature corresponding to the new computational model in the model adaptation layer. Within the function body of the adaptation function in the data processing system, the user sets up a delayed loading and invocation of the new computational model based on the needs of the water conservancy calculation request, and sets the execution logic to map the processing results to a result object with a unified structure. The function allocation layer of the data processing system maps the function signature of the adaptation function corresponding to the new computational model to a subcommand. The data processing system does not need to modify any existing mapping logic or existing model encapsulation logic. The entire addition process described above does not affect other existing computational modules in the data processing system, thus enabling the data processing system to be updated agilely and securely as business needs grow.

[0058] Figure 2 This is a flowchart illustrating a data processing method provided in an embodiment of this application. Figure 1 ,like Figure 2 As shown, applied to Figure 1 The data processing system mentioned in the corresponding embodiment includes the following steps: Step 201: In response to the input second sub-command, the control function allocation layer determines the target adapter function corresponding to the second sub-command based on the acquired second sub-command and the first sub-command of each adapter function.

[0059] The second sub-command is generated by the host system based on the water conservancy calculation request, while the first sub-command is generated by the function allocation layer based on the function signature of the adapting function.

[0060] In this step, when the data processing system starts, the function allocation layer automatically scans all the adapter functions defined in the model adaptation layer through the command distribution component, obtains the function signature of each adapter function, and maps the function signature of the adapter function to the corresponding first subcommand. The first subcommand includes the subcommand name (corresponding to the function name of the adapter function) and the parameter list of the adapter function. Each adapter function has a unique first subcommand.

[0061] In one embodiment, when the host system initiates a water conservancy calculation request, the host system generates a second sub-command based on the request. This second sub-command includes the target sub-command name and the parameters of the calculation model required for the current calculation. In response to the second sub-command, the data processing system's control function allocation layer performs string matching between the target sub-command name in the second sub-command and the sub-command names in each first sub-command. If a match is found, the corresponding adaptation function for each first sub-command is determined as the target adaptation function.

[0062] In one embodiment, prior to this step, the function signatures of each adaptor function are mapped to subcommands through a function allocation layer. This allows the host system to automatically locate and route the adaptor function simply by passing in the target subcommand name, without needing to write any specific calling logic for the computational model on the host system side.

[0063] Step 202: The control function allocation layer calls the target adaptation function to load the computational model corresponding to the target adaptation function from the model library layer, and loads the data adaptation module corresponding to the target adaptation function from the data adaptation layer.

[0064] In this step, the target adapter function references the computational model module and the data adapter module within its function body using local import statements. When the function allocation layer calls the target adapter function, and the program execution reaches the target adapter function's function body, the target adapter function executes the import statements within its function body, dynamically loading the computational model module corresponding to the adapter function from the model library layer, and dynamically loading the data adapter module corresponding to the adapter function from the data adapter layer. The loading operation is triggered only when the target adapter function is actually called, rather than during the data processing system startup phase or when the function allocation layer completes subcommand registration, thus reducing the latency and memory usage of the data processing system.

[0065] By placing the import statement inside the adapter function body, the loading of the computational model and data adaptation module is delayed until the adapter function is actually called, thus achieving on-demand loading and avoiding the loading overhead of irrelevant computational models and their dependent libraries.

[0066] Step 203: The control model adaptation layer and the data adaptation layer process the water conservancy calculation request according to the calculation model corresponding to the target adaptation function, the data adaptation module corresponding to the target adaptation function, and the water environment data to obtain the processing result.

[0067] In this step, the data adaptation layer converts the water environment data from its original format to a standard data format according to the data adaptation module corresponding to the target adaptation function. The target adaptation function calls the computational model, passing the standard data format data as input and the computational model parameters from the second sub-command, triggering the computational model to process the water conservancy calculation request. After completing the processing, the computational model returns the processing result to the target adaptation function, which then returns the processing result to the function allocation layer. The function allocation layer then maps the processing result to obtain a result object and sends the result object to the entry function. The entry function serializes the result object and outputs it to the host system.

[0068] The data adaptation layer automatically standardizes and converts data formats during model computation, freeing the host system from the need to concern itself with data format adaptation details. Simultaneously, the data input interface of the computational model is unified to a standard data format, allowing the output of any data adaptation module to serve as input for any computational model, thus decoupling data processing from model computation.

[0069] Furthermore, the control model adaptation layer and the data adaptation layer process the hydraulic calculation request based on the calculation model corresponding to the target adaptation function, the data adaptation module corresponding to the target adaptation function, and the water environment data to obtain the processing result, such as... Figure 3 As shown, step 203 includes the following implementation steps: Step 2031: The control data adaptation layer uses a data adaptation module to transform the water environment data to obtain the target data.

[0070] In this step, the data processing system calls the data adaptation module (hereinafter referred to as the target adaptation module) corresponding to the target adaptation function from the data adaptation layer, and controls the data adaptation layer to use the target adaptation module to read the water environment data stored in the file storage path, parse the water environment data including spatial coordinates, time series, physical quantity values ​​and attributes, and reorganize the parsed information into target data with a unified data structure according to the standard data format convention, and output it to the function allocation layer or cache it in memory for subsequent calculation models to read, so that the target data can be directly read and processed by any calculation model in the model library layer.

[0071] The target data can be at least one of the following: geographic information exchange format, character-separated value table format, or multidimensional numerical array format.

[0072] Step 2032: The control model adaptation layer processes the water conservancy calculation request according to the target data and the calculation model corresponding to the target adaptation function to obtain the processing result.

[0073] In this step, the data processing system controls the model adaptation layer to find the target adaptation function corresponding to the second sub-command. The target adaptation function corresponds to a calculation model. The target data is passed as input data to the calculation model corresponding to the target adaptation function, and the model parameters in the second sub-command are also passed in. The calculation model is used to process the water conservancy calculation request and generate calculation result data.

[0074] In this process, the target data generated in step 2031 is passed to the computational model without any format conversion, because both the target data output by the data adaptation module and the data input interface of the computational model conform to the standard data format. That is, the target data is directly input into the computational model's data interface, where the computational model reads the fields and values ​​according to the definition of the standard data format, completes the calculation, and outputs the result. The calculation result is then returned to the target adaptation function, which returns the result to the function allocation layer. The function allocation layer maps the processed result to obtain a result object and sends it to the entry function. The entry function serializes the result object and outputs it to the host system. Target data output by any data adaptation module in the data adaptation layer can be directly used by any computational model in the model library layer, achieving complete decoupling between the data access end (i.e., the target adaptation function) and the model computation end (i.e., the computational model).

[0075] Furthermore, the control model adaptation layer processes the hydraulic calculation request based on the target data and the calculation model corresponding to the target adaptation function to obtain the processing results, including: When there are N target adaptation functions, the control function allocation layer calls the N target adaptation functions in sequence according to the preset function order, so that the calculation model corresponding to the called target adaptation function processes the water conservancy calculation request according to the target data and obtains the processing result corresponding to the Nth target adaptation function.

[0076] The processing result of the nth computational model is used as the input data of the (n+1)th computational model, where n is an integer greater than 0 and less than N.

[0077] In one embodiment, in certain business scenarios, processing a single water conservancy calculation request requires the sequential execution of two or more target adaptation functions. Each target adaptation function corresponds to a computational model, meaning N computational models are needed to process the water conservancy calculation request. For example, first, the urban stormwater model is run to obtain the runoff generation process of the sub-catchment area, and then this runoff generation process is used as a boundary condition input to a one-dimensional hydrodynamic model for river channel calculation. To improve the computational efficiency of water conservancy calculation requests, this application encapsulates the processing of the aforementioned sequential computational models within a special adaptation function of the model adaptation layer. The special adaptation function has a pre-set function order. Within its function body, the special adaptation function sequentially and delay-loads the stormwater model module and the hydrodynamic model module as needed. It first calls the former to obtain the runoff generation result in a standard data format, then uses the runoff generation result as input to the one-dimensional hydrodynamic model, which then processes the runoff generation result to obtain the final processing result. By using the same method of calling a single target adaptation function to process water conservancy calculation requests, N target adaptation functions are called to process the water conservancy calculation requests. During this process, the host system does not need to be aware of how many calculation models are connected in series within the special adaptation functions of the model adaptation layer.

[0078] Furthermore, the model adaptation layer stores the third-party dependency libraries for each computational model. It controls the model adaptation layer and data adaptation layer to process the hydraulic calculation requests based on the computational model, data adaptation module, and water environment data corresponding to the target adaptation function, in order to obtain the processing results, such as... Figure 4 As shown, step 203 also includes the following steps: Step 2033: Load the third-party dependency library of the computational model corresponding to the target adaptation function from the model adaptation layer.

[0079] In this step, when the target adaptation function is called by the function allocation layer, the data processing system loads the target computation model corresponding to the target adaptation function from the model adaptation layer. Within its function body, the target adaptation function loads the third-party dependency libraries that the computation model corresponding to the target adaptation function depends on from the model adaptation layer according to the pre-set logical correspondence.

[0080] These third-party dependencies include, but are not limited to, numerical computation libraries, geospatial data processing libraries, deep learning data processing systems, parallel computing libraries, and hydrodynamic-specific interface libraries. Different computational models may have incompatible versions, underlying dynamic link libraries, or runtime environments among their dependent third-party libraries.

[0081] In one embodiment, the third-party dependency library is stored independently for each computational model in the model adaptation layer. Combined with the on-demand loading mechanism in the target adaptation function, the third-party dependency library is loaded only when the corresponding model is called, thereby achieving the isolation and coexistence of conflicting dependency libraries and reducing the system operation and maintenance burden.

[0082] Step 2034: The control model adaptation layer and the data adaptation layer process the water conservancy calculation request according to the calculation model corresponding to the target adaptation function, the third-party dependency library, the data adaptation module and the water environment data to obtain the processing result.

[0083] In this step, the data adaptation layer converts the water environment data from its original format to a standard data format based on the loaded data adaptation modules, obtaining the target data. The model adaptation layer processes the water conservancy calculation request based on the loaded third-party dependency libraries and calculation models. After the calculation model completes the processing, it returns the processing result to the target adaptation function. The target adaptation function maps the processing result into a result object with a unified structure, and after serialization processing by the function allocation layer, it outputs it to the host system.

[0084] Furthermore, the data processing method also includes the following implementation steps: The control function allocation layer uses a command dispatch component to map the function signature of each adapter function to the corresponding first sub-command.

[0085] In this implementation, when the data processing system starts, the command distribution component automatically scans all the adaptation functions defined in the model adaptation layer, obtains the function signature of each adaptation function, where the function signature includes the function name and parameter list information, generates the corresponding sub-command name based on the function name, generates the corresponding sub-command parameter specification based on the parameter list, combines the sub-command name and sub-command parameter specification into the first sub-command, and establishes a mapping relationship between the first sub-command and the adaptation function, which is stored in the command registry of the function distribution layer.

[0086] The following are embodiments of the apparatus described in this application, which can be used to execute the embodiments of the method described in this application. For details not disclosed in the apparatus embodiments of this application, please refer to the embodiments of the method described in this application.

[0087] Figure 5 This is a schematic diagram of the data processing device for multi-source heterogeneous water environment data provided in an embodiment of this application. Figure 5 As shown, the device includes: Processing module 501 is used to respond to the input second sub-command. The control function allocation layer determines the target adaptation function corresponding to the second sub-command based on the obtained second sub-command and the first sub-command of each adaptation function. The second sub-command is generated by the host system based on the water conservancy calculation request, and the first sub-command is generated by the function allocation layer based on the function signature of the adaptation function. Control module 502 is used to control the function allocation layer to call the target adaptation function, so as to load the calculation model corresponding to the target adaptation function from the model library layer and load the data adaptation module corresponding to the target adaptation function from the data adaptation layer. The control module 502 is also used to control the model adaptation layer and the data adaptation layer to process the water conservancy calculation request according to the calculation model corresponding to the target adaptation function, the data adaptation module corresponding to the target adaptation function and the water environment data, so as to obtain the processing result.

[0088] In one possible implementation, the control module 502 is specifically used for: The control data adaptation layer uses a data adaptation module to transform water environment data to obtain target data; The control model adaptation layer processes the water conservancy calculation request based on the target data and the calculation model corresponding to the target adaptation function to obtain the processing result.

[0089] In one possible implementation, the control module 502 is further configured to: When there are N target adaptation functions, the control model adaptation layer calls the N target adaptation functions in sequence according to the preset function order, so that the calculation model corresponding to the called target adaptation function processes the water conservancy calculation request according to the target data. The processing result of the nth calculation model is used as the input data of the (n+1)th calculation model to obtain the processing result corresponding to the Nth target adaptation function, where n is an integer greater than 0 and less than N.

[0090] In one possible implementation, the model adaptation layer stores third-party dependency libraries for each computational model, and the control module 502 is further configured to: Load the third-party dependency library of the computational model corresponding to the target adaptation function from the model adaptation layer; The control model adaptation layer and the data adaptation layer process the water conservancy calculation request based on the calculation model corresponding to the target adaptation function, the third-party dependency library, the data adaptation module and the water environment data to obtain the processing result.

[0091] In one possible implementation, the device further includes: The control module 502 is also used to control the function allocation layer to use the command distribution component to map the function signature of each adapting function to the corresponding first sub-command.

[0092] The apparatus provided in this application embodiment can be used to execute the determination method in any of the above embodiments. Its implementation principle and technical effect are similar, and will not be described again here.

[0093] It should be noted that the division of the various modules in the above device is merely a logical functional division. In actual implementation, they can be fully or partially integrated into a single physical entity, or they can be physically separated. Furthermore, these modules can be implemented entirely in software via processing element calls; they can be fully implemented in hardware; or some modules can be implemented in software via processing element calls, while others are implemented in hardware. Additionally, these modules can be fully or partially integrated together, or implemented independently. The processing element here can be an integrated circuit with signal processing capabilities. During implementation, each step of the above method or each of the above modules can be completed through the integrated logic circuits in the hardware of the processor element or through software instructions.

[0094] Figure 6 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application, such as... Figure 6 As shown, the electronic device may include: a processor 61, a memory 62, and computer program instructions stored in the memory 62 and executable on the processor 61. When the processor 61 executes the computer program instructions, it implements the method provided in any of the foregoing embodiments.

[0095] Optionally, the various components of the electronic device can be connected via a system bus.

[0096] The memory 62 can be a separate memory unit or a memory unit integrated into the processor 61. The number of processors 61 can be one or more.

[0097] It should be understood that processor 61 can be a Central Processing Unit (CPU), or other general-purpose processor 61, digital signal processor 61 (DSP), application-specific integrated circuit (ASIC), etc. The general-purpose processor 61 can be a microprocessor 61, or any conventional processor 61. The steps of the method disclosed in this application can be directly manifested as being executed by the hardware processor 61, or being executed by a combination of hardware and software modules within the processor 61.

[0098] The system bus can be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc. The system bus can be divided into address bus, data bus, control bus, etc. For ease of representation, only one thick line is used in the diagram, but this does not indicate that there is only one bus or one type of bus. Memory 62 may include random access memory (RAM) 62, and may also include non-volatile memory (NVM) 62, such as at least one disk storage device 62.

[0099] All or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a readable memory 62. When the program is executed, it performs the steps of the above method embodiments; and the aforementioned memory 62 (storage medium) includes: read-only memory 62 (ROM), RAM, flash memory 62, hard disk, solid-state hard disk, magnetic tape, floppy disk, optical disk, and any combination thereof.

[0100] The electronic device provided in this application embodiment can be used to execute the method provided in any of the above method embodiments. Its implementation principle and technical effect are similar, and will not be repeated here.

[0101] This application provides a computer-readable storage medium storing computer instructions that, when executed on a computer, cause the computer to perform the above-described method.

[0102] The aforementioned computer-readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory, electrically erasable programmable read-only memory, erasable programmable read-only memory, programmable read-only memory, read-only memory, magnetic storage, flash memory, magnetic disk, or optical disk. The readable storage medium can be any available medium accessible to a general-purpose or special-purpose computer.

[0103] Optionally, a readable storage medium can be coupled to a processor, enabling the processor to read information from and write information to the readable storage medium. Alternatively, the readable storage medium can be an integral part of the processor. Both the processor and the readable storage medium can reside in an Application Specific Integrated Circuit (ASIC). Alternatively, the processor and the readable storage medium can exist as discrete components within the device.

[0104] This application also provides a computer program product, which includes a computer program stored in a computer-readable storage medium. At least one processor can read the computer program from the computer-readable storage medium, and the at least one processor can implement the above-described method when executing the computer program.

[0105] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this application is limited only by the appended claims.

Claims

1. A data processing system for multi-source heterogeneous water environment data, characterized in that, include: A data adaptation layer is used to uniformly convert water environment data from different sources into a standard data format. The data adaptation layer includes multiple data adaptation modules, each of which is used to convert the corresponding water environment data into the standard data format. Each data adaptation module corresponds one-to-one with the water environment data. The model library layer is used to store computational models with different functions, and each computational model is used to process data in the standard data format. The model adaptation layer is used to encapsulate each computational model call in the model library layer into a corresponding adaptation function, and each adaptation function has a corresponding function signature; The function allocation layer is used to map the function signature of each of the adapting functions to a corresponding first sub-command, so that the host system can call the target adapting function corresponding to the second sub-command from each adapting function according to the second sub-command generated by the water conservancy calculation request and the first sub-command, so as to load the calculation model and data adaptation module corresponding to the target adapting function, and process the water conservancy calculation request according to the calculation model, data adaptation module and water environment data corresponding to the target adapting function.

2. The system according to claim 1, characterized in that, The model adaptation layer stores the third-party dependency library for each computational model. When the target adaptation function is called, the model adaptation layer loads the third-party dependency library of the computational model corresponding to the target adaptation function from the model adaptation layer.

3. The system according to claim 1, characterized in that, The standard data format includes at least one of the following: geographic information exchange format, character-separated value table format, and multidimensional numerical array format; the water environment data includes at least one of the following: computer graphics files, hydrodynamic model result data, network meteorological data, geographic information vector or raster files, network structured data, and network semi-structured data.

4. The system according to claim 1, characterized in that, When the system detects newly added water environment data, the data adaptation layer is also used to add a data adaptation module corresponding to the newly added water environment data; When a new computing model is identified in the system, the model library layer is used to add the new computing model, the model adaptation layer is used to add the adaptation function and function signature corresponding to the new computing model, and the function allocation layer is used to map the function signature corresponding to the new computing model to a sub-command.

5. A data processing method for multi-source heterogeneous water environment data, characterized in that, The system applied to any one of claims 1-4 includes the following steps: In response to the input second sub-command, the control function allocation layer determines the target adaptation function corresponding to the second sub-command based on the acquired second sub-command and the first sub-commands of each adaptation function. The second sub-command is generated by the host system based on the water conservancy calculation request, and the first sub-command is generated by the function allocation layer based on the function signature of the adaptation function. The function allocation layer is controlled to call the target adaptation function to load the computational model corresponding to the target adaptation function from the model library layer, and to load the data adaptation module corresponding to the target adaptation function from the data adaptation layer; The model adaptation layer and the data adaptation layer are controlled to process the water conservancy calculation request according to the calculation model corresponding to the target adaptation function, the data adaptation module corresponding to the target adaptation function, and the water environment data, so as to obtain the processing result.

6. The method according to claim 5, characterized in that, The control layer, model adaptation layer, and data adaptation layer process the water conservancy calculation request based on the calculation model corresponding to the target adaptation function, the data adaptation module corresponding to the target adaptation function, and the water environment data to obtain the processing result, including: The data adaptation layer is controlled to use the data adaptation module corresponding to the target adaptation function to transform the water environment data to obtain the target data; The control model adaptation layer processes the water conservancy calculation request according to the target data and the calculation model corresponding to the target adaptation function to obtain the processing result.

7. The method according to claim 6, characterized in that, The control model adaptation layer processes the water conservancy calculation request based on the target data and the calculation model corresponding to the target adaptation function to obtain the processing result, including: When there are N target adaptation functions, the model adaptation layer is controlled to call the N target adaptation functions in sequence according to the preset function order, so that the calculation model corresponding to the called target adaptation function processes the water conservancy calculation request according to the target data. The processing result of the nth calculation model is used as the input data of the (n+1)th calculation model to obtain the processing result corresponding to the Nth target adaptation function, where n is an integer greater than 0 and less than N.

8. The method according to claim 5, characterized in that, The model adaptation layer stores third-party dependency libraries for each computational model. The control layer, which manages the model adaptation layer and the data adaptation layer, processes the hydraulic computation request based on the computational model corresponding to the target adaptation function, the data adaptation module, and the water environment data to obtain the processing result, including: Load the third-party dependency library of the computational model corresponding to the target adaptation function from the model adaptation layer; The model adaptation layer and the data adaptation layer are controlled to process the water conservancy calculation request according to the calculation model corresponding to the target adaptation function, the third-party dependency library, the data adaptation module and the water environment data, so as to obtain the processing result.

9. The method according to claim 5, characterized in that, The method further includes: The control function allocation layer uses a command dispatch component to map the function signature of each of the adapted functions to the corresponding first sub-command.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method described in any one of claims 5-9.

Images