Data processing method, electronic device, and computer-readable storage medium

Abstract

The application provides a data processing method, an electronic device and a computer readable storage medium, wherein the method comprises: constructing a calculation model according to a calculation model definition request; determining a target calculation engine to be adapted according to a calculation logic feature of the calculation model, and acquiring a target engine plug-in corresponding to the target calculation engine from an engine plug-in library according to an identifier of the target calculation engine; running the calculation model according to a data processing instruction of a user; and the calculation model interacts with the target calculation engine through the target engine plug-in to execute the data processing instruction during running. The application uses the engine plug-in as an intermediate adaptation layer to shield the differences between different calculation engines, so that the same calculation model can run on multiple calculation engines without modification. In this way, not only is multi-engine compatibility achieved, but also the repeated development and maintenance costs caused by engine binding are significantly reduced, effectively solving the problem that the calculation model is difficult to adapt to multiple calculation engines in the prior art.

Description

Technical Field

[0001] This application relates to the field of computer technology, and more specifically, to a data processing method, an electronic device, and a computer-readable storage medium. Background Technology

[0002] In increasingly complex computing scenarios, it is crucial to flexibly allocate the most suitable computing resources for data computation while maintaining the consistency of business logic.

[0003] In existing technologies, the data computing tool of the computing model is mainly built around the specific computing engine Flink. Therefore, no matter which definition method the computing model adopts, it is tightly coupled with the underlying computing engine, which results in the computing model itself not having universality. Once it is necessary to change or adapt to a new computing engine, the original model definition results cannot be transferred, and a lot of repeated development and modification must be carried out.

[0004] Therefore, existing computing models cannot be adapted to different types of computing engines, have data computing and processing problems that do not support multi-engine adaptation, and have high development and maintenance costs. Summary of the Invention

[0005] The purpose of this application is to address the shortcomings of the prior art by providing a data processing method, electronic device, and computer-readable storage medium to solve the problems that the existing computing models cannot adapt to different types of computing engines, have data computing processing that does not support multi-engine adaptation, and have high development and maintenance costs.

[0006] To achieve the above objectives, the technical solutions adopted in the embodiments of this application are as follows: In a first aspect, embodiments of this application provide a data processing method, the method comprising: Receive a user-input request for a computational model definition, the request including computational model definition information, which includes computational logic written in a structured query language or a release package containing encapsulated computational logic; Construct the computation model based on the computation model definition request; Based on the computational logic characteristics of the computational model, the target computational engine to be adapted is determined, and the target engine plugin corresponding to the target computational engine is obtained from the engine plugin library according to the identifier of the target computational engine. The engine plugin library stores engine plugins for various computational engines. Run the computational model according to the user's data processing instructions; When the computing model is running, it interacts with the target computing engine through the target engine plugin to execute the data processing instructions.

[0007] As one possible implementation, the computational model, during runtime, interacts with the target computational engine through the target engine plugin to execute the data processing instructions, including: When the computing model is running, it calls the processing interface provided by the target engine plugin according to the data processing instructions, so as to interact with the target computing engine through the processing interface and execute the data processing instructions.

[0008] As one possible implementation, constructing the computational model based on the computational model definition request includes: If the computational model definition information is computational logic written based on a structured query language, then the computational logic is syntax-validated, and after the validation passes, standard computational logic is output, and the computational model is constructed based on the standard computational logic. If the computational logic is a release package that has encapsulated computational logic, then the release package is decompressed, the computational logic is extracted from the release package, and the computational model is constructed based on the extracted computational logic.

[0009] As one possible implementation, before determining the target computing engine to be adapted based on the computational logic characteristics of the computing model, the method further includes: verifying the computing model; The step of determining the target computing engine to be adapted based on the computing logic characteristics of the computing model includes: after the verification is passed, determining the target computing engine to be adapted based on the computing logic characteristics of the computing model.

[0010] As one possible implementation, the verification of the computational model includes: The input data stream is simulated in a preset test environment, and the computational model is run to perform corresponding processing on the input data stream to obtain the processing results; The processing result is compared with the preset result. If they match, the verification is considered successful.

[0011] As one possible implementation, determining the target computing engine to be adapted based on the computational logic characteristics of the computational model includes: Obtain the computational logic features corresponding to each computing engine from the engine capability feature library; The computational logic features of the computational model are matched with the computational logic features of each computational engine, and the matched computational engine is taken as the target computational engine.

[0012] As one possible implementation, before retrieving the target engine plugin corresponding to the target computing engine from the engine plugin library based on the identifier of the target computing engine, the method further includes: Based on the characteristics of each computing engine, construct engine plugins corresponding to each computing engine; The identifiers of each computing engine are associated with the corresponding engine plugins and stored in the engine plugin library.

[0013] As one possible implementation, the method further includes: During operation, the computing model collects its runtime information in real time and displays it visually. The runtime information includes: runtime status, runtime fault information, data failure information, computing throughput, and resource utilization.

[0014] In a second aspect, embodiments of this application provide an electronic device, including: a processor, a storage medium, and a bus. The storage medium stores machine-readable instructions executable by the processor. When the electronic device is running, the processor communicates with the storage medium via the bus, and the processor executes the machine-readable instructions to perform the steps of the data processing method as described in any of the first aspects above.

[0015] Thirdly, embodiments of this application provide a computer-readable storage medium storing a computer program, which, when executed by a processor, performs the steps of the data processing method described in any of the first aspects above.

[0016] According to the data processing method, electronic device, and computer-readable storage medium of this application, a computational model definition request input by a user is received. Based on the request, a computational model is constructed, a target computational engine to be adapted is determined, and a target engine plugin corresponding to the target computational engine is obtained from an engine plugin library based on the target computational engine's identifier. The engine plugin library stores engine plugins for various computational engines. The computational model is run according to the user's data processing instructions. During runtime, the computational model interacts with the target computational engine through the target engine plugin to execute the data processing instructions. According to the embodiments of this application, after receiving computational logic provided by the user in the form of a structured query language or a packaged release, a standardized computational model is uniformly constructed. Based on the computational logic characteristics of the computational model, the target computational engine to be adapted is determined. Then, the target engine plugin serves as an intermediate adaptation layer, shielding the differences in interfaces, protocols, and operating mechanisms of the target computational engines. Regardless of which computational engine the computational model is built around, the computational model definition remains unchanged, eliminating the need to reconstruct the computational model due to a change in the computational engine. This makes the computational model universal and flexibly adaptable to multiple computational engines. Furthermore, when switching computing engines is required, there is no need to modify the original computing logic of the computing model. The computing model can be run simply by calling the compatible target engine plugin. This means that the plugin mechanism establishes timely interaction between the computing model and the computing engine, avoiding extensive development work caused by engine changes. Thus, while maintaining the computing logic of the computing model, it allows for flexible selection or switching of computing engines, ensuring efficient utilization of computing resources even in complex computing scenarios. In summary, this application not only achieves true multi-engine compatibility but also significantly reduces the repetitive development and maintenance costs caused by engine binding, effectively solving the problem of existing technologies where computing models are difficult to adapt to multiple computing engines. Attached Figure Description

[0017] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 A schematic diagram of the architecture of a data processing platform provided in an embodiment of this application is shown; Figure 2 A flowchart illustrating a data processing method provided in an embodiment of this application is shown; Figure 3 A flowchart illustrating a computational model verification method provided in an embodiment of this application is shown. Figure 4A flowchart illustrating an engine plugin construction method provided in an embodiment of this application is shown. Figure 5 A schematic diagram of the structure of an electronic device provided in an embodiment of this application is shown. Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. It should be understood that the accompanying drawings in this application are for illustrative and descriptive purposes only and are not intended to limit the scope of protection of this application. Furthermore, it should be understood that the schematic drawings are not drawn to scale. The flowcharts used in this application illustrate operations implemented according to some embodiments of this application. It should be understood that the operations in the flowcharts may not be implemented in sequence, and steps without logical contextual relationships may be reversed or implemented simultaneously. In addition, those skilled in the art, guided by the content of this application, may add one or more other operations to the flowcharts, or remove one or more operations from the flowcharts.

[0020] Furthermore, the described embodiments are merely some, not all, of the embodiments of this application. The components of the embodiments of this application described and illustrated herein can typically be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely to illustrate selected embodiments of the application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

[0021] It should be noted that the term "comprising" will be used in the embodiments of this application to indicate the presence of the features declared thereafter, but does not exclude the addition of other features.

[0022] To address the problems existing in the prior art, this application provides a data processing platform and a data processing method. Based on the data processing platform, this application defines the computational logic and constructs the computational model in the form of Structured Query Language (SQL) or a release package. Furthermore, based on the engine characteristics of each computational engine and the construction of standard interface implementation classes, it achieves multi-engine plug-in adaptation, building a flexible and lightweight engine support extensibility capability. Thus, by using engine plug-ins as an intermediate adaptation layer, the differences between different computational engines are shielded, allowing the same computational model to run on multiple computational engines without modification, effectively solving the problem in the prior art where computational models are difficult to adapt to multiple computational engines.

[0023] The data processing platform and data processing method provided in the embodiments of this application will be described in detail below.

[0024] Figure 1 A schematic diagram of the architecture of a data processing platform provided in an embodiment of this application is shown. (Refer to...) Figure 1 As shown, the data processing platform includes an interaction layer, a core capability layer, and a platform support layer. It realizes closed-loop management of the entire process from user operation to underlying engine adaptation, and has high scalability, strong security and good operation and maintenance manageability. It supports the efficient deployment and operation of real-time data processing tasks on different computing engines such as Flink, Spark Streaming, and Kafka Streams.

[0025] Optionally, the interaction layer mainly includes three functional modules: a modeling center, a management center, and a monitoring center. The modeling center supports users defining computing models through either SQL editing or importing from release packages, meeting the diverse needs of development and production environments. The management center manages computing models and data, providing version control, instance start / stop, resource management, and other operational operations. The monitoring center monitors hosts, computing instances, and data quality, and can visually display host status, instance operation, data quality, and data lineage in graphical form, enabling visualized monitoring and problem tracing.

[0026] Optionally, the core capability layer mainly includes four functional models: computational model definition, computational model management, data management, and computational instance monitoring. The computational model definition is used to define abstract data source access paths and field structures through datasets, preventing the leakage of real connection credentials during development and ensuring data security. It also uses standard SQL semantics to unify computational logic expression and integrates functions such as development verification, test verification, cluster submission, and model import / export, forming standardized development specifications. Computational model management is used to manage basic information, version, instance start / stop, instance runtime resources, instance runtime exceptions, and instance runtime logs of computational models, ensuring the controllability and traceability of computational models in multiple environments. Data management is used for data timeliness management, data quality verification, and data lineage analysis to improve data credibility and governance. Computational instance monitoring is used for monitoring the runtime status, runtime faults, data timeliness, computation throughput, backpressure, and resource usage of computational instances. It can collect and report the runtime status, fault alarms, throughput, backpressure, and resource utilization of computational instances in real time, providing data support for the stability of the data processing platform.

[0027] Optionally, the platform support layer mainly includes two functional models: adaptation management and public management, to provide underlying capability guarantees. Adaptation management is the core of achieving flexible multi-engine adaptation. It defines four standard interfaces: a standard interface for computational logic, a standard interface for engine parameter configuration, a standard interface for computational engine manipulation, and a standard interface for instance monitoring; as well as two log-related interfaces: a log output management interface and a log aggregation management interface. Furthermore, each computational engine can form an independent engine plugin by implementing the above interfaces. Through plugin hot-loading management and the plugin hot-loading mechanism provided by multi-engine adaptation plugins, it achieves non-restart dynamic expansion, greatly improving the flexibility and maintainability of the data processing platform. Public management is responsible for the unified management of organizations, tenants, users, roles, permissions, authentication, projects, menus, data sources, and metadata. This supports multi-tenant isolation and fine-grained access control, ensuring the security and compliance of the data processing platform.

[0028] Based on this, the data processing platform provided in this application decouples the computing model from the underlying computing engine. On the one hand, users can develop based on unified SQL semantics and secure dataset definitions to avoid sensitive information leakage, and ensure model quality through a full-process process from verification, testing, deployment to monitoring. On the other hand, with the help of plug-in engine adaptation mechanism and hot reloading capability, the data processing platform can seamlessly support multiple heterogeneous computing engines. The execution environment can be switched on demand without modifying the business computing logic of the computing model to adapt to different computing engines. At the same time, combined with multi-tenant permission management, data quality inspection and visualization monitoring, the data processing platform significantly reduces the development threshold and operation and maintenance complexity while ensuring security and compliance. It effectively solves the pain points of strong engine binding, high migration costs and development and operation and maintenance in the prior art, thereby achieving efficient, secure and maintainable real-time data processing capabilities.

[0029] The following is in conjunction with the above. Figure 1 The data processing platform shown herein provides a detailed description of the data processing method provided in the embodiments of this application.

[0030] Figure 2 A flowchart illustrating a data processing method provided in an embodiment of this application is shown. (Refer to...) Figure 2 As shown, this method is applied to the aforementioned data processing platform and specifically includes the following steps: S201. Receive the user's input request for a computational model definition.

[0031] Optionally, the computation model definition request includes computation model definition information, which includes computation logic written in a structured query language or a release package that encapsulates computation logic.

[0032] Optionally, the data processing platform can receive user-submitted computational model definition requests through a front-end graphical user interface. These requests contain computational model definition information, which can take the form of computational logic written in Structured Query Language (SQL) to describe the data source, transformation rules, and output target. Alternatively, it can be a pre-packaged release package containing complete computational logic, dependency libraries, and configuration files. Upon receiving the user-input computational model definition request, the data processing platform performs format validation, permission checks, and metadata extraction to lay the foundation for subsequent computational model building.

[0033] S202. Construct the computation model according to the computation model definition request.

[0034] Optionally, the computational model definition information contained in the computational model definition request is parsed. If the computational model definition information is computational logic written in SQL, a syntax parser is invoked to perform semantic analysis on the SQL-based computational logic, generate a logic execution plan, and encapsulate the logic execution plan into a standardized model object. If the computational model definition information is a release package, the release package is loaded into the runtime environment, the internal program of the release package is loaded, and the internal program of the release package is registered according to the predefined interface specification and the execution context required for the internal program to run is initialized, thereby encapsulating the release package into a standardized model object. In this way, regardless of which definition method the computational model adopts, a callable computational model can be constructed. Furthermore, after constructing the computational model, computational features of the computational model, such as operation type, data dependency, resource requirements, etc., can be extracted to generate model metadata. The model metadata is then associated with the computational model and persistently stored in the model repository, forming a schedulable and version-manageable computational model instance.

[0035] S203. Based on the computational logic characteristics of the computational model, determine the target computational engine to be adapted, and obtain the target engine plugin corresponding to the target computational engine from the engine plugin library according to the identifier of the target computational engine.

[0036] Optionally, the engine plugin library stores engine plugins for various computing engines.

[0037] Optionally, after constructing the computational model, the constructed computational model can be verified. After the verification is passed, the computational logic characteristics of the computational model are matched with the computational logic characteristics of each computational engine in the engine capability feature library to determine the target computational engine to be adapted. Then, based on the unique identifier of the target computational engine, the target engine plugin corresponding to the target computational engine is retrieved from the pre-built engine plugin library and loaded. The target engine plugin encapsulates all the adaptation logic required to communicate with the target computational engine.

[0038] S204. Run the calculation model according to the user's data processing instructions.

[0039] Optionally, when a user initiates a specific data processing task, such as processing log data from the most recent hour or performing prediction on a specified dataset, the pre-built computational model is scheduled to run. The computational model runtime environment initializes the execution context, injects the user-specified input data source, parameter configuration, and output target, and triggers the execution flow of the computational model.

[0040] S205. During runtime, the computational model interacts with the target computational engine through the target engine plugin to execute data processing instructions.

[0041] Optionally, during the operation of the computing model, the computing model does not directly call the native API of the underlying computing engine. Instead, it interacts with the target engine through the standardized interface provided by the previously loaded target engine plugin. The target engine plugin is responsible for converting the high-level semantic instructions of the computing model into a task description that the target computing engine can recognize, and handling interactive processes such as data serialization and resource allocation. The entire interaction is transparent to the computing model, ensuring the cross-engine compatibility and execution reliability of the computing model.

[0042] Based on this, the data processing method according to the embodiments of this application, after receiving the computational logic provided by the user in the form of a structured query language or a packaged release, uniformly constructs a standardized computational model. Based on the computational logic characteristics of the computational model, it determines the target computational engine to be adapted. Then, using a target engine plugin as an intermediate adaptation layer, it shields the differences in interfaces, protocols, and operating mechanisms of the target computational engines. Regardless of which computational engine the computational model is built around, the computational model definition remains unchanged, eliminating the need to reconstruct the computational model due to engine changes. This makes the computational model universal and flexibly adaptable to multiple computational engines. Furthermore, when switching computational engines, there is no need to modify the original computational logic of the computational model; simply calling the adapted target engine plugin is sufficient to run the computational model. That is, the plugin mechanism establishes timely interaction between the computational model and the computational engine, avoiding extensive development work caused by engine changes. Thus, while maintaining the computational logic of the computational model, it allows for flexible selection or switching of computational engines, enabling effective utilization of computational resources even in complex computational scenarios. In summary, this application not only achieves true multi-engine compatibility, but also significantly reduces the cost of repetitive development and maintenance caused by engine binding, effectively solving the problem that the computing model in the prior art is difficult to adapt to multiple computing engines.

[0043] It is worth noting that this application not only supports multiple model input formats, but also determines the target computing engine to be matched based on the computational logic characteristics of the computational model. Furthermore, by shielding the differences between underlying engines through a plug-in architecture, it significantly improves the scalability and deployment efficiency of the computational model. Simultaneously, the decoupling design between the computational model and the computing engine reduces maintenance costs and enhances the adaptability of the computational model to heterogeneous computing environments. This allows the same computational logic to run seamlessly on different computing engines, thereby greatly simplifying the development and maintenance complexity of the computational model while ensuring its execution performance.

[0044] As one possible implementation, step S202 above constructs a computational model according to the computational model definition request, including: if the computational model definition information is computational logic written based on a structured query language, then performing syntax validation on the computational logic, and outputting standard computational logic after the validation passes, and constructing a computational model based on the standard computational logic.

[0045] Optionally, when the computational model definition information is computational logic written in Structured Query Language (SQL), the original computational logic is syntax-validated by calling an SQL parser. This SQL parser, based on the SQL specifications supported by the target computational engine, checks whether the statements in the original computational logic conform to grammatical rules, such as keyword validity, expression structural integrity, table name and field existence, and data type compatibility. Simultaneously, it verifies whether the table names and fields referenced in the original computational logic are within the authorized scope and structurally consistent, based on the registered dataset definitions in the data processing platform (i.e., logical table metadata), thereby ensuring the executability and security of the computational logic.

[0046] Furthermore, after passing syntax validation, the original computational logic is converted into a standardized intermediate representation, such as an Abstract Syntax Tree (AST) or a Directed Acyclic Graph (DAG). This standardized intermediate representation is then encapsulated into standard computational logic, which includes not only executable SQL statements but also the dataset identifiers, field mappings, and metadata information. Based on this standard computational logic, a complete computational model is built, including binding input / output data sources, setting default runtime parameters, and associating tenants and permission contexts. This ensures the computational model maintains consistency, portability, and security during subsequent multi-engine adaptation, deployment, and monitoring phases, laying the foundation for seamless cross-engine operation.

[0047] As one possible implementation, step S202 above, which constructs a computational model according to the computational model definition request, includes: if the computational logic is a release package that has encapsulated computational logic, then decompress the release package, extract the computational logic from the release package, and construct a computational model based on the extracted computational logic.

[0048] Optionally, when the computation model definition information is a release package that encapsulates computation logic, such as a JAR, ZIP, or a deployment package in a custom format, the release package is subjected to security verification, such as signature verification and virus scanning. After the security verification is passed, a decompression operation is performed to extract the internal structure of the release package. The release package typically contains standardized directories, such as a compiled DAG description file, a list of dataset identifiers that are declared as referenced, metadata such as record author, version, and engine compatibility, as well as optional configuration parameter files.

[0049] Furthermore, after parsing the release package, the original computational logic is obtained from the compiled DAG description file or equivalent logical description. Based on the dataset identifiers listed in the dataset identifier list referenced in the declaration, the dataset definitions registered in the data processing platform are associated to ensure field structure matching and user access permissions. On this basis, the extracted computational logic, dataset reference relationships, runtime parameters, and metadata are integrated to construct a standardized computational model that conforms to the data processing platform specifications. This computational model retains the original semantic integrity of the release package and is incorporated into the data processing platform's unified version management, access control, and multi-engine adaptation system, thereby achieving secure import, consistency verification, and cross-environment deployment of development results.

[0050] Based on this, this application employs two methods to construct the computational model: writing computational logic based on SQL or importing it through a release package, balancing flexibility and standardization. On the one hand, it allows developers to quickly define and debug computational logic directly using standard SQL in the modeling center, lowering the development threshold and improving iteration efficiency. On the other hand, it allows mature, tested models to be imported into the production environment with a single click through pre-packaged release packages, ensuring consistency between development and deployment. Furthermore, both computational model construction methods rely on a unified dataset definition mechanism, effectively shielding sensitive information from the underlying data source and ensuring security. Simultaneously, the generated computational model structure is standardized, allowing seamless integration with multi-engine adaptation, version management, quality verification, and monitoring systems, significantly improving the portability and maintainability of the computational model.

[0051] As one possible implementation, before determining the target computing engine to be adapted based on the computational logic characteristics of the computing model, the method also includes: verifying the computing model.

[0052] Figure 3 A flowchart illustrating a computational model verification method provided in an embodiment of this application is shown. (Refer to...) Figure 3 As shown, the above steps validate the computational model, specifically including the following steps: S301. Simulate the input data stream in the preset test environment and run the calculation model to perform corresponding processing on the input data stream to obtain the processing results.

[0053] For example, a preset test environment is constructed that is isolated from the deployment environment provided by the data processing platform but has the same configuration. This preset test environment can simulate the input data stream in real scenarios, such as time series data, batch data, or streaming data. Then, the computing model to be verified is loaded into the preset test environment and the computing model is run to perform a complete processing flow on the input data stream, including data preprocessing, execution of computing logic, and output of results, and finally the actual processing result of the computing model is obtained.

[0054] S302. Compare the processing result with the preset result. If they match, the verification is confirmed to be successful.

[0055] For example, the preset results are typically generated based on historical data of known correct behavior, mathematical derivations, or authoritative references, and undergo rigorous verification. The actual processing results obtained above are compared item by item with the preset results. If all key indicators meet the consistency requirements, the computational model is deemed to have passed verification and is qualified to enter the subsequent adaptation stage; otherwise, it is marked as verification failure and needs to be returned for correction. The comparison can employ various strategies such as exact matching, numerical tolerance range, or structural consistency, and the specific strategy can be selected according to the model output type.

[0056] Based on this, by introducing the aforementioned verification mechanism before adapting to the target computing engine, this application can effectively ensure the logical correctness and functional integrity of the computing model itself, avoiding invalid adaptation work or erroneous inference results due to model defects. This not only improves the reliability and efficiency of the overall adaptation process but also significantly reduces debugging costs and deployment risks caused by model errors, providing a solid quality guarantee for cross-platform migration and multi-engine compatibility.

[0057] Furthermore, based on the computational logic characteristics of the computational model, the target computational engine to be adapted is determined, including: after the verification is passed, the target computational engine to be adapted is determined based on the computational logic characteristics of the computational model.

[0058] Optionally, the computational logic features corresponding to each computing engine are obtained from the engine capability feature library, and the computational logic features of the computing model are matched with the computational logic features corresponding to each computing engine, and the matched computing engine is used as the target computing engine.

[0059] For example, after the verification is passed, in order to determine the target computing engine to be adapted, the engine capability feature library can be accessed. This engine capability feature library records in detail the functional characteristics, performance indicators, and supported data processing types of each computing engine. Based on this, the computing logic characteristics of the computing model itself, such as the required types of operations, data parallelism requirements, and memory usage, are matched and analyzed one by one with the computing logic characteristics of each computing engine obtained from the engine capability feature library. The performance of different computing engines in executing specific computing tasks is evaluated and compared, thereby finding a suitable target computing engine for the computing model.

[0060] Based on this, it is ensured that the computing model can run on the target computing engine that matches its characteristics, thereby achieving the best execution efficiency and resource utilization. This not only helps to improve the performance of the computing model, but also simplifies the process of computing model migration and optimization, provides a guarantee for the cross-platform compatibility and flexibility of the computing model, and promotes the rapid deployment and iterative updates of the computing model.

[0061] As one possible implementation, in step S205 above, the computing model interacts with the target computing engine through the target engine plugin during runtime to execute data processing instructions. This includes: during runtime, the computing model calls the processing interface provided by the target engine plugin according to the data processing instructions, so as to interact with the target computing engine through the processing interface and execute the data processing instructions.

[0062] Optionally, the target engine plugin encapsulates the protocols, API call methods, and data format conversion logic required for communication with the target computing engine. When the computing model receives data processing instructions during runtime, such as performing a certain type of aggregation, filtering, or machine learning inference task, it calls the corresponding processing interface in the target engine plugin through a predefined interface specification. The target engine plugin is used to translate the high-level semantic instructions of the computing model into low-level operation commands that the target computing engine can understand, and complete interactive processes such as data serialization, task submission, status monitoring, and result feedback, thereby achieving seamless collaboration between the computing model and the target computing engine.

[0063] Optionally, to improve versatility and scalability, target engine plugins typically follow a unified plugin architecture, such as based on interface abstraction or dependency injection, allowing the same computing model to be adapted to multiple computing engines simply by switching between different engine plugins. Additionally, engine plugins can include runtime optimization strategies, such as batch merging, memory reuse, or asynchronous scheduling, to further improve execution efficiency.

[0064] Based on this, this application introduces an engine plugin mechanism to decouple the computational model from the underlying computational engine, significantly enhancing the flexibility and maintainability of the data processing platform. On the one hand, when it is necessary to replace or upgrade the computational engine, only the corresponding engine plugin needs to be replaced or updated, without modifying the logic of the computational model itself. On the other hand, it supports dynamic adaptation of the same computational model across multiple computational engines, facilitating the selection of the optimal execution backend based on actual deployment environments such as resource constraints and performance requirements, thereby improving the overall adaptability and deployment efficiency of the data processing platform.

[0065] Figure 4 A flowchart illustrating an engine plugin construction method provided in an embodiment of this application is shown. (Refer to...) Figure 4 As shown, before retrieving the target engine plugin corresponding to the target computing engine from the engine plugin library based on the identifier of the target computing engine, the method further includes the following steps: S401. Based on the characteristics of each computing engine, construct the corresponding engine plugin for each computing engine.

[0066] Optionally, the engine characteristics of various computing engines, such as programming interfaces, task scheduling mechanisms, data format requirements, dependent environments, and runtime behavior, are abstracted and analyzed. Based on these characteristics, a dedicated engine plugin is developed for each computing engine. This plugin encapsulates all the logic required to interact with the corresponding computing engine, such as task submission, status query, result retrieval, exception handling, and resource adaptation, and provides a unified standard interface to shield the differences between the underlying computing engines. Furthermore, engine plugins typically employ a modular design to facilitate independent testing, updates, and extensions.

[0067] S402. Associate the identifier of each computing engine with the corresponding engine plugin and store it in the engine plugin library.

[0068] Optionally, a structured engine plugin library can be established, binding the unique identifier of each computing engine, such as the engine name and version number, to the corresponding engine plugin and storing this information in the engine plugin library. Based on this, once the target computing engine to be matched is determined, the corresponding engine plugin can be quickly retrieved from the engine plugin library based on the target computing engine's identifier. That is, fast indexing and dynamic loading are achieved through the association between the computing engine's identifier and the engine plugin, enabling efficient location and loading of the correct engine plugin at runtime based on the target computing engine's identifier. Simultaneously, the engine plugin library can support version management, compatibility tags, and metadata descriptions, such as supported model types and dependencies, to provide a basis for engine plugin selection and verification.

[0069] Based on this, this application pre-builds and structures the mapping relationship between storage engine plugins and computing engines, enabling one-time development and multi-engine adaptation of the computing model, significantly improving the flexibility and maintainability of model deployment. When adding or switching computing engines, only the corresponding engine plugin needs to be registered without modifying the core scheduling logic. Simultaneously, the standardized plugin interface reduces integration complexity and improves the scalability and stability of the data processing platform.

[0070] As one possible implementation, the method also includes: collecting the runtime information of the computational model in real time during runtime and visually displaying the runtime information.

[0071] Optionally, the operational information includes: operational status, operational fault information, data invalidation information, computational throughput, and resource utilization. Operational status includes states such as initialization, running, paused, and completed; operational fault data includes fault logs or exception stacks generated during operation; and data invalidation information includes the validity verification results of input data, such as expired timestamps or missing fields.

[0072] For example, various operational information of the computing model is collected in real time during runtime, and the collected information is sent to the monitoring backend via message queues or telemetry protocols. This information is then dynamically displayed on a visualization interface, which presents a clear overview of the computing model's operation in the form of charts, dashboards, or topology diagrams, enabling operations personnel to quickly locate problems. Furthermore, to ensure low overhead and high real-time performance, asynchronous sampling and hierarchical reporting strategies can be adopted to avoid significant performance interference with the main computing logic. Simultaneously, when anomalies are detected, such as a sudden drop in throughput, resource overruns, or excessively high data loss rates, notifications or intervention measures are automatically triggered.

[0073] Based on this, this application enables developers and operations personnel to promptly grasp the health status and performance of the computing model by collecting and visualizing the operational information of the computing model in real time, quickly identify and diagnose potential faults or performance bottlenecks, and provide data support for model iteration or engine tuning, ultimately ensuring the reliability and service quality of data processing tasks.

[0074] This application also provides an electronic device 500, such as... Figure 5 The diagram shown is a structural schematic of an electronic device 500 provided in an embodiment of this application, including: a processor 501, a memory 502, and optionally, a bus 503. The memory 502 stores machine-readable instructions executable by the processor 501. When the electronic device 500 is running, the processor 501 and the memory 502 communicate via the bus 503. When the machine-readable instructions are executed by the processor 501, the steps of the data processing method described in any of the preceding claims are performed.

[0075] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, performs the steps of the data processing method described in any of the preceding claims.

[0076] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems and devices described above can be referred to the corresponding processes in the method embodiments, and will not be repeated here. In the several embodiments provided in this application, it should be understood that the disclosed systems, devices, and methods can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. Furthermore, multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed mutual coupling or direct coupling or communication connection can be through some communication interfaces; the indirect coupling or communication connection of devices or modules can be electrical, mechanical, or other forms.

[0077] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. If the functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes: USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, optical disks, and other media capable of storing program code.

[0078] The above are merely specific embodiments of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application.

Claims

1. A data processing method, characterized in that, include: Receive a user-input request for a computational model definition, the request including computational model definition information, which includes computational logic written in a structured query language or a release package containing encapsulated computational logic; Construct the computation model based on the computation model definition request; Based on the computational logic characteristics of the computational model, the target computational engine to be adapted is determined, and the target engine plugin corresponding to the target computational engine is obtained from the engine plugin library according to the identifier of the target computational engine. The engine plugin library stores engine plugins for various computational engines. Run the computational model according to the user's data processing instructions; When the computing model is running, it interacts with the target computing engine through the target engine plugin to execute the data processing instructions.

2. The method according to claim 1, characterized in that, During runtime, the computational model interacts with the target computational engine through the target engine plugin to execute the data processing instructions, including: When the computing model is running, it calls the processing interface provided by the target engine plugin according to the data processing instructions, so as to interact with the target computing engine through the processing interface and execute the data processing instructions.

3. The method according to claim 1, characterized in that, The step of constructing a computational model based on the computational model definition request includes: If the computational model definition information is computational logic written based on a structured query language, then the computational logic is syntax-validated, and after the validation passes, standard computational logic is output, and the computational model is constructed based on the standard computational logic. If the computational logic is a release package that has encapsulated computational logic, then the release package is decompressed, the computational logic is extracted from the release package, and the computational model is constructed based on the extracted computational logic.

4. The method according to claim 1, characterized in that, Before determining the target computing engine to be adapted based on the computing logic characteristics of the computing model, the method further includes: verifying the computing model; The step of determining the target computing engine to be adapted based on the computing logic characteristics of the computing model includes: after the verification is passed, determining the target computing engine to be adapted based on the computing logic characteristics of the computing model.

5. The method according to claim 4, characterized in that, The verification of the computational model includes: The input data stream is simulated in a preset test environment, and the computational model is run to perform corresponding processing on the input data stream to obtain the processing results; The processing result is compared with the preset result. If they match, the verification is considered successful.

6. The method according to claim 1, characterized in that, The step of determining the target computing engine to be adapted based on the computational logic characteristics of the computational model includes: Obtain the computational logic features corresponding to each computing engine from the engine capability feature library; The computational logic features of the computational model are matched with the computational logic features of each computational engine, and the matched computational engine is taken as the target computational engine.

7. The method according to claim 1, characterized in that, Before retrieving the target engine plugin corresponding to the target computing engine from the engine plugin library based on the identifier of the target computing engine, the method further includes: Based on the characteristics of each computing engine, construct engine plugins corresponding to each computing engine; The identifiers of each computing engine are associated with the corresponding engine plugins and stored in the engine plugin library.

8. The method according to claim 1, characterized in that, The method further includes: During operation, the computing model collects its runtime information in real time and displays it visually. The runtime information includes: runtime status, runtime fault information, data failure information, computing throughput, and resource utilization.

9. An electronic device, characterized in that, include: A processor and a memory, the memory storing machine-readable instructions executable by the processor, wherein when the electronic device is in operation, the processor executes the machine-readable instructions to perform the steps of the data processing method as described in any one of claims 1 to 8.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, performs the steps of the data processing method as described in any one of claims 1 to 8.

Images