Method, device and equipment for fusing first frame and second frame
By loading and importing configuration files on the scheduling platform and building running containers, the compatibility problem between Flink and Spring Boot frameworks is solved, efficient migration and compatibility is achieved, memory and startup time is reduced, and code splitting is avoided.
Patent Information
- Application Number
- CN202411030416.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-30
- Publication Date
- 2024-12-03
AI Technical Summary
In the existing technology, the Spring Boot framework was incompatible when migrating to the Flink framework, which caused difficulty in business logic transformation and poor compatibility between frameworks, resulting in large memory and slow startup. It can only run in stand-alone mode, switch to Yarn. Various exceptions appear in cluster mode.
Obtain the business requirements set through the scheduling platform, start the running environment of the first framework, load the local configuration file into the memory, and import it into the environment variable configuration component of the second framework, build the running container, obtain the basic module and load it, and use the second framework's second framework's first framework to use the The operator of the second framework performs calculations in the container to achieve compatibility between frameworks.
It realizes compatibility between Flink and Spring Boot frameworks, reduces memory usage and startup time, can run stably in cluster mode, and avoids code splitting and maintenance costs.
Smart Images

Figure CN119065806A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of framework fusion, and in particular to a method, device and equipment for fusing a first framework and a second framework. Background Art
[0002] The Flink framework supports both unbounded data streaming and bounded data batch processing modes, and has high performance characteristics. However, it is not compatible with the Spring Boot framework. The research and development of existing banking businesses is generally based on the Spring Boot framework. If existing businesses are migrated to Flink, it means that the business logic of Spring Boot needs to be modified to decouple the existing business code from the various convenient components of Spring Boot. However, if business A exists in the form of a Spring Boot SDK package and is provided to other business modules B for embedding and use, there will be problems that are difficult to reconcile when migrating B to the Flink platform. Summary of the invention
[0003] In view of the above problems in the prior art, the purpose of this specification is to provide a method, device and equipment for integrating a first framework and a second framework to solve the incompatibility problem in the prior art when migrating the Spring Boot framework to the Flink framework.
[0004] In order to solve the above technical problems, the specific technical solutions of this specification are as follows:
[0005] On the one hand, this specification provides a method for integrating a first framework and a second framework, which is applied to a computing system, wherein the computing system includes a scheduling platform, a first framework, and a second framework, and the method includes:
[0006] The scheduling platform obtains a business requirement set and starts an operating environment of the first framework;
[0007] When the first framework obtains the corresponding operator as a target operator built into the second framework according to business requirements, the first framework loads a local configuration file in a specified directory into its memory, wherein the local configuration file is written in a language corresponding to the first framework;
[0008] The first framework imports the local configuration file in the memory into the environment variable configuration component of the second framework;
[0009] The second framework builds and runs a container according to the environment variable configuration component;
[0010] The first framework and the second framework obtain and load basic modules according to the business requirements;
[0011] The first framework enables the target operator of the second framework to calculate the business parameters in the business requirement in the running container and obtain the calculation result.
[0012] As an embodiment of the present specification, the first framework and the second framework obtain and load the basic module according to the business requirements, further comprising:
[0013] The basic module includes a data source component and a common component;
[0014] The first framework and the second framework both obtain and load at least one of the data source components according to the business requirements via the data source component factory;
[0015] The first framework and the second framework both obtain and load at least one of the common components according to the business requirements via a common component factory.
[0016] As an embodiment of the present specification, both the first framework and the second framework obtain and load at least one of the data source components according to the business requirements via the data source component factory, further comprising:
[0017] The data source component includes a data component, an SQL factory component and a transaction controller component, wherein the data component, the SQL factory component and the transaction controller component are written in the language corresponding to the first framework;
[0018] The first framework and the second framework obtain the data component, the SQL factory component and / or the transaction controller component in the data source component factory for loading according to the business requirements.
[0019] As an embodiment of the present specification, before the first framework acquires the corresponding operator as a target operator built into the second framework according to the business requirement, the method includes:
[0020] Generate task metadata according to the name of each business requirement in the business requirement set, the resources required by the business requirement and the corresponding business parameters;
[0021] Performing cross-frame orchestration of several task metadata according to an orchestration plan to obtain scheduling plan metadata, wherein the scheduling plan metadata represents a computing framework of each business requirement and several operators computing the business requirement within the framework, wherein the orchestration plan includes the business requirements executed within the first framework and the business requirements executed within the second framework;
[0022] The scheduling plan metadata is sent to the first framework, so that the first framework obtains an operator according to the scheduling plan metadata.
[0023] As an embodiment of the present specification, the first framework enables the target operator of the second framework to calculate the business parameters in the business requirement in the running container and obtain the calculation result, further including:
[0024] The first framework reads configuration information of business requirements from a database according to scheduling plan metadata;
[0025] The first framework enables the second framework to call the data source component and the interface exposed by the public component according to the configuration information;
[0026] The second framework imports the business parameters into a corresponding running container and operates on the business parameters in the business requirements, and the first framework obtains the operation results in the running container.
[0027] As an embodiment of the present specification, the scheduling platform obtains the service requirement set, further comprising:
[0028] The scheduling platform includes a physical buffer area, a buffer log, a channel module and a bus module;
[0029] The scheduling platform saves the acquired business requirement set to the physical cache area, and simultaneously saves the type of each business requirement and the business number corresponding to the business requirement in the cache log;
[0030] The channel module collects the business number in the cache log and publishes it to the bus module, so that the real-time calculation module of the first framework subscribes to the business number, and after subscription, obtains the business demand according to the business number for calculation.
[0031] As an embodiment of the present specification, the real-time computing module of the first framework subscribes to a service number, and after subscription, obtains the service demand according to the service number for calculation, further comprising:
[0032] The real-time computing module includes a coordinator, a timer and a blocking wake-up device;
[0033] After the coordinator subscribes to the service number, it obtains the service demand corresponding to the service number and a preset coordination list, wherein the coordination list represents the conflicting service demand and the compatible service demand corresponding to the service demand;
[0034] The coordinator obtains the execution status of all service requirements in the first framework and the second framework in response to the query indication of the timer, wherein the execution status includes executed, to be executed and being executed, wherein the query indication is sent by the timer according to the sampling period;
[0035] The coordinator controls the blocking wake-up device to block the business demand to be executed according to the coordination list and execution status of each business demand, or controls the blocking wake-up device to wake up the blocked business demand to be executed.
[0036] As an embodiment of the present specification, the coordinator controls the blocking wake-up device to block the business requirements to be executed according to the coordination list and execution status of each business requirement, or controls the blocking wake-up device to wake up the blocked business requirements to be executed, further comprising:
[0037] The coordinator determines, according to the coordination list of each business requirement, whether the business requirement to be executed conflicts with the business requirements being executed in the first framework and the second framework;
[0038] If there is a conflict, the business requirements to be executed will be blocked
[0039] If there is no conflict, the pending business requirement will be run, or the blocked pending business requirement will be awakened.
[0040] As an embodiment of the present specification, after the process of running the service requirement to be executed or waking up the blocked service requirement to be executed if there is no conflict, the process includes:
[0041] When the business demand is an operator operation within the first framework, the first framework splits the business demand into subtasks corresponding to the number of operators of the same type, and sends the subtasks to multiple idle operators of the same type within the first framework for calculation.
[0042] As an embodiment of the present specification, after the process of running the service requirement to be executed or waking up the blocked service requirement to be executed if there is no conflict, the process includes:
[0043] When the business demand is an operator operation in the first framework, the operator in the first framework divides the business demand into several subtasks, and uses multiple processes within itself to perform calculations respectively.
[0044] On the other hand, the present specification also provides a device integrating a first framework and a second framework, which is applied to a computing system, wherein the computing system includes a scheduling platform, a first framework and a second framework, and the device includes:
[0045] A demand acquisition unit, used for the scheduling platform to acquire a business demand set and start the operating environment of the first framework;
[0046] A file loading unit, configured to load a local configuration file in a specified directory into its memory when the first framework obtains a corresponding operator as a target operator built into the second framework according to business requirements, wherein the local configuration file is written in a language corresponding to the first framework;
[0047] A configuration import unit, used by the first framework to import the local configuration file in the memory into the environment variable configuration component of the second framework;
[0048] A container building unit, used for the second framework to build and run a container according to the environment variable configuration component;
[0049] A module loading unit, used for the first framework and the second framework to obtain and load basic modules according to the business requirements;
[0050] The result acquisition unit is used for the first framework to make the target operator of the second framework calculate the business parameters in the business demand in the running container and obtain the calculation result.
[0051] On the other hand, the present specification also provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program according to any one of the methods for integrating the first framework and the second framework.
[0052] On the other hand, the present specification also provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, any one of the methods for fusing the first framework and the second framework is implemented.
[0053] By adopting the above technical solution, the business requirement set is obtained through the scheduling platform, and the operating environment of the first framework is started, so that the business requirement set of each time period can be obtained and the operating environment of the first framework can be built;
[0054] When the first framework obtains the corresponding operator as the target operator built into the second framework according to business needs, the first framework loads the local configuration file in the specified directory into its memory, wherein the local configuration file is written in the language corresponding to the first framework, so that the local configuration file written by the first framework can be loaded into the memory of the first framework; the first framework imports the local configuration file in the memory into the environment variable configuration component of the second framework, so that the local configuration file written by the first framework can be imported into the environment variable configuration component of the second framework, so that the second framework adds the missing content and boot configuration at startup; the second framework builds and runs the container according to the environment variable configuration component, so that the second framework can The framework is built using the underlying files of the first framework, so that the operating environment of the second framework can be compatible with various components in the first framework, and the first framework can use several components in the second framework for calculations; the first framework and the second framework obtain and load basic modules according to the business needs, so that various basic modules required by the first framework and the second framework can be loaded from the same location, so that a set of basic modules can be maintained for use by the two frameworks, thereby increasing the versatility of the components; through the first framework, the target operator of the second framework is used to calculate the business parameters in the business needs in the running container and obtain the calculation results, the operator of the second framework can be used to calculate the business needs, and the calculation results can be directly obtained.
[0055] In order to make the above and other purposes, features and advantages of the present specification more obvious and easy to understand, the following specifically cites preferred embodiments and describes them in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS
[0056] In order to more clearly illustrate the embodiments of this specification or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this specification. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0057] Figure 1 An overall system diagram of a computing system according to an embodiment of the present specification is shown;
[0058] Figure 2 A schematic diagram showing the steps of a method for fusing a first frame and a second frame according to an embodiment of the present specification is shown;
[0059] Figure 3 A schematic diagram of a method for dividing business requirements according to an embodiment of this specification is shown;
[0060] Figure 4A schematic diagram of the control method of the embodiment of this specification is shown;
[0061] Figure 5 A schematic diagram of a device for integrating a first frame and a second frame according to an embodiment of the present specification is shown;
[0062] Figure 6 A schematic diagram of a computer device according to an embodiment of the present specification is shown.
[0063] Description of the accompanying symbols:
[0064] 101. Dispatching platform;
[0065] 102. First frame;
[0066] 103. Second frame;
[0067] 104. Database;
[0068] 105. Component library;
[0069] 501. Demand acquisition unit;
[0070] 502, file loading unit;
[0071] 503. Configuration import unit;
[0072] 504, container building unit;
[0073] 505, module loading unit;
[0074] 506. result acquisition unit;
[0075] 602. Computer equipment;
[0076] 604, processor;
[0077] 606. Memory;
[0078] 608, driving mechanism;
[0079] 610, input / output module;
[0080] 612. Input devices;
[0081] 614. Output devices;
[0082] 616. Presentation equipment;
[0083] 618. Graphical user interface;
[0084] 620, network interface;
[0085] 622, communication link;
[0086] 624. Communication bus. DETAILED DESCRIPTION
[0087] The following will be combined with the drawings in the embodiments of this specification to clearly and completely describe the technical solutions in the embodiments of this specification. Obviously, the described embodiments are only part of the embodiments of this specification, not all of the embodiments. Based on the embodiments in this specification, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this specification.
[0088] It should be noted that the terms "first", "second", etc. in the description and claims of this specification and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of this specification described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, device, product or equipment that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or equipment.
[0089] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties.
[0090] Some of the existing publicly available Spring Boot frameworks and Flink frameworks are often compatible by embedding the complete Spring Boot framework in the Flink framework. This approach is immature and the framework is heavy. It loads a lot of content and the content is loaded independently (there is a situation where similar functional components / objects / configurations are loaded repeatedly). It requires a lot of memory and has a slow startup time. According to tests, it can basically only run in stand-alone mode. When the operating environment is switched to Yarn cluster mode, various exceptions will occur, such as the failure to load the configuration file in the container of the SpringBoot framework, and the inability to find the class file when running the business requirements of the Flink framework.
[0091] like Figure 1 The overall system diagram of a computing system shown includes: a scheduling platform 101, a first framework 102, a second framework 103, a database 104 and a component library 105;
[0092] In this specification, the first framework 102 may be a Flink framework, and the second framework 103 may be a Spring Boot framework. When the scheduling platform 101 receives a number of business requirements input by the user, it may start the first framework 102. In this specification, the first framework 102 is the underlying framework, and the second framework 103 runs on the basis of the first framework 102. This specification uses custom components (different from embedding a complete Spring Boot framework in the Flink framework), which are lightweight and compatible with some Spring Boot framework technical components required by the Flink framework and business logic; the Flink framework and the Spring Boot framework configuration system are connected, and the two frameworks can share components / code objects with the same functions without repeated loading and wasting resources.
[0093] In this specification, the database 104 stores data required for various business needs.
[0094] In this specification, when loading components, both the Flink framework and the Spring Boot framework use and load components in a component library 105, in which the original components and custom components are stored, so that when loading, the two frameworks can share components in a component library 105 without repeated loading. And because the SpringBoot framework loads some configuration files in the Flink framework when it starts, the configuration system between the Flink framework and SpringBoot can be connected.
[0095] In this specification, the database stores the computing resources required for each business requirement and a preset coordination list.
[0096] In the prior art, there is an incompatibility problem when migrating the Spring Boot framework to the Flink framework, and the Spring Boot framework is usually used for batch processing. The SDK business package of the Spring Boot framework is often developed by other teams and new versions are released regularly. If the business logic of the SDK package of the Spring Boot framework is also transformed into a pure Flink framework, a new code branch independent of the Spring Boot framework will have to be maintained. When the Spring Boot framework releases a new version again, it needs to be transformed again and synchronized to the new branch. This split code branch and the workload of the dimension are unacceptable.
[0097] In order to solve the above problems, the embodiments of this specification provide a method for integrating the first framework and the second framework, which can avoid the problem that the Flink framework and the Spring Boot framework are incompatible and cannot be uniformly scheduled. Figure 2It is a schematic diagram of the steps of a method for integrating a first framework and a second framework provided in an embodiment of this specification. This specification provides method operation steps as described in the embodiment or flowchart, but may include more or fewer operation steps based on conventional or non-creative labor. The order of steps listed in the embodiment is only one way of executing the steps among many orders, and does not represent the only order of execution. When the actual system or device product is executed, it can be executed in the order of the method shown in the embodiment or the drawings or in parallel. Specifically, Figure 2 As shown, the method may include:
[0098] Step 201: The scheduling platform obtains a business requirement set and starts the operating environment of the first framework;
[0099] Step 202: When the first framework obtains the corresponding operator as a target operator built into the second framework according to the business requirement, the first framework loads the local configuration file in the specified directory into its memory, wherein the local configuration file is written in the language corresponding to the first framework;
[0100] Step 203: The first framework imports the local configuration file in the memory into the environment variable configuration component of the second framework;
[0101] Step 204: The second framework builds and runs a container according to the environment variable configuration component;
[0102] Step 205: The first framework and the second framework obtain and load basic modules according to the business requirements;
[0103] Step 206: The first framework instructs the target operator of the second framework to calculate the business parameters in the business requirement in the running container and obtain the calculation result.
[0104] When the first framework obtains the corresponding operator as the target operator built into the second framework according to business needs, the first framework loads the local configuration file in the specified directory into its memory, wherein the local configuration file is written in the language corresponding to the first framework, so that the local configuration file written by the first framework can be loaded into the memory of the first framework; the first framework imports the local configuration file in the memory into the environment variable configuration component of the second framework, so that the local configuration file written by the first framework can be imported into the environment variable configuration component of the second framework, so that the second framework adds the missing content and boot configuration at startup; the second framework builds and runs the container according to the environment variable configuration component, so that the second framework can The framework is built using the underlying files of the first framework, so that the operating environment of the second framework can be compatible with various components in the first framework, and the first framework can use several components in the second framework for calculations; the first framework and the second framework obtain and load basic modules according to the business needs, so that various basic modules required by the first framework and the second framework can be loaded from the same location, so that a set of basic modules can be maintained for use by the two frameworks, thereby increasing the versatility of the components; through the first framework, the target operator of the second framework is used to calculate the business parameters in the business needs in the running container and obtain the calculation results, the operator of the second framework can be used to calculate the business needs, and the calculation results can be directly obtained.
[0105] As an embodiment of the present specification, in step 201, the scheduling platform includes a physical buffer area, a buffer log, a channel module and a bus module;
[0106] The scheduling platform saves the acquired business requirement set to the physical cache area, and simultaneously saves the type of each business requirement and the business number corresponding to the business requirement in the cache log;
[0107] The channel module collects the business number in the cache log and publishes it to the bus module, so that the real-time calculation module of the first framework subscribes to the business number, and after subscription, obtains the business demand according to the business number for calculation.
[0108] Typically, streaming data is processed using the first framework and batch data is processed using the second framework.
[0109] In this specification, an important technical optimization in the business requirements of this specification is to migrate the batch processing of night account entries, such as interest, interest, penalty, delayed calculation, processing, quota update and bill calculation, to the real-time computing module in the new generation of the first framework. The purpose is to reduce the time for centralized batch running at night, leaving more processing time and emergency time for downstream systems to issue reports and accounting flow, which can improve production robustness and reduce failure rates. In addition, some centralized batch processing can be optimized to real-time streaming processing. In layman's terms, it means processing each piece of data generated, rather than accumulating a whole day's data for night processing. This avoids the pressure on the database and the increase in network resources, CPU, and memory resources during cumulative processing. The impact on real-time financial transactions in the shared database can be reduced.
[0110] In this specification, the business parameters required for business needs depend on the database (such as a distributed cluster based on Mysql), and the business parameters of the database can be regarded as two parts, one stored in the physical cache area (database hard disk), and the other stored in the cache log. In this specification, streaming data can be processed using cache logs, and the types of each business demand and the business numbers corresponding to the business demand are stored in the cache logs. Specifically, the cache logs are collected in real time by the channel module, published to the bus module using the business numbers of each business demand, and provided to the real-time computing module in the first framework for subscription. The data changes in the physical cache area will be subscribed to by the real-time computing module within a maximum of 1 second through such a synchronization mechanism. The original batch processing for the database can be converted into streaming data through this subscription method and processed in real time by the real-time computing module.
[0111] In addition, this manual can also hash the business number and publish it to the bus module. This subscription method also has a good avoidance mechanism for potential data processing conflicts caused by high concurrency. For example, customer data required by different business needs may be processed by multiple processes / threads at the same time, causing database updates to fail. Taking customer data as an example, using the business number as the sharding rule for processes and threads at two nodes can ensure that the data of the same customer is always processed by the same thread in the same process. The first advantage is the multi-partition publishing of the bus module. Publishing the full amount of data to different partitions in the manner of hashing the business number can ensure that the data of the same customer is only processed by the process in the same operator. The second advantage is that within the same operator, the data is grouped by the grouping operator and then processed by multiple threads, which can ensure that the data of the same customer is always processed by one thread.
[0112] In this manual, the input and output of the real-time computing module supports multiple formats, including file systems, Kafka, ES, HBase, relational databases, Hive, CSV files, Redis, etc. The flexible and diverse upstream and downstream data formats make this module widely applicable. It can support the migration of business logic based on the Spring Boot framework, the migration of bounded data scheduled batch processing based on Spring Batch, and even extract part of the batch processing to stream data processing, so as to reduce the total time consumption of the batch processing scheduling plan and leave more flexible processing time for batch processing upstream and downstream emergencies.
[0113] like Figure 3 The schematic diagram of the business requirement division method shown in the figure is an embodiment of the present specification, and before the first framework obtains the corresponding operator as the target operator built into the second framework according to the business requirement, it includes:
[0114] Step 301: Generate task metadata according to the name of each business requirement in the business requirement set, the resources required by the business requirement, and the corresponding business parameters;
[0115] The user adjusts the required business data, name (specific code entry class name in the batch processing project to be executed), parameters and other configuration information of the business needs through the front-end internal management page of the scheduling voucher. These configuration information will be saved in the database to form task metadata.
[0116] Step 302: perform cross-frame orchestration on a plurality of task metadata according to an orchestration plan to obtain scheduling plan metadata, wherein the scheduling plan metadata represents a computing framework of each business requirement and a plurality of operators computing the business requirement within the framework, wherein the orchestration plan includes business requirements executed within the first framework and business requirements executed within the second framework;
[0117] Multiple task metadata are connected in series and parallel through the page into a series of continuous task plans to jointly complete the scheduling plan metadata required for an overall business requirement. The scheduling plan metadata contains the execution order of each business requirement, series connection, parallel connection, execution time, manual execution / automatic execution and other information, as well as the operators required for each business requirement.
[0118] In this specification, the scheduling platform can use a scheduling plan to assign each business requirement in a business requirement group to different frameworks. For example, when the data type of the business requirement is batch processing, the business requirement is assigned to the second framework; when the data type of the business requirement is stream processing, the business requirement is assigned to the first framework.
[0119] Step 303: Send the scheduling plan metadata to the first framework, so that the first framework obtains operators according to the scheduling plan metadata.
[0120] In this specification, when the business requirement is stream processing, the first framework uses the scheduling plan metadata to obtain operators.
[0121] When the first framework obtains the corresponding operator as a target operator built into the second framework according to business requirements, the first framework loads a local configuration file in a specified directory into its memory, wherein the local configuration file is written in a language corresponding to the first framework;
[0122] The first framework imports the local configuration file in the memory into the environment variable configuration component of the second framework;
[0123] This manual sorts out the source code of the Flink framework and the Spring Boot framework, and uses a self-compiled local configuration file to solve the compatibility of the two frameworks. It works smoothly in both cluster mode and stand-alone mode. And after compatibility, only the required Spring Boot class files and technical components are loaded, so less memory is required and startup is faster.
[0124] Specifically, the local configuration file mainly includes the custom Spring Boot environment processor and the components required for the second framework to build the environment.
[0125] The environment processor is used to load necessary environment variables before Spring Boot dependency injection and use of technical components. Configuration file information includes the initialization configuration required for technical components and the configuration required for business components. These configurations come from the content read from the Flink framework configuration file when FlinkJob starts. The environment processor injects the part of the configuration file information in the Flink framework memory that Spring Boot requires into the Spring Boot environment variable configuration component to ensure that SpringBoot can effectively read the relevant configuration before dependency injection into business components and initialization of technical components (such as Spring Cache) to avoid exceptions.
[0126] As an embodiment of the present specification, the second framework builds and runs the container according to the environment variable configuration component. Specifically, it includes:
[0127] After the environment variable configuration component of the second framework is loaded by the local configuration file, the second framework can build a running environment under the first framework. In this specification, each container can run at least one operator, and several operators are combined to calculate and solve a business requirement.
[0128] As an embodiment of the present specification, the first framework and the second framework obtain and load the basic module according to the business requirements, further comprising:
[0129] The basic module includes a data source component and a common component;
[0130] The first framework and the second framework both obtain and load at least one of the data source components according to the business requirements via the data source component factory;
[0131] Specifically, the data source component includes a data component, an SQL factory component, and a transaction controller component, wherein the data component, the SQL factory component, and the transaction controller component are written in the language corresponding to the first framework;
[0132] The first framework and the second framework obtain the data component, the SQL factory component and / or the transaction controller component in the data source component factory for loading according to the business requirements.
[0133] The first framework and the second framework both obtain and load at least one of the common components according to the business requirements via a common component factory.
[0134] Specifically, in this specification, all environment variable configuration components can be loaded when the computing system is first opened, and do not need to be loaded repeatedly in subsequent computing. In subsequent computing, the first framework and the second framework can directly obtain the required components from the data source component factory and the public component factory.
[0135] As an embodiment of the present specification, the first framework enables the target operator of the second framework to calculate the business parameters in the business requirement in the running container and obtain the calculation result, further including:
[0136] The first framework reads configuration information of business requirements from a database according to scheduling plan metadata;
[0137] The first framework enables the second framework to call the data source component and the interface exposed by the public component according to the configuration information; specifically, the interface exposed by the public component of the data source component is called, and the underlying task pulls up the task from the task management service data source component to the running container. The core of the task management service is based on the second framework, which is an open source data integration, batch processing, and real-time processing tool set. It provides execution control functions for tasks in different operating environments, and can orchestrate, manage, and deploy tasks to the operating environment.
[0138] The second framework imports the business parameters into a corresponding running container and operates on the business parameters in the business requirements, and the first framework obtains the operation results in the running container.
[0139] like Figure 4The schematic diagram of the control method shown is an embodiment of the present specification. After the operation system is initially run, various business requirements within the operation system need to be controlled. Specifically,
[0140] Step 401, the real-time computing module includes a coordinator, a timer and a blocking wake-up device;
[0141] Step 402: After the coordinator subscribes to the service number, it obtains the service demand corresponding to the service number and a preset coordination list, wherein the coordination list represents the conflicting service demand and the compatible service demand corresponding to the service demand;
[0142] Step 403: The coordinator obtains the execution status of all service requirements in the first framework and the second framework in response to the query indication of the timer, wherein the execution status includes executed, to be executed, and being executed, wherein the query indication is sent by the timer according to the sampling period;
[0143] Step 404: The coordinator controls the blocking wake-up device to block the business demand to be executed according to the coordination list and execution status of each business demand, or controls the blocking wake-up device to wake up the blocked business demand to be executed.
[0144] Specifically, the coordinator determines whether the business requirement to be executed conflicts with the business requirements being executed in the first framework and the second framework according to the coordination list of each business requirement;
[0145] If there is a conflict, the business requirements to be executed will be blocked
[0146] If there is no conflict, the pending business requirement will be run, or the blocked pending business requirement will be awakened.
[0147] Specifically, when the business demand is an operator operation within the first framework, the first framework splits the business demand into subtasks corresponding to the number of operators of the same type, and sends the subtasks to multiple idle operators of the same type within the first framework for calculation.
[0148] Alternatively, when the business demand is an operator operation in the first framework, the operator in the first framework divides the business demand into a plurality of subtasks, and uses a plurality of processes within itself to perform calculations respectively.
[0149] like Figure 5 A schematic diagram of a device integrating a first framework and a second framework is shown, which is applied to a scheduling platform. The scheduling platform includes the first framework and the second framework. The device includes:
[0150] A demand acquisition unit 501 is used for the scheduling platform to acquire a business demand set and start the operating environment of the first framework;
[0151] A file loading unit 502, configured to load a local configuration file in a specified directory into its memory when the first framework obtains a corresponding operator as a target operator built into the second framework according to business requirements, wherein the local configuration file is written in a language corresponding to the first framework;
[0152] A configuration importing unit 503, used for the first framework to import the local configuration file in the memory into the environment variable configuration component of the second framework;
[0153] A container building unit 504, used for the second framework to build and run a container according to the environment variable configuration component;
[0154] A module loading unit 505, used for the first framework and the second framework to obtain and load basic modules according to the business requirements;
[0155] The result acquisition unit 506 is used for the first framework to instruct the target operator of the second framework to calculate the business parameters in the business requirement in the running container and obtain the calculation result.
[0156] When the first framework obtains the corresponding operator as the target operator built into the second framework according to business needs, the first framework loads the local configuration file in the specified directory into its memory, wherein the local configuration file is written in the language corresponding to the first framework, so that the local configuration file written by the first framework can be loaded into the memory of the first framework; the first framework imports the local configuration file in the memory into the environment variable configuration component of the second framework, so that the local configuration file written by the first framework can be imported into the environment variable configuration component of the second framework, so that the second framework adds the missing content and boot configuration at startup; the second framework builds and runs the container according to the environment variable configuration component, so that the second framework can The framework is built using the underlying files of the first framework, so that the operating environment of the second framework can be compatible with various components in the first framework, and the first framework can use several components in the second framework for calculations; the first framework and the second framework obtain and load basic modules according to the business needs, so that various basic modules required by the first framework and the second framework can be loaded from the same location, so that a set of basic modules can be maintained for use by the two frameworks, thereby increasing the versatility of the components; through the first framework, the target operator of the second framework is used to calculate the business parameters in the business needs in the running container and obtain the calculation results, the operator of the second framework can be used to calculate the business needs, and the calculation results can be directly obtained.
[0157] like Figure 6As shown, a computer device provided by an embodiment of this specification is used to run the method of fusing the first framework and the second framework described herein, and the computer device 602 may include one or more processors 604, such as one or more central processing units (CPUs), and each processing unit may implement one or more hardware threads. The computer device 602 may also include any memory 606, which is used to store any kind of information such as code, settings, data, etc. Non-limiting, for example, the memory 606 may include any one or more combinations of the following: any type of RAM, any type of ROM, flash memory device, hard disk, optical disk, etc. More generally, any memory can use any technology to store information. Further, any memory can provide volatile or non-volatile retention of information. Further, any memory can represent a fixed or removable component of the computer device 602. In one case, when the processor 604 executes an associated instruction stored in any memory or a combination of memories, the computer device 602 can perform any operation of the associated instruction. The computer device 602 also includes one or more drive mechanisms 608 for interacting with any memory, such as a hard disk drive mechanism, an optical disk drive mechanism, etc.
[0158] The computer device 602 may also include an input / output module 610 (I / O) for receiving various inputs (via input devices 612) and for providing various outputs (via output devices 614). A specific output mechanism may include a presentation device 616 and an associated graphical user interface (GUI) 618. In other embodiments, the input / output module 610 (I / O), input device 612, and output device 614 may not be included, and the computer device 602 may be used as a computer device in a network. The computer device 602 may also include one or more network interfaces 620 for exchanging data with other devices via one or more communication links 622. One or more communication buses 624 couple the components described above together.
[0159] The communication link 622 may be implemented in any manner, for example, through a local area network, a wide area network (e.g., the Internet), a point-to-point connection, etc., or any combination thereof. The communication link 622 may include any combination of hardwired links, wireless links, routers, gateway functions, name servers, etc. governed by any protocol or combination of protocols.
[0160] Corresponds to Figure 2-Figure 4 The method in the embodiment of the present specification also provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the above method are executed.
[0161] The embodiment of the present specification also provides a computer-readable instruction, wherein when the processor executes the instruction, the program therein causes the processor to execute the following Figure 2-Figure 4 The method shown.
[0162] It should be understood that in the various embodiments of this specification, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this specification.
[0163] It should also be understood that in the embodiments of this specification, the term "and / or" is only a description of the association relationship of the associated objects, indicating that three relationships can exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this specification generally indicates that the associated objects before and after are in an "or" relationship.
[0164] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed in this specification can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this specification.
[0165] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.
[0166] In the several embodiments provided in this specification, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, or it can be an electrical, mechanical or other form of connection.
[0167] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the embodiments of this specification.
[0168] In addition, each functional unit in each embodiment of this specification may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above integrated unit may be implemented in the form of hardware or in the form of software functional units.
[0169] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this specification is essentially or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of this specification. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.
[0170] Specific embodiments are used in this specification to illustrate the principles and implementation methods of this specification. The description of the above embodiments is only used to help understand the methods and core ideas of this specification. At the same time, for those skilled in the art, according to the ideas of this specification, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on this specification.
Claims
1. A method for fusing a first frame and a second frame, characterized in that: Applied to a computing system, the computing system includes a scheduling platform, a first framework and a second framework, the method includes: The scheduling platform obtains a business requirement set and starts an operating environment of the first framework; When the first framework obtains the corresponding operator as a target operator built into the second framework according to business requirements, the first framework loads a local configuration file in a specified directory into its memory, wherein the local configuration file is written in a language corresponding to the first framework; The first framework imports the local configuration file in the memory into the environment variable configuration component of the second framework; The second framework builds and runs a container according to the environment variable configuration component; The first framework and the second framework obtain and load basic modules according to the business requirements; The first framework enables the target operator of the second framework to calculate the business parameters in the business requirement in the running container and obtain the calculation result.
2. The method for fusing the first frame and the second frame according to claim 1, characterized in that: The first framework and the second framework obtain and load the basic module according to the business requirements, and further include: The basic module includes a data source component and a common component; The first framework and the second framework both obtain and load at least one of the data source components according to the business requirements via the data source component factory; The first framework and the second framework both obtain and load at least one of the common components according to the business requirements via a common component factory.
3. The method for fusing the first frame and the second frame according to claim 2, characterized in that: The first framework and the second framework both obtain and load at least one of the data source components according to the business requirements via the data source component factory, and further include: The data source component includes a data component, an SQL factory component and a transaction controller component, wherein the data component, the SQL factory component and the transaction controller component are written in the language corresponding to the first framework; The first framework and the second framework obtain the data component, the SQL factory component and / or the transaction controller component in the data source component factory for loading according to the business requirements.
4. The method for fusing the first frame and the second frame according to claim 1, characterized in that: Before the first framework acquires the corresponding operator as a target operator built into the second framework according to the business requirement, the method includes: Generate task metadata according to the name of each business requirement in the business requirement set, the resources required by the business requirement and the corresponding business parameters; Performing cross-frame orchestration of several task metadata according to an orchestration plan to obtain scheduling plan metadata, wherein the scheduling plan metadata represents a computing framework of each business requirement and several operators computing the business requirement within the framework, wherein the orchestration plan includes the business requirements executed within the first framework and the business requirements executed within the second framework; The scheduling plan metadata is sent to the first framework, so that the first framework obtains an operator according to the scheduling plan metadata.
5. The method for fusing the first frame and the second frame according to claim 1, characterized in that: The first framework instructs the target operator of the second framework to calculate the business parameters in the business requirement in the running container and obtain the calculation result, further including: The first framework reads configuration information of business requirements from a database according to scheduling plan metadata; The first framework enables the second framework to call the data source component and the interface exposed by the public component according to the configuration information; The second framework imports the business parameters into a corresponding running container and operates on the business parameters in the business requirements, and the first framework obtains the operation results in the running container.
6. The method for fusing the first frame and the second frame according to claim 5, characterized in that: The scheduling platform obtains the business requirement set, further comprising: The scheduling platform includes a physical buffer area, a buffer log, a channel module and a bus module; The scheduling platform saves the acquired business requirement set to the physical cache area, and simultaneously saves the type of each business requirement and the business number corresponding to the business requirement in the cache log; The channel module collects the business number in the cache log and publishes it to the bus module, so that the real-time calculation module of the first framework subscribes to the business number, and after subscription, obtains the business demand according to the business number for calculation.
7. The method for fusing the first frame and the second frame according to claim 6, characterized in that: The real-time calculation module of the first framework subscribes to the service number, and after the subscription, obtains the service demand according to the service number for calculation, further comprising: The real-time computing module includes a coordinator, a timer and a blocking wake-up device; After the coordinator subscribes to the service number, it obtains the service demand corresponding to the service number and a preset coordination list, wherein the coordination list represents the conflicting service demand and the compatible service demand corresponding to the service demand; The coordinator obtains the execution status of all service requirements in the first framework and the second framework in response to the query indication of the timer, wherein the execution status includes executed, to be executed and being executed, wherein the query indication is sent by the timer according to the sampling period; The coordinator controls the blocking wake-up device to block the business demand to be executed according to the coordination list and execution status of each business demand, or controls the blocking wake-up device to wake up the blocked business demand to be executed.
8. The method for fusing the first frame and the second frame according to claim 7, characterized in that: The coordinator controls the blocking wake-up device to block the business demand to be executed according to the coordination list and execution status of each business demand, or controls the blocking wake-up device to wake up the blocked business demand to be executed, further comprising: The coordinator determines, according to the coordination list of each business requirement, whether the business requirement to be executed conflicts with the business requirements being executed in the first framework and the second framework; If there is a conflict, the business requirements to be executed will be blocked If there is no conflict, the pending business requirement will be run, or the blocked pending business requirement will be awakened.
9. The method for fusing the first frame and the second frame according to claim 8, characterized in that: After the process of running the business requirement to be executed if there is no conflict, or waking up the blocked business requirement to be executed, includes: When the business demand is an operator operation within the first framework, the first framework splits the business demand into subtasks corresponding to the number of operators of the same type, and sends the subtasks to multiple idle operators of the same type within the first framework for calculation.
10. The method for fusing the first frame and the second frame according to claim 8, characterized in that: After the process of running the business requirement to be executed if there is no conflict, or waking up the blocked business requirement to be executed, includes: When the business demand is an operator operation in the first framework, the operator in the first framework divides the business demand into several subtasks, and uses multiple processes within itself to perform calculations respectively.
11. A device for integrating a first frame and a second frame, characterized in that: Applied to a computing system, the computing system includes a scheduling platform, a first framework and a second framework, and the device includes: A demand acquisition unit, used for the scheduling platform to acquire a business demand set and start the operating environment of the first framework; A file loading unit, configured to load a local configuration file in a specified directory into its memory when the first framework obtains a corresponding operator as a target operator built into the second framework according to business requirements, wherein the local configuration file is written in a language corresponding to the first framework; A configuration import unit, used by the first framework to import the local configuration file in the memory into the environment variable configuration component of the second framework; A container building unit, used for the second framework to build and run a container according to the environment variable configuration component; A module loading unit, used for the first framework and the second framework to obtain and load basic modules according to the business requirements; The result acquisition unit is used for the first framework to make the target operator of the second framework calculate the business parameters in the business demand in the running container and obtain the calculation result.
12. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the method for fusing the first frame and the second frame as described in any one of claims 1 to 10 is implemented.
13. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method for fusing a first frame and a second frame according to any one of claims 1 to 10 is implemented.
