Streaming and batch integrated big data modeling method and device

Abstract

The invention relates to a flow and batch integrated big data modeling method and device, and belongs to the technical field of modeling, the method comprises the following steps: in a design stage, formulating a principle and a scheme of low-code design, and carrying out specific implementation steps based on the principle and the scheme; in the development stage, a rapid development framework is designed, the process of developing and debugging a small number of codes is visualized, JAR or SQL jobs of Flinks are continuously integrated in a visualized mode, in the release stage, JAR or SQL jobs of online Flinks are deployed in a visualized mode, workflow is constructed in a mode of arranging job nodes with low codes, and the JAR or SQL jobs of online Flinks are distributed in a visualized mode. And the workflow and related jobs and tasks are operated in a visual mode. According to the method, modeling of the batch processing business of the big data flow comprises low-code and visualization of design, development and release processes, so that the technical threshold of modeling is reduced, the model production efficiency is improved, and the delivery cycle of the business model is shortened; through visualization, the modeling process is transparent, simple and convenient.

Description

Technical Field

[0001] The present invention belongs to the field of modeling technology, and in particular relates to a stream-batch integrated big data modeling method and device. Background Art

[0002] Judging from the development of this field in the industry over the years, business modeling of big data stream batch processing usually relies on very professional programming skills and complex tool chains, such as Hadoop, Spark, Flink, etc. In the production process of the entire process of business modeling, although there are some visualization tools, most of them focus on data display, and the visualization support for modeling design, development and release processes is very limited. From the user's perspective, it is difficult to intuitively understand the process of data flow and model building. Therefore, for technical personnel such as development and operation and maintenance, the threshold and requirements of this work will appear to be particularly high. The authorized patent that is currently the closest to the present invention is: "Method and system for real-time processing of massive data on heterogeneous platforms in a batch-stream integrated process (CN117724706A)". The technical solution of this invention generally includes the following steps: deploying the environment and initializing the basic data; creating a new workspace, and creating a new workflow, process configuration and node configuration in the workspace, and publishing the workflow to the job scheduling service module; the job scheduling service module obtains the detailed information of the workflow to be scheduled, and performs job scheduling grouping according to the message nodes of the workflow, and publishes the scheduled job to the batch-stream integrated computing platform; the workflow service module provides a visual graphic component; relies on the asynchronous transmission mechanism of the message service component to achieve efficient big data processing and real-time data processing; realizes the full process from data acquisition to data output; adopts a change data capture solution to realize incremental loading of data in real time, thereby achieving efficient processing of massive data on heterogeneous platforms.

[0003] Existing problems: The high technical threshold for the full-process production of big data stream batch processing business modeling in the industry (including design, development, release, etc.) restricts the participation of a large number of non-professional or even low-professional users, reducing the popularity of big data applications. In addition, the transparency of the entire modeling process is insufficient, and it is difficult to quickly locate problems and optimize models from the user's perspective. On the other hand, after a detailed review of the relevant information of the patent listed above, it was found that the patent only provided a certain degree of low-code or visualization support in the process of modeling and release, while the process of modeling design and development still lacked low-code and visualization support. Therefore, from an overall perspective, like the current problems in the industry, each time a workflow instance of a new big data stream batch processing business model is produced, the technical threshold from design, development to release is still high, the production efficiency is still low, and the implementation cycle is still long. Summary of the invention

[0004] In view of the above-mentioned deficiencies in the prior art, the purpose of the invention is to provide a stream-batch integrated big data modeling method and device, which can greatly reduce the technical threshold of this work, improve the production efficiency of the entire process, and shorten the delivery cycle of the business model by providing low-code and visual support for the design, development, and release processes of big data stream batch processing business modeling. In addition, visualization makes the modeling process transparent, and it is also easier to continuously iterate and improve the user experience and model quality of related business models.

[0005] The first aspect of the present invention provides a stream-batch integrated big data modeling method, comprising:

[0006] In the design phase, formulate the principles and plans for low-code design, and carry out specific implementation steps based on the principles and plans;

[0007] During the development phase, we designed a rapid development framework, developed visualizations for debugging small amounts of code, and continuously integrated Flink's JAR or SQL jobs in a visual way.

[0008] During the release phase, Flink's JAR or SQL jobs are deployed online in a visual way, workflows are built by low-code orchestration of job nodes, and workflows and their related jobs and tasks are run in a visual way.

[0009] The second aspect of the present invention further provides a stream-batch integrated big data modeling device, comprising:

[0010] Development module and implementation step module: used to develop low-code design principles and plans during the design phase, and to carry out specific implementation steps based on low-code design principles and plans;

[0011] Design module, visual development module, and visual integration module: used to design a rapid development framework during the development phase, develop visualizations for debugging a small amount of code, and continuously integrate Flink's JAR or SQL jobs in a visual way;

[0012] Visual operation module: It is used to deploy Flink's JAR or SQL jobs in a visual way during the release phase, build workflows by orchestrating job nodes with low code, and run workflows and related jobs and tasks in a visual way.

[0013] According to a third aspect of the present invention, an electronic device is provided, comprising: a processor and a memory;

[0014] The processor calls the program or instruction stored in the memory to execute any one of the above-mentioned stream-batch integrated big data modeling methods.

[0015] In a fourth aspect of the present invention, a computer-readable storage medium is proposed, which stores a program or instruction, and the program or instruction enables a computer to execute any one of the above-mentioned stream-batch integrated big data modeling methods.

[0016] The beneficial effects of the present invention are as follows: In the design stage, the present invention formulates the principles and schemes of low-code design, and performs specific implementation steps based on the low-code design principles and schemes; in the development stage, a rapid development framework is designed, and visualization of the process of developing and debugging a small amount of code is developed and debugged, and Flink's JAR or SQL jobs are continuously integrated in a visual way; in the release stage, Flink's JAR or SQL jobs are deployed online in a visual way, and workflows are constructed by low-code orchestration of job nodes, and workflows and related jobs and tasks are run in a visual way. The present invention improves the efficiency of building business models and shortens the delivery cycle through the principles and schemes of low-code design, and performs specific implementation steps based on the principles and schemes of low-code design to extract and abstract reusable jobs from the design of the implemented workflow definition, and construct the workflow of the new business model by re-orchestration and combination of job nodes; a rapid development framework is designed to achieve the effect of low code, and the modeling process is made transparent through visualization, which is easy to iterate to improve the user experience and business model quality of related business models. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The accompanying drawings are only used to illustrate specific embodiments and are not considered to limit the present invention. In the entire drawings, the same reference symbols represent the same components. Obviously, the drawings described below are only some embodiments recorded in the embodiments of the present invention. For those of ordinary skill in the art, other drawings can also be obtained based on these drawings.

[0018] Figure 1 A flow-batch integrated big data modeling method diagram provided in an embodiment of the present invention;

[0019] Figure 2 A method diagram for formulating principles and solutions for low-code design provided by an embodiment of the present invention;

[0020] Figure 3 A method diagram of specific implementation steps based on low-code design principles and solutions provided for an embodiment of the present invention;

[0021] Figure 4 A schematic diagram of specific implementation steps based on low-code design principles and solutions provided for an embodiment of the present invention;

[0022] Figure 5 A method diagram for designing a rapid development framework provided by an embodiment of the present invention;

[0023] Figure 6 A diagram comparing the process of a rapid development framework for design provided by an embodiment of the present invention with that of the prior art;

[0024] Figure 7 A diagram of a method for visualizing the process of developing and debugging a small amount of code provided by an embodiment of the present invention;

[0025] Figure 8 A system implementation architecture diagram of a Web IDE service provided in an embodiment of the present invention;

[0026] Fig. 9 A diagram of a method for continuously integrating Flink's JAR or SQL jobs in a visual manner provided by an embodiment of the present invention;

[0027] Fig.10 A diagram of a stream-batch integrated big data modeling device provided in an embodiment of the present invention;

[0028] Fig.11 A schematic block diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0029] In order to enable those skilled in the art to better understand the technical solutions in the embodiments of the present invention, the technical solutions of the present invention will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all of the embodiments. It should be understood that these descriptions are only exemplary and are not intended to limit the scope of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in the field without making creative work should fall within the scope of protection of the present invention.

[0030] Furthermore, in the following description, descriptions of well-known structures and technologies are omitted to avoid unnecessarily obscuring the concepts disclosed in the present invention.

[0031] In the description of the present invention, the terms "first", "second", and "third" are used for descriptive purposes only and should not be understood as indicating or implying relative importance. The terms "installed", "connected", and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be internal communication between two components. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0032] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present invention. Instead, they are merely examples of methods and systems consistent with some aspects of the present invention as detailed in the appended claims.

[0033] The present invention proposes a stream-batch integrated big data modeling method, device, electronic device and storage medium. By providing low-code and visual support for the design and development processes of big data stream batch processing business modeling, the technical threshold of this work can be greatly reduced, the production efficiency of the entire process can be improved, and the delivery cycle of the business model can be shortened. In addition, visualization makes the modeling process transparent, and it is also easier to continuously iterate and improve the user experience and model quality of related business models.

[0034] Before introducing the embodiments of the present invention, professional terms involved in the present invention are first introduced.

[0035] Low-code: A visual application development method that uses less code and delivers applications at a faster speed. It can even automate the code that programmers do not want to develop in a more efficient way based on graphical drag-and-drop, parameterized configuration, etc., which can be called low-code.

[0036] Stream processing: refers to the immediate processing of real-time data streams so that a quick response can be made when the data arrives. This processing method is particularly important for scenarios that require instant feedback, such as financial transactions and IoT monitoring.

[0037] Batch processing: centralized processing of historical data to extract valuable information and insights. This processing method is usually used in scenarios such as data analysis and data mining, which helps companies gain a deeper understanding of the patterns and trends behind the data.

[0038] Stream-batch integrated big data modeling: a technical approach that requires processing real-time data and historical data simultaneously or combines the advantages of stream processing and batch processing, aiming to improve the efficiency, accuracy and value mining capabilities of big data processing.

[0039] Method Embodiment

[0040] Figure 1 A diagram of a stream-batch integrated big data modeling method provided in an embodiment of the present invention.

[0041] The first aspect of the present invention proposes a big data modeling method that integrates stream and batch Figure 1 ,include:

[0042] During the design phase, formulate low-code design principles and plans, and carry out specific implementation steps based on the low-code design principles and plans.

[0043] Specifically, in an embodiment of the present invention, principles and plans for low-code design are formulated, and methods for performing specific implementation steps based on the principles and plans for low-code design are described in detail below.

[0044] During the development phase, we design a rapid development framework, develop visualizations for debugging small amounts of code, and continuously integrate Flink's JAR or SQL jobs in a visual way.

[0045] Specifically, in an embodiment of the present invention, a rapid development framework is designed, a visualization of the process of developing and debugging a small amount of code is developed, and a method for continuously integrating Flink's JAR or SQL jobs in a visualized manner is described in detail below.

[0046] In the present invention, the component selection of the stream-batch integrated big data distributed computing processing engine is determined to be Flink. Flink is a unified stream-batch processing framework. From the perspective of Flink, all data can be considered as streams, stream data is unbounded streams, and batch data is bounded streams. Therefore, Flink can process both bounded batch data sets and unbounded real-time stream data, providing a unified programming model for batch processing and stream processing. The Flink code is mainly implemented in the Java language, and part of the code is implemented in the Scala language. For simple logical data processing models, the present invention adopts the method of writing Flink SQL code to call the SQL interface of the Flink engine to realize the stream batch processing of big data in the business model. For more complex data processing models, the method of writing Flink JAR code is adopted to call the DataStream API interface of the Flink engine to realize the stream batch processing of big data in the business model.

[0047] During the release phase, Flink's JAR or SQL jobs are deployed online in a visual way, workflows are built by low-code orchestration of job nodes, and workflows and their related jobs and tasks are run in a visual way.

[0048] Specifically, in an embodiment of the present invention, a JAR or SQL job of Flink is deployed online in a visual manner, and a workflow is constructed by low-code orchestration of job nodes, including: the interface of the modeling and publishing system receives the user's operation, creates a project, creates and configures a JAR job or SQL job of Flink, selects a Flink cluster for running the job, and saves and submits all jobs; retrieves the job components that have been previously stored and just deployed online that are needed this time from the job component library; orchestrates the job components as nodes into a workflow in DAG mode by dragging and dropping; configures the job parameters and dependencies of each node, sets the scheduling and alarm strategies, and saves and submits the workflow definition; configures the JAR job or SQL job of Flink, including: selecting the job type, uploading files, setting parameters and resources. The workflow and its related jobs and tasks are run in a visual manner, including: the interface of the modeling and publishing system receives the user's operation, sets the online strategy for the workflow; automatically generates and starts a new workflow instance according to the online strategy, and each job within the workflow will also be linked to generate and start executing new task instances; wherein the online strategy includes: immediate online and scheduled online.

[0049] Figure 2 A method diagram for formulating principles and solutions for low-code design provided in an embodiment of the present invention.

[0050] Furthermore, in the above-mentioned stream-batch integrated big data modeling method, the principles and solutions of low-code design are formulated, combined with Figure 2 , including six steps from S21 to S26:

[0051] S21: Identify multiple jobs in a large data stream batch processing process and classify them according to their functions, uses, and characteristics.

[0052] Specifically, in an embodiment of the present invention, the job categories include at least: jobs in the data collection and ingestion category, jobs in the data storage and caching category, jobs in the data preprocessing and conversion category, jobs in the data analysis and calculation category, and jobs in the data persistence and storage category.

[0053] For jobs in the Data Collection and Ingestion category:

[0054] Stream processing: collect data from various data sources (such as logs, sensors, message queues) in real time and process them immediately; batch processing: periodically load data in batches from data sources (such as databases, file systems) for subsequent processing.

[0055] For jobs in the Data Storage and Cache category:

[0056] Stream processing: Use message queues and cache systems (such as Redis and Kafka) to temporarily store real-time data streams for subsequent processing. Batch processing: Store data in a distributed file system (such as HDFS) or database for subsequent batch processing jobs;

[0057] For jobs in the Data Preprocessing and Transformation category:

[0058] Stream processing: cleans, transforms, filters, and other operations on real-time data streams to prepare data for subsequent analysis; batch processing: performs similar operations on batch data, but may involve more complex data transformation and cleaning logic.

[0059] For Data Analysis and Computing assignments:

[0060] Stream processing: real-time analysis such as aggregation, statistics, and machine learning model prediction on real-time data streams; Batch processing: large-scale data analysis, data mining, and machine learning model training on batch data;

[0061] For jobs in the data persistence and storage category:

[0062] Stream processing: persisting real-time analysis results or processed data to a database, data warehouse, or file system; Batch processing: storing the output of batch processing jobs in similar targets for subsequent query and analysis.

[0063] S22: For each type of job, abstract the commonalities and variable parts of the job to form a job template.

[0064] Specifically, in an embodiment of the present invention, a job template includes at least: job name and description, input and output, parameter configuration, logical flow, and dependencies and resources; job name and description: briefly describe the function and purpose of the job; input and output: define the input data and output data of the job, including data type, format, storage location, etc.; parameter configuration: list the configurable parameters in the job, such as processing window size, batch size, algorithm parameters, etc.; logical flow: describe the internal processing logic of the job, including data reading, processing, writing and other steps; dependencies and resources: list the dependencies and resources required for the job to run, such as library files, computing resources, storage resources, etc.

[0065] S23: Solidify and componentize the job template to obtain a job component.

[0066] Specifically, in an embodiment of the present invention, the job template is solidified and componentized to obtain a job component. The job component is a reusable component so that it can be directly called in a subsequent workflow. The componentization method at least includes: encapsulation into functions, methods or classes, modular design and configuration management; Encapsulation into functions, methods or classes: In a programming language, the job logic is encapsulated as functions, methods or classes, and a unified interface is provided for calling; Modular design: The job is split into multiple modules, each module implements a specific function, and a complete job is constructed by combining modules; Configuration management: The configuration parameters of the job are externalized and managed through configuration files or databases for easy modification and reuse.

[0067] S24: Design workflow to assemble job components.

[0068] Specifically, in the embodiments of the present invention, the factors considered in workflow design include at least: job dependencies, parallel and serial processing, error handling and recovery, and scalability and flexibility; job dependencies: clarify the dependencies between jobs to ensure the correctness and sequentiality of data flow; parallel and serial processing: reasonably arrange parallel and serial processing according to job characteristics and resource conditions to improve processing efficiency; error handling and recovery: design error handling mechanisms, such as retry, rollback, logging, etc., to ensure the stability and reliability of the workflow; scalability and flexibility: consider the scalability and flexibility of the workflow to cope with changes in future business needs.

[0069] S25: In a specific business scenario, instantiate a specific workflow and configure it according to the workflow design and job component library.

[0070] Specifically, in an embodiment of the present invention, the configuration includes at least: selecting job components, defining data flow, resource allocation and scheduling strategies; selecting job components: selecting the required job components from the component library and setting their parameters; defining data flow: defining the data flow path between various jobs by dragging and dropping or programming; resource allocation: allocating the required computing resources, storage resources and network resources to the workflow; scheduling strategy: setting the scheduling strategy of the workflow, such as timed execution, event triggering, etc.

[0071] S26: Testing and verification workflow;

[0072] Specifically, in the embodiment of the present invention, testing the workflow at least includes: including: functional testing, performance testing, stability testing, and testing and verifying the workflow to ensure that the workflow can meet business requirements and run stably.

[0073] Figure 3 A method diagram of specific implementation steps based on low-code design principles and solutions is provided for an embodiment of the present invention.

[0074] Figure 4 A schematic diagram of specific implementation steps based on low-code design principles and solutions provided for an embodiment of the present invention.

[0075] Furthermore, in the above-mentioned stream-batch integrated big data modeling method, specific implementation steps are carried out based on low-code design principles and solutions, combined with Figure 3 , including six steps from S31 to S36:

[0076] S31: Analyze existing workflow.

[0077] Specifically, in the embodiment of the present invention, combined with Figure 4 ,Analyzing existing workflows includes: identifying repetitive patterns after combing existing workflows; combing existing workflows is to list all workflows in the current system, and describe each workflow, including its goals, inputs, outputs, key steps and dependencies; identifying repetitive patterns is to find those jobs or task patterns that recur in different workflows during the combing process, and mark the commonalities and differences of these repetitive patterns.

[0078] S32: Abstract and define job templates.

[0079] Specifically, in the embodiment of the present invention, combined with Figure 4 , abstracting and defining job templates, including: defining job interfaces, creating job templates and documenting job templates; defining a job interface is to define a common job interface for each identified repetitive pattern, and the interface includes input parameters, output data, exception handling, etc.; creating a job template is to create a reusable job template based on the job interface. The template should contain the core logic of the job, but leave enough flexibility to adapt to different business scenarios; documenting the job template is to write detailed documents for each job template, including its function, purpose, input and output, parameter description and examples.

[0080] S33: Extract and package operation components.

[0081] Specifically, in the embodiment of the present invention, combined with Figure 4 , extract and encapsulate job components, including: code refactoring, encapsulation into components and management of job component versions; code refactoring is to refactor the code in the existing workflow, extract repeated logic into independent functions, methods or classes, and ensure that these functions, methods or classes conform to the previously defined job interface; encapsulation into components is to encapsulate the reconstructed functions, methods or classes into reusable job components, and the components should contain necessary configuration files, dependencies and test codes; management of job component versions is to manage job components with a version control system (such as Git) to ensure that each modification is recorded.

[0082] S34: Build component library.

[0083] Specifically, in the embodiment of the present invention, combined with Figure 4 , building a component library, including: classifying and storing job components in the component library and indexing the component library; classified storage is to classify and store job components in the component library according to the type and purpose of the job. The component library can be a local file system or a remote code repository; indexing and searching are used to index the component library to facilitate quick search and retrieval of required job components.

[0084] S35: Rearrange and combine operations.

[0085] Specifically, in the embodiment of the present invention, combined with Figure 4 , re-orchestrating and combining jobs, including: designing new business model workflows, selecting and configuring components, and integrating the configured job components into the new business model workflows for testing; designing new business model workflows is to design new workflows based on new business requirements. During the design process, priority is given to using existing job components; selecting and configuring components is to select the required job components from the component library and configure them according to the new business model. The configuration may include setting parameters, specifying input and output paths, defining dependencies, etc.; integrating the configured job components into the workflow of the new business model, and conducting comprehensive testing on the workflow of the new business model to ensure its correctness and performance.

[0086] S36: Continuously optimize job components and iterate job component library and workflow.

[0087] Specifically, in the embodiment of the present invention, combined with Figure 4 , continuously optimize job components, iterate job component libraries and workflows, including: collect feedback on new workflows from users and business teams, continuously optimize job components based on feedback, and iterate job component libraries and workflows. Based on feedback, optimize and improve job components, update versions in the component library, and ensure that all related workflows are updated. Repeat these two steps of collecting feedback on new workflows from users and business teams and continuously optimizing job components based on feedback to continuously iterate and improve the design capabilities of job component libraries and workflows.

[0088] Figure 5 A method diagram for designing a rapid development framework provided in an embodiment of the present invention.

[0089] Figure 6 A flowchart comparing a rapid development framework provided in an embodiment of the present invention with the prior art.

[0090] Furthermore, in the above-mentioned stream-batch integrated big data modeling method, a rapid development framework is designed, combined with Figure 5, including three steps from S51 to S53:

[0091] S51: Set multiple parameters of the program in a common configuration file according to a specific format, abstract a common configuration template, and unify the program configuration.

[0092] Specifically, in the embodiment of the present invention, combined with Figure 6 The parameters in the public configuration file are divided into three categories: basic parameters of Flink jobs, data source connectors and other customized parameters, and clean pure SQL statements.

[0093] Among them, the basic parameters of Flink jobs include the parameters supported by the flinkrun command (such as -c / --class - the full class name of the main class where the program main method is located,

[0094] -m / --jobmanager——JobManager connection address, etc.), all standard flink parameters (parameters following the command -D, such as parallelism.default——program parallelism, taskmanager.numberOfTaskSlots——TaskManager task slot number, etc.), and FlinkSQL related configuration items (such as planner——TablePlanner, mode——TableMode, etc.).

[0095] The clean pure SQL statement part will retain the format of the entire SQL content, and the key point is to retain the line break. You can directly define multiple SQLs at one time and distinguish them through the unique ID set at the front, so that they can be referenced in the code through the ID of these SQLs. Of course, each SQL must follow the format and specifications specified by FlinkSQL. For the data source connector and other custom parameters, the rapid development framework has also carried out secondary development or adaptation and encapsulation for a series of data sources that Flink officially provides connectors and does not provide connectors. At the same time, it also provides more convenient extensibility in the development of configurations that support other custom parameters; for example: Flink officially provides connectors for Kafka, JDBC and other data sources. If the native Flink development method is adopted, a lot of connection information in the data source needs to be defined right away. All parameters are hard-coded in the code, which is very insensitive. The rapid development framework further encapsulates the Connectors of various data sources on the official website, shielding many details, simplifying the development steps, and making data reading and writing easier.

[0096] S52: Encapsulate the context execution environment and design an interface function for extension during the encapsulation process.

[0097] Specifically, in the embodiment of the present invention, combined with Figure 6 Among the objects of the native Flink context execution environment, there are mainly three core classes: StreamExecutionEnvironment, TableEnvironment, and StreamTableEnvironment. The development framework inherits and encapsulates them respectively. In the process of encapsulating the above three context execution environments, the development framework designs an interface function for extension. Its function is to use the design mode of the hook method to allow developers to further set more personalized parameters through code when necessary. Specifically, all the parsed parameters in the public configuration file and the initialized StreamExecutionEnvironment object, or the initialized TableConfig object in TableEnvironment, will be passed to the developer to complete more parameter settings.

[0098] From a design perspective, the hook method is used to intervene in the execution process, making the control behavior process more flexible and more in line with actual business needs, and its return value is generally suitable for the return value of the conditional branch statement; from a code perspective, it is an empty implementation of an abstract method or a method defined in an interface, and is also an implementation of the template method design pattern.

[0099] S53: Abstract the logic code for job verification, encapsulating job submission parameters, submitting the job, and asynchronously waiting for the job submission result.

[0100] Specifically, in the embodiments of the present invention, the logical codes for job verification, encapsulating job submission parameters, submitting jobs, and asynchronously waiting for job submission results are abstracted, and the coding work can be automatically processed or standardized and simplified by the framework, which greatly improves the development difficulty and efficiency.

[0101] Exemplary, combined Figure 6 In the prior art, the main coding process for native Flink jobs can be abstracted as: ①—>②—>③—>④, and all of them require developers to complete related coding work. However, through the rapid development framework of this application, most of the coding work in links ① and ④ is automatically processed or standardized and simplified by the rapid development framework. Only link ⑤, which is newly decomposed from link ④, together with the original links ② and ③, still requires developers to complete related coding work. Therefore, after the introduction of the framework, the main coding process of Flink jobs is roughly adjusted to: framework—>⑤—>②—>③—>framework, and both development difficulty and development efficiency will be improved.

[0102] Figure 7 A diagram of a method for visualizing the process of developing and debugging a small amount of code provided by an embodiment of the present invention.

[0103] Figure 8 A system implementation architecture diagram of a Web IDE service provided in an embodiment of the present invention.

[0104] Furthermore, in the above-mentioned stream-batch integrated big data modeling method, the visualization of the process of developing and debugging a small amount of code is combined with Figure 7 , including four steps from S71 to S74:

[0105] S71: Develop an integrated Web IDE service in the modeling development system to realize the visualization of the process of developing and debugging a small amount of code. The system implementation architecture of the Web IDE service includes: Web background service, browser, IDE kernel and plug-in process;

[0106] S72: When the browser on the modeling development system interface receives an operation from the user to access the service address of the Web backend service, the login interface of the Web IDE is displayed;

[0107] S73: After the login interface receives the account and password input by the user and completes the authentication, the Web background service loads the IDE kernel code, starts passing in custom parameters, and forwards the corresponding http request;

[0108] S74: The IDE kernel main process starts the plug-in process.

[0109] Specifically, in the embodiment of the present invention, combined with Figure 8The Web backend service is a lightweight backend service based on the Express framework. The Web backend service allows the IDE kernel to be provided as a service, enabling it to have the ability to be deployed on a server. Later, when the user accesses the service address of the Web backend through the browser on the modeling development system interface, he will first see the login interface of the Web IDE. This is the authentication capability provided by the Web backend service. After the user enters the account and password to complete the authentication, the Web background service starts to load the IDE kernel code; the IDE kernel, as a static server for front-end resources, continuously returns the resources required for front-end loading, such as html / js / css. Then, after the front-end loading is completed, a websocket request will be sent to the background. When the websocket connection is established for the first time, the http protocol is used as the carrier and forwarded to the IDE kernel process by the Web background service. After the connection is established, it has nothing to do with the Web background service. File access modification, terminal startup, plug-in startup, etc. are all communicated directly between the browser foreground and the IDE background through websocket. However, the main process of the IDE kernel is far from enough. Without plug-ins, the IDE kernel is no different from a simple online text editor. Functions such as debugging, code jumping, and highlighting must rely on the power of plug-ins, so the plug-in process needs to be started.

[0110] It should be understood that although the plug-in process is started by the IDE kernel main process, it communicates directly with the browser through the websocket protocol. At the same time, the plug-in process and the browser window correspond one to one. A main process can start multiple plug-in processes, but one window must correspond to a unique plug-in process. The Web background service and the IDE kernel actually run in one process, but logically they are independent.

[0111] Furthermore, in the above-mentioned stream-batch integrated big data modeling method, Flink's JAR or SQL job is continuously integrated in a visual manner, including: realizing the visualization of the continuous integration of Flink JAR or SQL job through the operation interface of the modeling development system.

[0112] Fig. 9 A diagram of a method for continuously integrating Flink's JAR or SQL jobs in a visual manner provided by an embodiment of the present invention.

[0113] Furthermore, in the above-mentioned stream-batch integrated big data modeling method, Flink's JAR or SQL job is continuously integrated in a visual way, combined with Fig. 9 , including six steps from S91 to S96:

[0114] S91: The operation interface of the modeling development system receives the configuration operation of the user, and the modeling development system automatically pulls the code, specifies the branch, and uses the Maven tool to automatically build the JAR file.

[0115] Specifically, in the embodiment of the present invention, a private Git code repository, a Maven private server, a Flink cluster, and all data source middleware or database environments involved in the specific business model requirements have been pre-deployed. The modeling development system can be integrated with the specified code repository. Users can use the configuration operations on the interface to let the system automatically pull the code, specify the branch, and use the Maven tool to automatically build the JAR file.

[0116] S92: Automatically trigger build and deployment actions when code is updated.

[0117] Specifically, in the embodiment of the present invention, the building and deployment actions are automatically triggered when the code is updated.

[0118] S93: After the building and deployment actions are completed, when the operation interface of the modeling development system receives an operation from the user to submit a job in FlinkJAR mode or SQL mode, the job in FlinkJAR mode or SQL mode is submitted.

[0119] Specifically, in the embodiment of the present invention, after the building and deployment actions are completed, the user can also submit a job in Flink JAR mode or SQL mode on the operation interface of the modeling development system.

[0120] In some embodiments, the configuration interface of a specific job can even specify different pom modules in a Maven project, and can also transparently set the basic parameters of the Flink job in the public configuration file in the project. In this way, developers can even directly integrate and develop multiple pom modules and multiple sets of main method entry main class codes in one project.

[0121] S94: In the continuous integration stage, through the project code, clone or modify the job configuration on the operation interface of the modeling development system, and submit various types of Flink jobs.

[0122] Specifically, in the embodiment of the present invention, when it comes to the continuous integration stage, the user can take the code of a project and directly clone or modify the configuration of the job through the operation interface of the modeling development system to complete the submission of a wide variety of Flink jobs of different sizes and types.

[0123] S95: Various types of Flink jobs are submitted and started in the Flink cluster;

[0124] S96: Monitor and track the operation status of each task in real time through the operating interface of the development system.

[0125] Specifically, in the embodiment of the present invention, after the job is submitted and started in the Flink cluster, the user can monitor and track the running status of each job in real time through the operation interface of the development system; the entire continuous integration process is very transparent, simple and convenient.

[0126] Device Embodiment

[0127] Fig.10 A diagram of a stream-batch integrated big data modeling device provided in an embodiment of the present invention.

[0128] In a second aspect of the present invention, a stream-batch integrated big data modeling device is also proposed, combining Fig.10 ,include:

[0129] Development module 101 and implementation step module 102: used to develop low-code design principles and plans during the design phase, and to perform specific implementation steps based on the low-code design principles and plans.

[0130] Design module 103, visualization development module 104 and visualization integration module 105: used to design a rapid development framework, develop and debug visualization of a small amount of code during the development phase, and continuously integrate Flink's JAR or SQL jobs in a visual way.

[0131] Visual operation module 106: used to deploy Flink's JAR or SQL job in a visual way during the release phase, build workflows by orchestrating job nodes with low code, and run workflows and their related jobs and tasks in a visual way.

[0132] According to a third aspect of the present invention, an electronic device is provided, comprising: a processor and a memory;

[0133] The processor calls the program or instruction stored in the memory to execute any one of the above-mentioned stream-batch integrated big data modeling methods.

[0134] In a fourth aspect of the present invention, a computer-readable storage medium is proposed, which stores a program or instruction, and the program or instruction enables a computer to execute any one of the above-mentioned stream-batch integrated big data modeling methods.

[0135] Fig.11 It is a schematic block diagram of an electronic device provided by an embodiment of the present invention.

[0136] like Fig.11As shown, the electronic device includes: at least one processor 1101, at least one memory 1102 and at least one communication interface 1103. The various components in the electronic device are coupled together through a bus system 1104. The communication interface 1103 is used to transmit information between external devices. It can be understood that the bus system 1104 is used to achieve connection and communication between these components. In addition to the data bus, the bus system 1104 also includes a power bus, a control bus and a status signal bus. However, for the sake of clarity, Fig.11 Various buses are labeled as bus system 1104 .

[0137] It can be understood that the memory 1102 in this embodiment can be a volatile memory or a non-volatile memory, or can include both volatile and non-volatile memories.

[0138] In some embodiments, the memory 1102 stores the following elements, executable units or data structures, or a subset thereof, or an extended set thereof: an operating system and application programs.

[0139] The operating system includes various system programs, such as a framework layer, a core library layer, a driver layer, etc., which are used to implement various basic services and process hardware-based tasks. The application includes various application programs, such as a media player (MediaPlayer), a browser (Browser), etc., which are used to implement various application services. A program that implements any method in a stream-batch integrated big data modeling method provided in an embodiment of the present invention can be included in the application.

[0140] In an embodiment of the present invention, the processor 1101 calls the program or instructions stored in the memory 1102, specifically, the program or instructions stored in the application, and the processor 1101 is used to execute the steps of each embodiment of a stream-batch integrated big data modeling method provided in an embodiment of the present invention.

[0141] In the design phase, formulate the principles and plans for low-code design, and carry out specific implementation steps based on the principles and plans;

[0142] During the development phase, we designed a rapid development framework, developed visualizations for debugging small amounts of code, and continuously integrated Flink's JAR or SQL jobs in a visual way.

[0143] Any method in the stream-batch integrated big data modeling method provided in an embodiment of the present invention can be applied to the processor 1101, or implemented by the processor 1101. The processor 1101 can be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by an integrated logic circuit of hardware in the processor 1101 or an instruction in the form of software. The above-mentioned processor 1101 can be a general-purpose processor, a digital signal processor (Digital Signal Processor, DSP), an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), a field programmable gate array (Field Programmable Gate Array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc.

[0144] The steps of any method in a stream-batch integrated big data modeling method provided by an embodiment of the present invention can be directly embodied as being executed by a hardware decoding processor, or being executed by a combination of hardware and software units in a decoding processor. The software unit can be located in a mature storage medium in the art such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 1102, and the processor 1101 reads the information in the memory 1102 and completes the steps of the method in combination with its hardware.

[0145] Those skilled in the art will appreciate that although some embodiments described herein include certain features included in other embodiments but not other features, the combination of features from different embodiments is meant to be within the scope of the present invention and to form different embodiments.

[0146] Those skilled in the art will appreciate that the description of each embodiment has its own emphasis, and for parts not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0147] Although the embodiments of the present invention are described in conjunction with the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present invention, and such modifications and variations all fall within the scope defined by the appended claims. The above are only specific embodiments of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art who is familiar with the technical field can easily think of various equivalent modifications or substitutions within the technical scope disclosed by the present invention, and these modifications or substitutions should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the protection scope of the claims.

[0148] The above are only specific embodiments of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed by the present invention, and these modifications or replacements should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be based on the protection scope of the claims.

Claims

1. A big data modeling method integrating stream and batch, characterized in that: include: In the design phase, formulate the principles and plans for low-code design, and carry out specific implementation steps based on the principles and plans; During the development phase, we designed a rapid development framework, developed visualizations for debugging small amounts of code, and continuously integrated Flink's JAR or SQL jobs in a visual way. During the release phase, Flink's JAR or SQL jobs are deployed online in a visual way, workflows are built by low-code orchestration of job nodes, and workflows and related jobs and tasks are run in a visual way.

2. The method for big data modeling integrating stream and batch according to claim 1, characterized in that: The principles and plans for developing low-code designs include: Identify multiple jobs in a large data stream batch processing flow and classify them according to their functions, uses, and characteristics; For each type of job, abstract the common and variable parts of the job to form a job template; Solidify and componentize the job template to obtain the job component; Design workflows to assemble job components; In specific business scenarios, instantiate specific workflows and configure them based on workflow design and job component libraries; Testing and validation workflows; The job categories include at least: jobs in the data collection and ingestion category, jobs in the data storage and caching category, jobs in the data preprocessing and conversion category, jobs in the data analysis and calculation category, and jobs in the data persistence and storage category; The job template includes at least: job name and description, input and output, parameter configuration, logical flow, dependencies and resources; The componentization approach includes at least: encapsulation into functions, methods or classes, modular design and configuration management; Factors to consider in workflow design include at least: job dependencies, parallel and serial processing, error handling and recovery, and scalability and flexibility; Configuration, including at least: selecting job components, defining data flow, resource allocation and scheduling strategies; The testing workflow shall at least include: functional testing, performance testing, and stability testing.

3. The method for big data modeling integrating stream and batch according to claim 1, characterized in that: The specific implementation steps based on low-code design principles and solutions include: Analyze existing workflows; Abstraction and definition job templates; Extract and package job components; Build component libraries; Rearrange and combine assignments; Continuously optimize job components, iterate job component library and workflow; Among them, analyzing existing workflows includes: identifying repetitive patterns after combing through existing workflows; Abstract and define job templates, including: define job interfaces, create job templates, and document job templates; Extract and encapsulate job components, including code refactoring, encapsulation into components, and version management of job components; Building a component library, including: classifying and storing the job components in the component library, and establishing an index for the component library; Re-arrange and combine jobs, including: designing new business model workflows, selecting and configuring components, and integrating configured job components into the new business model workflows for testing; Continuously optimize job components, iterate job component libraries and workflows, including: collecting feedback about new workflows from users and business teams, continuously optimizing job components based on feedback, and iterating job component libraries and workflows.

4. The method for modeling big data by integrating stream and batch processing according to claim 1, characterized in that: Design a rapid development framework, including: Set multiple parameters of the program in a common configuration file according to a specific format, abstract a common configuration template, and unify the program configuration; Encapsulate the context execution environment, and design an interface function for extension during the encapsulation process; Abstract the logic code for job verification, encapsulating job submission parameters, submitting jobs, and asynchronously waiting for job submission results.

5. The method for big data modeling integrating stream and batch processing according to claim 1, characterized in that: Develop visualizations for debugging small amounts of code, including: In the modeling development system, we develop an integrated Web IDE service to realize the visualization of the process of developing and debugging a small amount of code. The system implementation architecture of the Web IDE service includes: Web background service, browser, IDE kernel and plug-in process; When the browser on the modeling development system interface receives the user's operation of accessing the service address of the Web backend service, the login interface of the Web IDE is displayed; After the login interface receives the account and password entered by the user and completes the authentication, the Web background service loads the IDE kernel code, passes in the custom parameters at startup, and forwards the corresponding http request; The IDE kernel main process starts the plug-in process.

6. The method for big data modeling integrating stream and batch processing according to claim 1, characterized in that: Continuously integrate Flink's JAR or SQL jobs in a visual way, including: realizing the visualization of the continuous integration of Flink JAR or SQL jobs through the operation interface of the modeling development system.

7. The method for big data modeling integrating stream and batch processing according to claim 6, characterized in that: Continuously integrate Flink's JAR or SQL jobs in a visual way, including: The operation interface of the modeling development system receives the user's configuration operation, and the modeling development system automatically pulls the code, specifies the branch, and uses the Maven tool to automatically build the JAR file; Automatically trigger build and deployment actions when code is updated; After the build and deployment actions are completed, when the operation interface of the modeling development system receives the user's operation to submit a job in FlinkJAR mode or SQL mode, the job in FlinkJAR mode or SQL mode is submitted; In the continuous integration stage, we clone or modify the job configuration on the operation interface of the modeling development system through the project code and submit various types of Flink jobs. Various types of Flink jobs are submitted and started in the Flink cluster; Monitor and track the operation status of each operation in real time through the operation interface of the development system.

8. A big data modeling device integrating stream and batch, characterized in that: include: Development module and implementation step module: used to develop low-code design principles and plans during the design phase, and to carry out specific implementation steps based on low-code design principles and plans; Design module, visual development module, and visual integration module: used to design a rapid development framework during the development phase, develop visualizations for debugging a small amount of code, and continuously integrate Flink's JAR or SQL jobs in a visual way; Visual operation module: It is used to deploy Flink's JAR or SQL jobs in a visual way during the release phase, build workflows by orchestrating job nodes with low code, and run workflows and related jobs and tasks in a visual way.

9. An electronic device, characterized in that: include: Processor and memory; The processor is used to execute the stream-batch integrated big data modeling method as described in any one of claims 1 to 7 by calling the program or instruction stored in the memory.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores programs or instructions, and the programs or instructions enable a computer to execute a stream-batch integrated big data modeling method as described in any one of claims 1 to 7.

Images