Method, device and readable storage medium for processing data flow of service

Abstract

This invention provides a method, apparatus, and readable storage medium for processing business data streams. The method includes: receiving a business data stream; determining a preset processing strategy based on the data stream; processing the data stream based on the preset processing strategy; and executing the processed data stream according to a preset execution method. This invention enables the pre-processing of technical details of the business data stream according to a preset processing strategy, thereby isolating the technical details of the business process from the logic editing process. This allows developers to focus solely on developing the business logic without needing to consider the technical details, thus improving business development efficiency.

Description

Technical Field

[0001] This invention relates to the field of data stream processing technology, and more specifically, to a method, apparatus, and readable storage medium for processing business data streams. Background Technology

[0002] Currently, the processing of business data flows can be highly complex, involving multiple steps, intricate business logic, and various complex technical details, including but not limited to validation, transformation, splitting, merging, forwarding, and persistence. Existing technologies for processing such business data require developers to both develop the business logic and handle the technical details of the data flow. This technical handling often necessitates the introduction of parallel execution frameworks or big data processing platforms, requiring developers to learn additional knowledge of these complex frameworks or platforms. This places higher demands on developers, preventing them from focusing on business logic development and impacting development efficiency.

[0003] Therefore, how to propose a method for processing business data flow so that developers can focus more on business logic development without having to pay too much attention to technical details, thereby improving development efficiency, has become an urgent problem to be solved. Summary of the Invention

[0004] The present invention aims to solve at least one of the technical problems existing in the prior art or related art.

[0005] Therefore, the first objective of this invention is to provide a method for processing business data streams.

[0006] A second objective of the present invention is to provide a data stream processing apparatus for a business.

[0007] A third objective of this invention is to provide a data stream processing apparatus for a business.

[0008] A fourth objective of this invention is to provide a readable storage medium.

[0009] To achieve the above objectives, the first aspect of the present invention provides a method for processing a business data stream, comprising the following steps: receiving a business data stream; determining a preset processing strategy based on the data stream; processing the data stream based on the preset processing strategy; and executing the processed data stream according to a preset execution method.

[0010] The data stream processing method for services provided by this invention, upon receiving the data stream of the service to be processed, first obtains the preset processing strategy corresponding to the data stream. Specifically, the preset processing strategy is the technical detail processing procedure required for the data stream. That is, the preset processing strategy is a preset technical detail processing strategy. Then, the data stream is processed according to the determined preset processing strategy to realize the technical detail processing of the data stream, obtaining a technically detailed processed data stream, which can then be executed according to a preset execution method.

[0011] The technical solution of this application fully considers the problem in related technologies where, when processing complex business data flows, developers need to consider both complex business logic and various intricate technical details. These technical detail processing processes are often cumbersome, preventing developers from focusing on business logic development and impacting development efficiency. In this application, the system server pre-processes the business data flow according to a preset processing strategy, thus isolating the technical details from the logic editing process. This allows developers to concentrate solely on business logic development without worrying about technical details, thereby improving business development efficiency.

[0012] In addition, the data flow processing method for services in the above-mentioned technical solution provided by the present invention may also have the following additional technical features:

[0013] In the above technical solution, determining a preset processing strategy based on the data stream and processing the data stream based on the preset processing strategy specifically includes: determining a preset processing strategy based on the data stream; determining a corresponding preset process framework based on the preset processing strategy; processing the data stream based on the preset process framework; the preset process framework is developed based on the preset processing strategy; there are multiple preset processing strategies, and each preset processing strategy is correspondingly set with a preset process framework.

[0014] In this technical solution, the preset process framework is developed based on preset processing strategies. There are many preset processing strategies, and each preset processing strategy has a corresponding preset process framework. Based on one or more preset process strategies corresponding to the data stream, one or more preset process frameworks corresponding to those strategies are determined, and then the data stream is processed according to the preset process frameworks. This application provides a simple and easy-to-use development framework, which allows for the processing of technical details of the data stream, greatly improving development efficiency.

[0015] In one possible technical solution, multiple pre-defined process frameworks are connected using process coding to form a complete set of general business code. This business code contains the complete details of the data flow processing for the business. This allows the business code to be deployed to different platforms simultaneously, enabling multi-platform use. This avoids the situation where developers, while handling the technical details of the business, are limited to learning only one big data processing platform, preventing the business from being tied to that platform and unable to handle its technical details on other big data processing platforms.

[0016] Furthermore, the preset processing strategies include one or more of the following: proofreading strategy, conversion strategy, splitting strategy, merging strategy, forwarding strategy, persistence strategy, and exception handling strategy.

[0017] In the above technical solution, the execution of the processed data stream according to the preset execution method specifically includes: selecting a business executor based on the size of the data stream and / or the execution history; and executing the processed data stream through the business executor.

[0018] In this technical solution, the system server can automatically switch to the appropriate business executor based on the size of the data stream and / or execution history. This allows for automatic adaptation to different business data volumes, such as large data volumes with high concurrency and small data volumes with low concurrency, thereby ensuring the compatibility of business data volume with the infrastructure, reducing overall business operating costs, and protecting IT infrastructure investments.

[0019] Furthermore, business executors include serial executors, parallel executors, and distributed executors.

[0020] Serial actuators are suitable for simple scenarios, small data volumes, testing and debugging, etc. The system server executes the data stream step by step in the process sequence.

[0021] Parallel executors are suitable for medium-sized data scenarios. They use a parallel framework to send data to different system servers so that the data stream can be processed simultaneously.

[0022] Distributed executors are suitable for big data scenarios. When the amount of business data is too large, the local system server cannot handle the large amount of business data. In this case, it is necessary to send the common business code to different big data platforms to execute the data flow processing of the business.

[0023] In one possible technical solution, different preset execution frameworks can be developed based on the type of business executor. These preset execution frameworks include serial execution frameworks, parallel execution frameworks, and distributed execution frameworks. The system server can automatically switch between different preset execution frameworks based on the size of the data stream and / or the execution history.

[0024] In the above technical solution, the step of selecting a business executor based on the size of the data stream and / or its execution history includes: when the data stream has a matching execution history, selecting a business executor based on the matching execution history; when the data stream does not have a matching execution history, selecting a business executor based on the size of the data stream.

[0025] In this technical solution, the system server determines whether the data stream has a matching execution history. If so, it selects a suitable business executor based on the matching execution history. If a suitable business executor cannot be selected based on the matching execution history, then the size of the data stream needs to be determined.

[0026] Furthermore, the step of selecting the business executor based on the size of the data stream includes: selecting a serial executor as the business executor when the data stream is less than a first threshold; selecting a parallel executor as the business executor when the data stream is greater than the first threshold but less than a second threshold; and selecting a distributed executor as the business executor when the data stream is greater than the second threshold.

[0027] In this technical solution, when the data flow is less than a first threshold, a serial executor is selected as the business executor; when the data flow is greater than the first threshold but less than a second threshold, a parallel executor is selected; and when the data flow is greater than the second threshold, a distributed executor is selected. This allows for automatic adaptation to different business data volumes, such as large data volumes with high concurrency and small data volumes with low concurrency, thereby ensuring the adaptability of business data and facilities, reducing overall business operating costs, and protecting IT infrastructure investments.

[0028] In one possible technical solution, the system server contains an intelligent executor that can automatically select a suitable business executor or a preset execution framework based on the size of the data stream and / or the execution history.

[0029] In the above technical solution, the steps of executing the processed data stream through the business executor include: determining one or more servers that need to be used in the business execution process based on the data stream; and executing the processed data stream on the determined one or more servers through the business executor.

[0030] In this technical solution, the number of servers required during business execution can be determined in advance based on the data flow. Specifically, for medium-sized data scenarios, if the number of servers required during business execution is not determined in advance, the servers will process the data flow based on the existing number of servers, for example, using 3 servers for data flow processing. However, in this case, due to the slightly larger data volume, the efficiency of 3 servers for data flow processing is low, and an additional server needs to be considered for data flow processing. In the solution of this application, the appropriate number of servers to process the business data is directly determined based on the data flow size, and then the processed data flow is executed, realizing dynamic server scaling. This greatly ensures the adaptability of data and servers, making the best use of IT facilities.

[0031] In the above technical solution, the cost required to execute the data stream is calculated based on the computational load during the data stream processing.

[0032] In this technical solution, the system server can calculate the cost of the service based on the computational load during the data stream processing, such as CPU (central processing unit) usage and runtime, thereby fulfilling the demand for pay-as-you-go elastic computing.

[0033] The second aspect of the present invention provides a data stream processing apparatus for a business, comprising: a receiving unit for receiving a data stream; an acquisition unit for acquiring a preset processing strategy corresponding to the data stream; a processing unit for processing the data stream; and an execution unit for executing the processed data stream.

[0034] According to the data stream processing apparatus for services provided by the present invention, after the receiving unit receives the data stream of the service to be processed, the acquiring unit acquires the preset processing strategy corresponding to the data stream. Specifically, the preset processing strategy is the process of technical detail processing required for the data stream. That is, the preset processing strategy is a preset technical detail processing strategy. Then, the processing unit processes the data stream according to the determined preset processing strategy to realize the technical detail processing of the data stream, and obtains the technical detail processed data stream. Finally, the execution unit can execute the technical detail processed data stream according to a preset execution method. In the data stream processing apparatus for services provided by this application, the system server performs technical detail processing on the data stream of the service in advance according to the preset processing strategy. This isolates the technical detail process of the service from the logic editing process, allowing developers to focus only on the development of business logic without having to consider the technical detail processing issues, thereby improving the efficiency of service development.

[0035] The third aspect of the present invention provides a data stream processing apparatus for services, comprising: a memory storing programs and / or instructions; and a processor executing the programs and / or instructions; wherein, when executing the programs and / or instructions, the processor implements the steps of the data stream processing method for services as described in any of the first aspects.

[0036] The data stream processing apparatus for services provided by the present invention is capable of implementing the steps of the data stream processing method for services as described in any of the first aspects. Therefore, the apparatus also possesses all the beneficial effects of the data stream processing method for services in any of the first aspects, which will not be elaborated further here.

[0037] The fourth aspect of the present invention provides a readable storage medium having a program and / or instructions stored thereon, which, when the program and / or instructions are executed, implements the steps of the data stream processing method for business as described in any of the first aspects.

[0038] Additional aspects and advantages of the invention will become apparent in the following description or may be learned by practice of the invention. Attached Figure Description

[0039] Figure 1 This is one of the flowcharts illustrating a data stream processing method for business operations according to an embodiment of the present invention;

[0040] Figure 2 This is a second flowchart illustrating a data stream processing method for business operations according to an embodiment of the present invention.

[0041] Figure 3 This is a third flowchart illustrating a data stream processing method for business operations according to an embodiment of the present invention.

[0042] Figure 4 This is a fourth flowchart illustrating a data stream processing method for business operations according to an embodiment of the present invention.

[0043] Figure 5 This is the fifth flowchart illustrating a data stream processing method for business operations according to an embodiment of the present invention;

[0044] Figure 6 This is one of the block diagrams of a data stream processing apparatus for business according to an embodiment of the present invention;

[0045] Figure 7 This is a second block diagram of a data stream processing apparatus for business operations according to an embodiment of the present invention.

[0046] in, Figure 6 and Figure 7The correspondence between the reference numerals and component names in the attached drawings is as follows:

[0047] 1 processing device, 10 receiving unit, 12 acquiring unit, 14 processing unit, 16 execution unit, 2 processing device, 22 memory, 24 processor. Detailed Implementation

[0048] To better understand the above-mentioned objectives, features, and advantages of the present invention, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be noted that, unless otherwise specified, the embodiments and features described in these embodiments can be combined with each other.

[0049] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and therefore the scope of protection of the invention is not limited to the specific embodiments disclosed below.

[0050] The following reference Figures 1 to 7 This invention describes a data flow processing method, processing apparatus, and readable storage medium for services in some embodiments.

[0051] Example 1

[0052] According to one embodiment of this application, a method for processing business data streams is proposed. For example... Figure 1 A flowchart illustrating a data flow processing method for a service according to an embodiment of the present invention is shown. The method includes:

[0053] S102, Receive service data stream;

[0054] S104, determine the preset processing strategy based on the data stream;

[0055] S106, Process the data stream based on a preset processing strategy;

[0056] S108, execute the processed data stream according to the preset execution method.

[0057] In this embodiment, such as Figure 1 As shown, after receiving the data stream of the service to be processed, the system first obtains the preset processing strategy corresponding to the data stream. Specifically, the preset processing strategy is the technical detail processing procedure required for the data stream. That is, the preset processing strategy is a preset technical detail processing strategy. Then, the data stream is processed according to the determined preset processing strategy to realize the technical detail processing of the data stream, resulting in a technically processed data stream. Then, the technically processed data stream can be executed according to the preset execution method.

[0058] This embodiment fully considers the problem in related technologies where, when processing complex business data flows, developers need to consider both complex business logic and various intricate technical details. These technical detail processing processes are often cumbersome, preventing developers from focusing on business logic development and impacting development efficiency. In this application, the system server pre-processes the business data flow for technical details according to a preset processing strategy. This isolates the technical details of the business process from the logic editing process, allowing developers to concentrate solely on business logic development without worrying about technical details, thereby improving business development efficiency.

[0059] Example 2

[0060] Figure 2 A flowchart illustrating a data flow processing method for a service according to another embodiment of the present invention is shown. The method includes:

[0061] S202, Receive service data stream;

[0062] S204, Determine the preset processing strategy based on the data stream;

[0063] S206, Process the data stream based on a preset processing strategy;

[0064] S208, execute the processed data stream according to the preset execution method;

[0065] S210, calculate the cost required to execute the data stream based on the computational load during the data stream processing.

[0066] In this embodiment, while the system server executes the data of the business, it calculates the cost required to execute the business based on the amount of computation involved in the business processing, such as CPU usage and runtime, thereby realizing the requirement of pay-as-you-go for elastic computing.

[0067] Example 3

[0068] Figure 3 A flowchart illustrating a data flow processing method for a service according to another embodiment of the present invention is shown. The method includes:

[0069] S302, Receive service data stream;

[0070] S304, determine the preset processing strategy based on the data stream;

[0071] S306, Determine the corresponding preset process framework according to the preset processing strategy;

[0072] S308 processes data streams based on a preset process framework;

[0073] S310, the preset process framework is developed based on the preset processing strategy;

[0074] S312, there are multiple preset processing strategies, and each preset processing strategy has a corresponding preset process framework;

[0075] S314, Select the business executor based on the size of the data stream and / or the execution history;

[0076] S316, determine one or more servers needed during business execution based on the data flow;

[0077] S318 executes the processed data stream on one or more identified servers through a business executor.

[0078] In this embodiment, after receiving the data stream of the service to be processed, the system first obtains the preset processing strategy corresponding to the data stream. Specifically, the preset processing strategy is the technical detail processing procedure required for the data stream. That is, the preset processing strategy is a preset technical detail processing strategy. Then, the data stream is processed according to the determined preset processing strategy to achieve technical detail processing of the data stream, resulting in a technically processed data stream. The technically processed data stream can then be executed according to a preset execution method.

[0079] In the above embodiments, determining a preset processing strategy based on the data stream and processing the data stream based on the preset processing strategy specifically includes: determining a preset processing strategy based on the data stream; determining a corresponding preset process framework based on the preset processing strategy; processing the data stream based on the preset process framework; the preset process framework is developed based on the preset processing strategy; there are multiple preset processing strategies, and each preset processing strategy is correspondingly set with a preset process framework.

[0080] In this embodiment, the preset process framework is developed based on preset processing strategies. These preset processing strategies include many types, and each preset processing strategy has a corresponding preset process framework. Based on one or more preset process strategies corresponding to the data stream, one or more preset process frameworks corresponding to those strategies are determined, and then the data stream is processed according to the preset process frameworks. This application provides a simple and easy-to-use development framework. Utilizing this framework to process the technical details of the data stream greatly improves development efficiency.

[0081] In one possible embodiment, multiple pre-defined process frameworks are connected using process coding to form a complete set of general business code. This business code contains the complete detailed processing procedures for the business data flow. In this way, the business code can be deployed to different platforms simultaneously, enabling multi-platform use. This avoids the situation where developers, while handling the technical details of the business, are limited to learning only one big data processing platform, causing the business to be bound to that platform and unable to process the technical details of the business on other big data processing platforms.

[0082] Furthermore, the preset processing strategies include one or more of the following: proofreading strategy, conversion strategy, splitting strategy, merging strategy, forwarding strategy, persistence strategy, and exception handling strategy.

[0083] In the above embodiments, executing the processed data stream according to the preset execution method specifically includes: selecting a business executor based on the size of the data stream and / or the execution history; and executing the processed data stream through the business executor.

[0084] In this embodiment, the system server can automatically switch to the appropriate business executor based on the size of the data stream and / or execution history. This allows for automatic adaptation to different business data types, such as large data volumes with high concurrency and small data volumes with low concurrency, thereby ensuring the compatibility of business data volume with the facilities, reducing overall business operating costs, and protecting IT infrastructure investments.

[0085] In the above embodiments, the step of executing the processed data stream through the business executor includes: determining one or more servers that need to be used in the business execution process based on the data stream; and executing the processed data stream on the determined one or more servers through the business executor.

[0086] In this embodiment, the number of servers required during business execution can be determined in advance based on the data flow. Specifically, for medium-sized data scenarios, if the number of servers required during business execution is not determined in advance, the servers will process the data flow based on the existing number of servers. For example, three servers might be used for data flow, but if the data volume is slightly large, the efficiency of three servers is low, and an additional server needs to be added. However, in the solution of this application, the appropriate number of servers to process the business data is determined directly based on the data flow size, and then the processed data flow is executed, realizing dynamic server scaling. This greatly ensures the compatibility between data and servers, making the best use of IT facilities.

[0087] The second aspect of the present invention provides a data stream processing device 1 for services, such as... Figure 6 As shown, it includes a receiving unit 10, an acquiring unit 12, a processing unit 14, and an execution unit 16. Specifically:

[0088] After receiving the data stream of the business to be processed, the receiving unit 10 obtains the preset processing strategy corresponding to the data stream. The preset processing strategy is the technical detail processing process required for the data stream. That is, the preset processing strategy is a preset technical detail processing strategy. Then, the processing unit 14 processes the data stream according to the determined preset processing strategy to realize the technical detail processing of the data stream, and obtains the technical detail processed data stream. Finally, the execution unit 16 can execute the technical detail processed data stream according to the preset execution method. In the business data stream processing device 1 provided in this application, the system server performs technical detail processing on the business data stream in advance according to the preset processing strategy. This isolates the technical detail process and the logic editing process of the business, so that developers only need to focus on the development of business logic without having to consider the technical detail processing issues, thereby improving the development efficiency of the business.

[0089] Furthermore, the data stream processing apparatus 1 provided by the technical solution of the present invention can also implement the steps of the service data stream processing method provided by any of the technical solutions of the first aspect of the present invention. Therefore, the service data stream processing apparatus has all the technical effects of the service data stream processing method, which will not be repeated here.

[0090] The third aspect of the present invention provides a data stream processing device 2 for business applications, such as... Figure 7 As shown, it includes: a memory 22 storing programs and / or instructions; a processor 24 executing programs and / or instructions; wherein, when executing programs and / or instructions, the processor 24 implements the steps of a data flow processing method for services as described in any of the technical solutions of the first aspect.

[0091] The data stream processing apparatus 2 for services provided according to the present invention includes: a memory 22 and a processor 24. The memory 22 stores programs and / or instructions. When the programs and / or instructions are executed by the processor, they implement the steps of the data stream processing method for services as described in any of the first aspects. Since this apparatus is capable of implementing the steps of the data stream processing method for services as described in any of the first aspects, it also possesses all the beneficial effects of the data stream processing method for services as described in any of the first aspects, which will not be elaborated further here.

[0092] The fourth aspect of the present invention provides a readable storage medium having a program and / or instructions stored thereon, which, when the program and / or instructions are executed, implements the steps of the data stream processing method for business as described in any of the first aspects.

[0093] Example 4

[0094] Figure 4 A flowchart illustrating a data flow processing method for a service according to another embodiment of the present invention is shown. The method includes:

[0095] S402, receives data streams from services;

[0096] S404, determine a preset processing strategy based on the data stream, and process the data stream based on the preset processing strategy;

[0097] S406, determine whether the data stream has a matching execution history. If the result is yes, execute S408; otherwise, execute S410.

[0098] S408, Select the business executor based on the matched execution history and execute S412;

[0099] S410, Select the business executor based on the size of the data stream, and execute S412;

[0100] S412, determine one or more servers needed during business execution based on the data flow;

[0101] S414 executes the processed data stream on one or more identified servers via a business executor.

[0102] In this embodiment, after receiving the data stream of the service to be processed, the preset processing strategy corresponding to the data stream is first obtained. Specifically, the preset processing strategy is the technical detail processing procedure required for the data stream. That is, the preset processing strategy is a preset technical detail processing strategy. Then, the data stream is processed according to the determined preset processing strategy to realize the technical detail processing of the data stream, resulting in a technically detailed processed data stream. Then, the technically detailed processed data stream can be executed according to a preset execution method.

[0103] Furthermore, business executors include serial executors, parallel executors, and distributed executors. Serial executors are suitable for simple scenarios, small data volumes, and testing / debugging scenarios, where the system server executes the data stream step-by-step according to the process sequence. Parallel executors are suitable for medium-sized data scenarios, using a parallel framework to send data to different system servers for simultaneous processing. Distributed executors are suitable for big data scenarios. In scenarios with excessively large business data volumes, the local system server cannot handle the massive amounts of data; in such cases, it is necessary to send the common business code to different big data platforms to execute the data processing.

[0104] In one possible embodiment, different preset execution frameworks can be developed depending on the type of business executor. These preset execution frameworks include serial execution frameworks, parallel execution frameworks, and distributed execution frameworks. The system server can automatically switch between different preset execution frameworks based on the size of the data stream and / or the execution history.

[0105] In the above embodiments, the step of selecting a business executor based on the size of the data stream and / or its execution history includes: when the data stream has a matching execution history, selecting a business executor based on the matching execution history; when the data stream does not have a matching execution history, selecting a business executor based on the size of the data stream.

[0106] In this embodiment, the system server determines whether the data stream has a matching execution history. If so, it selects a suitable business executor based on the matching execution history. If a suitable business executor cannot be selected based on the matching execution history, then the size of the data stream needs to be determined.

[0107] Furthermore, the step of selecting the business executor based on the size of the data stream includes: selecting a serial executor as the business executor when the data stream is less than a first threshold; selecting a parallel executor as the business executor when the data stream is greater than the first threshold but less than a second threshold; and selecting a distributed executor as the business executor when the data stream is greater than the second threshold.

[0108] In this embodiment, when the data flow is less than a first threshold, a serial executor is selected as the business executor; when the data flow is greater than the first threshold but less than a second threshold, a parallel executor is selected as the business executor; and when the data flow is greater than the second threshold, a distributed executor is selected as the business executor. This allows for automatic adaptation to different business data volumes, such as large data volumes with high concurrency and small data volumes with low concurrency, thereby ensuring the adaptability of business data and facilities, reducing the overall operating cost of the business, and protecting IT infrastructure investment.

[0109] In one possible embodiment, the system server contains an intelligent executor that can automatically select a suitable business executor or a preset execution framework based on the size of the data stream and / or the execution history.

[0110] In the above embodiments, the step of executing the processed data stream through the business executor includes: determining one or more servers that need to be used in the business execution process based on the data stream; and executing the processed data stream on the determined one or more servers through the business executor.

[0111] In this embodiment, the number of servers required during business execution can be determined in advance based on the data flow. Specifically, for medium-sized data scenarios, if the number of servers required during business execution is not determined in advance, the servers will process the data flow based on the existing number of servers, for example, using 3 servers. However, in this case, due to the slightly large data volume, the efficiency of 3 servers processing the data flow is somewhat low, so it is necessary to consider adding an additional server to process the data flow. In the solution of this application, the appropriate number of servers to process the business data is directly determined based on the data flow size, and then the processed data flow is executed, realizing dynamic server scaling. This can greatly ensure the adaptability of data and servers, making full use of IT facilities.

[0112] Example 5

[0113] This embodiment proposes a method for processing business data streams. The method is as follows: Figure 5 As shown, it includes:

[0114] S502, develop business processing logic based on annotations;

[0115] S504 is an execution strategy based on code, configuration files, and database configuration.

[0116] S506, the actuator determines the execution plan based on business complexity and load;

[0117] S508 executes business logic.

[0118] Specifically, annotation-based data processing development frameworks involve various scenarios during business flow processing, such as splitting, merging, persistence, and exception handling. This involves various technical details such as parallel execution and database transactions.

[0119] A data processing flow orchestration framework based on code, configuration files, and databases. Process coding can be based on code, configuration files, or data.

[0120] Serial executor: Designed for simple scenarios, small data volumes, testing and debugging, etc., to execute data processes sequentially.

[0121] Parallel Executor: For medium-sized data scenarios, a parallel framework is used to execute data streams in multiple threads.

[0122] Distributed executor based on big data platform: For big data scenarios, the same business code can be deployed to the big data platform for execution.

[0123] Intelligent actuator: Based on data volume, execution history, and current system status, it automatically selects the appropriate actuator.

[0124] The solution presented in this application has the following advantages over the prior art:

[0125] 1. Provides a development standard to protect the IT development investment of enterprises; business code is written once and can be executed on multiple platforms.

[0126] 2. It provides a convenient, annotation-based, and easy-to-use development framework that greatly improves development efficiency, allowing developers to focus more on business logic development without having to pay too much attention to technical details.

[0127] 3. Systematically solves the connection between development and execution; business code can be executed in a local container or based on a big data platform.

[0128] 4. The system can automatically switch execution frameworks based on load and business volume, reducing overall operating costs and protecting IT infrastructure investments.

[0129] 5. By isolating technical complexity from business complexity, IT developers can focus on developing business logic.

[0130] 6. This invention solves the problem of automatic adaptation to large data volumes and high concurrency, and small data volumes and low concurrency.

[0131] 7. This invention solves the compatibility problem of various big data platforms. Companies can deploy the code to different big data platforms according to their business needs, or degrade to sequential execution for simple, low-concurrency scenarios.

[0132] 8. Dynamic expansion and dynamic switching of actuators make the best use of IT settings, while direct elastic computing and pay-as-you-go billing.

[0133] In the description of this specification, the terms "connection," "installation," and "fixing," etc., should be interpreted broadly. For example, "connection" can be a fixed connection, a detachable connection, or an integral connection; it can be a direct connection or an indirect connection through an intermediate medium. Those skilled in the art can understand the specific meaning of the above terms in this invention according to the specific circumstances.

[0134] In the description of this specification, it should be understood that the terms "upper" and "lower" indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, and are not intended to indicate or imply that the device or unit referred to must have a specific orientation or be constructed and operated in a specific orientation. Therefore, they should not be construed as limiting the present invention.

[0135] In the description of this specification, the terms "one embodiment," "some embodiments," "specific embodiment," etc., refer to a specific feature, structure, material, or characteristic described in connection with that embodiment or example, which is included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0136] The above are merely preferred embodiments of the present invention and are not intended to limit the present invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for processing business data streams, characterized in that, include: Receive the data stream of the service; A preset processing strategy is determined based on the data stream, and the data stream is processed based on the preset processing strategy; The processed data stream is executed according to a preset execution method; The step of determining a preset processing strategy based on the data stream and processing the data stream based on the preset processing strategy specifically includes: A preset processing strategy is determined based on the data stream; Determine the corresponding preset process framework based on the preset processing strategy; The data stream is processed based on the preset process framework; The preset process framework is developed based on the preset processing strategy; The preset processing strategies include multiple ones, and each preset processing strategy is correspondingly set with the preset process framework; The preset process framework is connected in the form of process coding to form a complete general code; The step of executing the processed data stream according to a preset execution method specifically includes: Select the business executor based on the size of the data stream and / or the execution history; The processed data stream is executed by the business executor; The business executor includes serial executors, parallel executors, and distributed executors. The step of selecting a business executor based on the size of the data stream and / or execution history includes: When the data stream has a matching execution history, a business executor is selected based on the matching execution history; When the data stream does not have a matching execution history, a business executor is selected based on the size of the data stream; The steps for selecting a service executor based on the size of the data stream include: When the data flow is less than a first threshold, the serial executor is selected as the business executor; When the data stream is greater than a first threshold and less than a second threshold, the parallel executor is selected as the business executor. When the data flow exceeds the second threshold, the distributed executor is selected as the business executor.

2. The data stream processing method for a business according to claim 1, characterized in that, The preset processing strategy includes: One or more of the following strategies: proofreading, conversion, splitting, merging, forwarding, persistence, and exception handling.

3. The data stream processing method for a service according to claim 1, characterized in that, The step of executing the processed data stream through the business executor specifically includes: Based on the data stream, determine one or more servers that need to be used during the business execution process; The processed data stream is executed on one or more designated servers by the business executor.

4. The data stream processing method for a service according to claim 1, characterized in that, Also includes: The cost required to execute the data stream is calculated based on the computational load during the data stream processing.

5. A data stream processing apparatus (1) for a business application, characterized in that, The data stream processing apparatus for the service is used to implement the steps of the data stream processing method for the service as described in any one of claims 1 to 4, including: The receiving unit (10) is used to receive the data stream of the service; The acquisition unit (12) is used to acquire a preset processing strategy corresponding to the data stream; Processing unit (14) is used to process the data stream; The execution unit (16) is used to execute the processed data stream.

6. A data stream processing device (2) for a business application, characterized in that, include: Memory (22) that stores programs and / or instructions; Processor (24), which executes the program and / or instructions; Wherein, when the processor (24) executes the program and / or instructions, it implements the steps of the data flow processing method for the service as described in any one of claims 1 to 4.

7. A readable storage medium, characterized in that, include: The readable storage medium stores a program and / or instructions, which, when executed, implement the steps of the data stream processing method for the service as described in any one of claims 1 to 4.

Images