A method for executing a batch task, a related device, equipment and a storage medium

Abstract

The application discloses a batch task execution method, related devices, equipment and a storage medium. The method comprises the following steps: obtaining a first logical unit set, the first logical unit set comprising K logical units required for executing a same batch task; for each logical unit in the first logical unit set, obtaining a basic operator corresponding to the logical unit; for each logical unit in the first logical unit set, obtaining a target operator corresponding to the logical unit according to an operation indicated by the logical unit and the corresponding basic operator; performing splicing processing on the K target operators to obtain a first batch task developed based on a target language; and in response to a subscription operation for the first batch task, executing the first batch task. For a business under an ecological environment of other languages, the application does not require a business party to make a cross-language call, thereby reducing the difficulty of business development and maintenance. In addition, lightweight deployment can be achieved, and therefore, the use threshold is low.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a method, related apparatus, device, and storage medium for executing batch tasks. Background Technology

[0002] Batch processing is a task that automates the processing of batches of data according to rules and plans within a computer system environment. It primarily targets large datasets that can originate from various sources, such as databases and file systems. One key characteristic of batch processing is its high degree of automation, which significantly improves data processing efficiency.

[0003] Currently, Apache Flink is an open-source stream processing framework built for distributed, high-performance stream processing applications. Apache Flink not only provides support for high-throughput real-time computation but also enables batch data processing. Compared to other data processing engines on the market, it uses stream computing to simulate the processing flow of batch tasks.

[0004] However, the inventors discovered at least the following problems with the current solution: because Apache Flink is a tool within the Java language ecosystem, business applications running in other language ecosystems require cross-language calls. This not only increases the complexity of developing and maintaining the business applications but also leads to higher R&D costs. Therefore, an effective method is urgently needed to solve these problems. Summary of the Invention

[0005] This application provides a method, related apparatus, device, and storage medium for executing batch tasks. For businesses operating in other language ecosystems, cross-language calls are unnecessary, thus reducing the difficulty of business development and maintenance. Furthermore, lightweight deployment is possible, resulting in a low barrier to entry.

[0006] In view of this, this application provides a method for executing batch tasks, including:

[0007] Obtain a first set of logical units, wherein the first set of logical units includes K logical units required to execute the same batch task, each logical unit is used to indicate an operation, and K is an integer greater than or equal to 1;

[0008] For each logical unit in the first set of logical units, obtain the basic operator corresponding to the logical unit, wherein the basic operator is written in the target language;

[0009] For each logical unit in the first set of logical units, obtain the target operator corresponding to the logical unit based on the operation indicated by the logical unit and the corresponding basic operator;

[0010] The K target operators are concatenated to obtain the first batch processing task developed based on the target language, wherein the K target operators include the target operators corresponding to each logical unit in the first set of logical units;

[0011] In response to the subscription operation for the first batch task, execute the first batch task.

[0012] This application also provides another batch task execution device, including:

[0013] The acquisition module is used to acquire a first set of logical units, wherein the first set of logical units includes K logical units required to execute the same batch task, each logical unit is used to indicate an operation, and K is an integer greater than or equal to 1;

[0014] The acquisition module is also used to acquire the basic operator corresponding to each logical unit in the first set of logical units, wherein the basic operator is written based on the target language;

[0015] The acquisition module is also used to acquire the target operator corresponding to each logical unit in the first set of logical units, based on the operation indicated by the logical unit and the corresponding basic operator.

[0016] The processing module is used to concatenate K target operators to obtain a first batch task developed based on the target language, wherein the K target operators include the target operators corresponding to each logical unit in the first set of logical units;

[0017] The execution module is used to execute the first batch task in response to a subscription operation for the first batch task.

[0018] In one possible design, in a first implementation of another aspect of the embodiments of this application, the first batch processing task includes a first target operator and a second target operator, wherein the second target operator is a target operator concatenated after the first target operator;

[0019] The execution module is specifically used to acquire N first data streams, where N is greater than or equal to 1;

[0020] The first target operator is called to process N first data streams to obtain M second data streams, where M is an integer greater than or equal to 1;

[0021] The second objective operator is invoked to process the M second data streams to obtain Q third data streams, where Q is an integer greater than or equal to 1.

[0022] In one possible design, in a first implementation of another aspect of the embodiments of this application, the first target operator is a target mapping operator;

[0023] The execution module is specifically used to call the interface of the target mapping operator to obtain the element mapping rules;

[0024] According to the element mapping rules, each element in the first data stream is transformed to obtain the second data stream.

[0025] In one possible design, in a first implementation of another aspect of the embodiments of this application, the first target operator is a target splicing mapping operator;

[0026] The execution module is specifically used to call the interface of the target concatenation mapping operator to obtain the data mapping rules;

[0027] According to the data mapping rules, each element in the first data stream is transformed to obtain the element sequence corresponding to each element;

[0028] Based on the element order of the first data stream, the element sequence corresponding to each element is concatenated to obtain the second data stream.

[0029] In one possible design, in a first implementation of another aspect of the embodiments of this application, the first target operator is a target filtering operator;

[0030] The execution module is specifically used to call the interface of the target filtering operator to obtain the element filtering rules;

[0031] According to the element filtering rules, each element in the first data stream is filtered to obtain a Boolean value for each element. The Boolean value is used to indicate whether the element meets the element filtering rules.

[0032] Based on the Boolean value of each element, retain all elements in the first data stream that satisfy the element filtering rules to obtain the second data stream.

[0033] In one possible design, in a first implementation of another aspect of the embodiments of this application, the first target operator is a target reduction operator;

[0034] The execution module is specifically used to call the interface of the target reduction operator to obtain the element calculation rules and batch output rules;

[0035] According to the element calculation rules, at least two elements in the first data stream are calculated step by step. Under the condition of satisfying the batch output rules, the target element corresponding to a batch is obtained, wherein the batch includes at least two elements in the first data stream.

[0036] A second data stream is generated based on the target element corresponding to each batch.

[0037] In one possible design, in a first implementation of another aspect of the embodiments of this application, the first target operator is a target viewing operator;

[0038] The execution module is specifically used to call the interface of the target viewing operator to obtain the element viewing rules;

[0039] Based on the element viewing rules, the first data stream is bypassed, and a second data stream is generated, wherein the elements in the second data stream are consistent with the elements in the first data stream.

[0040] In one possible design, in a first implementation of another aspect of the embodiments of this application, the first target operator is a target merging operator;

[0041] The execution module is specifically used to merge at least two first data streams using a target merging operator to obtain a second data stream, wherein all elements in the second data stream originate from at least two first data streams.

[0042] In one possible design, in a first implementation of another aspect of the embodiments of this application, the first target operator is a target grouping operator;

[0043] The execution module is specifically used to call the interface of the target grouping operator to obtain the element grouping rules;

[0044] Based on the element grouping rules, each element in the first data stream is grouped to obtain the group identifier corresponding to each element;

[0045] Based on the group identifier corresponding to each element, elements belonging to the same group identifier are added to the same data stream to obtain at least one second data stream.

[0046] In one possible design, in a first implementation of another aspect of the embodiments of this application, the first target operator is a target identity operator;

[0047] The execution module is specifically used to generate a second data stream based on the first data stream and the target identity operator, wherein the elements in the second data stream are consistent with the elements in the first data stream.

[0048] In one possible design, in a first implementation of another aspect of the embodiments of this application,

[0049] The execution module is specifically used to respond to the subscription operation for the first batch task, call the interface of the target observation operator, and obtain the data processing rules.

[0050] According to the data processing rules, execute the first batch processing task.

[0051] In one possible design, in a first implementation of another aspect of the embodiments of this application,

[0052] The acquisition module is specifically used to acquire the basic operator corresponding to each logical unit in the first set of logical units, provided that the Software Development Kit (SDK) provides the basic operator corresponding to the logical unit.

[0053] For each logical unit in the first set of logical units, if the SDK does not provide the basic operator corresponding to the logical unit, the basic operator corresponding to the logical unit is generated in response to the operator construction operation.

[0054] In one possible design, in a first implementation of another aspect of the embodiments of this application,

[0055] The acquisition module is also used to acquire a second set of logical units, wherein the second set of logical units includes X logical units required to execute the same batch task, each logical unit is used to indicate an operation, and X is an integer greater than or equal to 1;

[0056] The acquisition module is also used to acquire Y logical units from the second logical unit set when there are duplicate logical units in the second logical unit set and the first logical unit set. The Y logical units include logical units that are not duplicated in the second logical unit set and the first logical unit set, and Y is an integer greater than or equal to 1 and less than X.

[0057] The acquisition module is also used to acquire the basic operators corresponding to the logical units among the Y logical units;

[0058] The acquisition module is also used to acquire the target operator corresponding to each of the Y logical units, based on the operation indicated by the logical unit and the corresponding basic operator.

[0059] The processing module is also used to concatenate X target operators to obtain a second batch task, wherein the X target operators include the target operators corresponding to each logical unit in the second set of logical units;

[0060] The execution module is also used to execute the second batch task in response to a subscription operation for the second batch task.

[0061] Another aspect of this application provides a computer-readable storage medium storing instructions that, when executed on a computer, cause the computer to perform the methods described above.

[0062] Another aspect of this application provides a computer program product or computer program including computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the methods provided in the above aspects.

[0063] As can be seen from the above technical solutions, the embodiments of this application have the following advantages:

[0064] This application provides a method for executing batch processing tasks. First, the various logical units required for the same batch processing task are obtained. Next, the corresponding basic operator is obtained for each logical unit; this basic operator is written in the target language. Based on this, the corresponding target operator is obtained according to the operation indicated by each logical unit and its corresponding basic operator. These target operators are concatenated to obtain the first batch processing task. Upon receiving a subscription operation, the first batch processing task is executed. Through the above method, a tool for other language (e.g., Go, C++, etc.) ecosystems is designed. That is, for businesses developed in other languages, this tool can be used to execute batch processing tasks. On the one hand, it eliminates the need for cross-language calls by the business side, reducing the difficulty of business development and maintenance. On the other hand, the tool designed in this application can achieve lightweight deployment and has a low barrier to entry. Attached Figure Description

[0065] Figure 1 This is a schematic diagram illustrating a scenario applied to email sending in an embodiment of this application;

[0066] Figure 2 This is a schematic diagram of a scenario applied to document management in an embodiment of this application;

[0067] Figure 3 This is a schematic diagram of a scenario applied to system maintenance in an embodiment of this application;

[0068] Figure 4 This is a schematic diagram of an implementation environment for the batch task execution method in this application embodiment;

[0069] Figure 5 This is a flowchart illustrating the batch task execution method in an embodiment of this application.

[0070] Figure 6 This is a schematic diagram illustrating the construction of batch processing tasks based on a set of logical units in an embodiment of this application.

[0071] Figure 7 This is a schematic diagram illustrating the input and output relationship between target operators in an embodiment of this application;

[0072] Figure 8 This is a schematic diagram of data processing based on the target mapping operator in an embodiment of this application;

[0073] Figure 9 This is a schematic diagram of data processing based on the target splicing mapping operator in an embodiment of this application;

[0074] Figure 10 This is a schematic diagram of data processing based on a target filtering operator in an embodiment of this application;

[0075] Figure 11 This is a schematic diagram of data processing based on the target reduction operator in an embodiment of this application;

[0076] Figure 12 This is a schematic diagram of data processing based on the target viewing operator in an embodiment of this application;

[0077] Figure 13 This is a schematic diagram of data processing based on the target merging operator in an embodiment of this application;

[0078] Figure 14 This is a schematic diagram of data processing based on the target grouping operator in an embodiment of this application;

[0079] Figure 15 This is a schematic diagram of data processing based on the target identity operator in an embodiment of this application;

[0080] Figure 16 This is a schematic diagram illustrating the execution of multiple batch processing tasks in an embodiment of this application;

[0081] Figure 17 This is a schematic diagram of a topology structure for a batch processing task in an embodiment of this application;

[0082] Figure 18 This is a schematic diagram of a batch task execution device in an embodiment of this application;

[0083] Figure 19 This is a schematic diagram of the structure of a computer device in an embodiment of this application. Detailed Implementation

[0084] This application provides a method, related apparatus, device, and storage medium for executing batch tasks. For businesses operating in other language ecosystems, cross-language calls are unnecessary, thus reducing the difficulty of business development and maintenance. Furthermore, lightweight deployment is possible, resulting in a low barrier to entry.

[0085] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a particular order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented, for example, in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “corresponding,” and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0086] It is understood that, in the specific implementation of this application, data related to batch processing tasks requires user permission or consent before being used in specific products or technologies. That is, before collecting user data, users can be notified through prompts, pop-ups, or voice prompts. The process of collecting user data only begins after obtaining user permission or consent. In other words, all user data collected in this application is collected with the user's consent, and the collection, use, and processing of this data must comply with the relevant laws, regulations, and standards of the relevant countries and regions.

[0087] Batch processing refers to the batch processing of data within an application. A batch processing task is a task that automates the processing of large amounts of data according to predetermined rules, time intervals, or event triggers. Batch processing tasks are typically implemented using scripting languages ​​or specialized tools. Scripts or configuration tools are used to define the data source, processing logic, and target storage location. Typically, batch processing tasks are characterized by large data volumes (i.e., they generally involve processing a large amount of data), automation (i.e., they are required to run automatically at specified times or frequencies), high performance (i.e., they are required to complete the processing task as quickly as possible), and observability (i.e., the task execution status can be monitored).

[0088] Currently, the two mainstream big data processing frameworks in the industry, Apache Flink and Apache Spark, both support batch task processing well. Apache Flink is an open-source stream processing framework designed for distributed and high-performance stream processing applications. It not only provides support for high-throughput real-time computing but also enables batch data processing based on stream computing. Apache Spark is an open-source distributed data processing framework for rapid iteration, large-scale data analysis, and machine learning, providing a unified computing model for both batch and stream processing.

[0089] While big data processing frameworks like Apache Flink and Apache Spark offer high-performance and reliable solutions for batch processing tasks, their integration, deployment, and maintenance costs are relatively high. Taking Apache Flink as an example, it's geared towards professional big data developers, often requiring specific optimizations based on data volume, computation type, and other factors. Achieving numerous statistical metrics incurs significant development costs. Furthermore, both Apache Flink and Apache Spark are tools within the Java language ecosystem. For businesses operating in other language ecosystems, integration and use are even more cumbersome, requiring cross-language calls.

[0090] Based on this, this application provides a batch task execution method adapted for business use in other language ecosystems. The basic operators required to build the batch task are provided to the business party in the form of a software development kit (SDK), thus reducing the integration cost. This application can be used for stream processing, where operations on the stream are continuously called on their respective coroutines, thereby fully utilizing multi-core central processing units (CPUs). Furthermore, it has the advantages of high code reusability and flexible addition or removal of functions.

[0091] Before introducing the specific methods of this application, the application scenarios of this application will be illustrated by example. It should be understood that the following application scenarios are merely illustrative and are not limited to these examples.

[0092] (1) Email sending scenario:

[0093] Batch tasks provide an efficient way to execute and manage email sending. They can integrate a large number of email sending operations, process emails in batches according to predetermined rules and order, and make the originally complex and scattered email sending work orderly and easy to manage.

[0094] For example, please refer to Figure 1 , Figure 1 This is a schematic diagram illustrating a scenario applied to email sending in this application embodiment. As shown in the figure, enterprises often need to send emails in batches, such as marketing emails, bill reminders, and system notifications. In this process, batch processing tasks can integrate scattered email sending needs and process them in batches according to predetermined rules on a scheduled basis. Based on this, the email settings interface allows users to set the task name, email subject, sender nickname, task notification type, etc. After clicking "Next," users can continue to edit the email content and set the recipient's address.

[0095] (2) Document management scenario;

[0096] Batch processing is a crucial tool for efficient document management. When documents are going live, batch processing can quickly publish approved documents to designated platforms in batches, improving efficiency. For documents that need to be blocked, batch processing can swiftly lock or delete them based on instructions, preventing the spread of illegal or harmful information. In terms of classification and tagging, batch processing can classify and tag large numbers of documents according to preset rules and algorithms, making document management more organized.

[0097] For example, please refer to Figure 2 , Figure 2 This is a schematic diagram illustrating a scenario applied to document management in this application embodiment. As shown in the figure, backend personnel can trigger a batch task to classify and tag all WeChat official account articles from the past year. Based on this, backend personnel can view the execution progress and task details of the batch task through the document management interface.

[0098] (3) System maintenance scenario;

[0099] System maintenance encompasses numerous periodic tasks, and batch processing is a crucial method for executing these tasks. For example, databases periodically clean up redundant data, and system logs are archived. Batch processing tasks can be automatically triggered and completed according to preset times and rules. Simultaneously, system maintenance provides a stable and reliable operating environment for batch processing tasks, ensuring their smooth execution. However, if the system experiences a failure, batch processing tasks may also be affected and fail.

[0100] For example, please refer to Figure 3 , Figure 3 This is a schematic diagram illustrating a system maintenance scenario in this application embodiment. As shown in the figure, maintenance personnel can trigger multiple batch tasks, such as "Task 1" for redundant data and "Task 2" for system log archiving. Thus, by periodically executing tasks such as cleanup and optimization, the system can maintain a good operating state.

[0101] It should be noted that the above application scenarios are merely examples. The batch task execution method provided in this embodiment can also be applied to other scenarios, and is not limited here.

[0102] The method provided in this application can be applied to... Figure 4The illustrated implementation environment includes a terminal 401 and a server 402, and the terminal 401 and server 402 can communicate via a network 403. The network 403 uses standard communication technologies and / or protocols, typically the Internet, but can also be any network, including but not limited to Bluetooth, a local area network (LAN), a metropolitan area network (MAN), a wide area network (WAN), mobile, private networks, or any combination of virtual private networks. In some embodiments, customized or dedicated data communication technologies may be used to replace or supplement the aforementioned data communication technologies.

[0103] The terminal 401 involved in this application includes, but is not limited to, mobile phones, tablets, laptops, desktop computers, intelligent voice interaction devices, virtual reality devices, smart home appliances, vehicle terminals, and aircraft. The client is deployed on the terminal 401 and can run on the terminal 401 via a browser, a standalone application (APP), or a mini-program.

[0104] The server 402 involved in this application can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence (AI) platforms.

[0105] In the above implementation environment, in step A1, terminal 401 downloads the SDK from server 402 via network 403. In step A2, for a given batch processing task, it is broken down into several logical units, each corresponding to an operation. In step A3, based on the decomposed logical units, the basic operator corresponding to each logical unit is determined. In step A4, the operation corresponding to each logical unit is implemented according to the requirements of the basic operator, thereby obtaining the target operator corresponding to each logical unit. In step A5, these target operators are concatenated to construct the batch processing task. In step A6, when a subscription operation is received, the batch processing task is executed.

[0106] Based on the above introduction, the execution method of the batch processing task in this application will be described below. Please refer to [link / reference]. Figure 5The batch task execution method in this application embodiment can be completed independently by the server, independently by the terminal, or jointly by the terminal and the server. The method provided in this application includes:

[0107] S501. Obtain a first set of logical units, wherein the first set of logical units includes K logical units required to execute the same batch task, each logical unit is used to indicate an operation, and K is an integer greater than or equal to 1;

[0108] In one or more embodiments, the processing logic of a batch processing task may be quite complex. Therefore, it is necessary to decompose a batch processing task into several independent logical units, thereby obtaining a first set of logical units including K logical units. Typically, K is an integer greater than 2, that is, the K logical units include a logical unit corresponding to a starting operation, logical units corresponding to several intermediate operations, and a logical unit corresponding to a terminal operation.

[0109] Specifically, these logical units are connected via channels. Each operation belongs to an independent logical unit, achieving high cohesion and low coupling. Here, "operation" represents the processing of data on a stream. "Channels" are a way to implement concurrency in Go, allowing multiple goroutines to communicate and exchange data. A "stream" represents a sequence of elements that can be processed one by one during a single traversal. Operators receive one or more streams, process them, and output one or more streams. A stream generally corresponds to an operation, but operations on a stream are not executed immediately; instead, they are lazily evaluated, only triggered when a terminal operation is encountered. Operations on streams can be defined through a series of method calls, making the code more readable. In this application, the definition of a stream is abstracted as follows:

[0110] type Streaminterface{

[0111] dataChan()<-chan interface{} / / Returns elements from the stream

[0112] start(context.Context) / / Starts the function that processes the elements on the stream.

[0113] }

[0114] The above definition includes two key components of a stream: data and operations. The `dataChan()` method returns the data on the stream, implemented using Go's pipes. The `start()` method initiates operator operations on the stream and runs them on a dedicated goroutine, also implemented using Go's goroutines.

[0115] S502. For each logic unit in the first logic unit set, obtain the basic operator corresponding to the logic unit, wherein the basic operator is written based on the target language;

[0116] In one or more embodiments, the basic operator corresponding to the logic unit can be obtained through the SDK. If the SDK does not have the required basic operator, a new type of basic operator can be abstracted, that is, the basic operator corresponding to the logic unit can be obtained by manually writing the code.

[0117] It should be noted that the target language used in this application may be Go, C++, Python, or Hypertext Preprocessor (PHP), etc. This application uses Go as the target language for description, but it should not be construed as a limitation of this application.

[0118] Understandably, the basic operators provided by the SDK may or may not require user-implemented interfaces as parameters. An "interface" represents a set of method signatures, an abstract type in Go. An "operator" represents an abstract unit of computation; operators can transform one or more streams into a new stream. Multiple operators can form a data flow topology, resulting in a batch processing task.

[0119] Specifically, for ease of understanding, please refer to Figure 6 , Figure 6 This is a schematic diagram illustrating the construction of a batch processing task based on a set of logical units in an embodiment of this application. As shown in the figure, the first set of logical units includes five logical units: a logical unit for "reading local files," a logical unit for "getting the main text," a logical unit for "tracking and reporting," a logical unit for "filtering documents that meet the requirements," and a logical unit for "querying file metadata." Based on this, the basic operator corresponding to each logical unit is obtained. For example, the logical unit corresponding to "getting the main text" can use the basic mapping (Map) operator, and the logical unit corresponding to "tracking and reporting" can use the basic peek (Peek) operator.

[0120] S503. For each logic unit in the first logic unit set, obtain the target operator corresponding to the logic unit according to the operation indicated by the logic unit and the corresponding basic operator.

[0121] In one or more embodiments, the target operator is obtained by implementing the interface according to the requirements of the basic operator for the operation corresponding to each logical unit in the first logical unit set. Taking the basic Map operator as an example, the corresponding target Map operator is obtained by implementing the plugin mapper interface provided by the basic Map operator.

[0122] It should be noted that the basic operators (e.g., basic Map operator, basic Peek operator, etc.) and target operators (e.g., target Map operator, target Peek operator, etc.) involved in this application are all written in Go.

[0123] Specifically, for better understanding, please refer to [the relevant documentation / reference]. Figure 6 As shown in the figure, taking the logic unit corresponding to "Get Text" using the basic Map operator as an example, after implementing the interface parameters required by the basic Map operator, the target Map operator is obtained. Based on this, the file identifier is received through the target Map operator, and the file text is output. Taking the logic unit corresponding to "Trace Reporting" using the basic Peek operator as another example, after implementing the interface parameters required by the basic Peek operator, the target Peek operator is obtained. Based on this, data can be extracted from the upstream data stream through the target Peek operator and then reported.

[0124] S504. The K target operators are concatenated to obtain the first batch task developed based on the target language, wherein the K target operators include the target operators corresponding to each logical unit in the first set of logical units;

[0125] In one or more embodiments, the implemented K target operators are concatenated according to a certain topological structure to obtain a specific batch processing task, that is, to obtain the first batch processing task developed based on the target language. For example, suppose a batch processing task is used to convert a string to a number. Based on this, a basic Just operator and a basic Map operator can be used. The basic Just operator reads the data in the parameters, performs no processing, and outputs a new data stream as is. Therefore, the basic Just operator can be directly used as the target Just operator. The operation of converting a string to an integer (string2Int) requires implementing the interface of the basic Map operator to obtain the target Map operator. The specific implementation method is as follows:

[0126] s:=Just("1","2","3","4")

[0127] s = Map(s, new(string2Int))

[0128] Here, s represents the data stream. It can be seen that the target Map operator maps the data stream returned by the target Just operator into a new data stream, which is the numeric form of the string data in the data stream returned by the target Just operator.

[0129] Specifically, for better understanding, please refer to [the relevant documentation / reference]. Figure 6 As shown in the figure, the target operators for reading local files, querying file metadata, filtering documents that meet the requirements, reporting traces, and obtaining the main text are concatenated in sequence to obtain the topology of the first batch task.

[0130] S505, In response to the subscription operation for the first batch task, execute the first batch task.

[0131] In one or more embodiments, after the first batch task is concatenated, it is not executed immediately; that is, the operations on the stream are not invoked immediately. The first batch task will only run after receiving a subscription operation triggered by the terminal for the first batch task.

[0132] Specifically, taking the data stream returned by the Map operator of the subscription target as an example, that is,

[0133] <- Subscribe(context.Background(), s, observer.Observer) / / Subscribe to and consume data from the data stream "s"

[0134] As can be seen, the subscription operation enables the entire batch processing task to run. Consequently, operations on the data stream are continuously invoked within their respective coroutines, thus fully utilizing the multi-core CPU.

[0135] It should be noted that the term "in response to" in this application refers to the conditions or states upon which the execution of an operation depends, and one or more operations that can be executed when certain conditions or states are met. These operations can be real-time or have a certain delay.

[0136] This application provides a method for executing batch tasks. Through this approach, a tool is designed for use in other language ecosystems (e.g., Go, C++, Python, or PHP). Specifically, for applications developed using Go, this tool can be used to execute batch tasks. On one hand, it eliminates the need for cross-language calls by the application side, reducing the difficulty of business development and maintenance. On the other hand, the tool designed in this application enables lightweight deployment and has a low barrier to entry.

[0137] Optionally, in the above Figure 5 Based on one or more corresponding embodiments, in another optional embodiment provided by the present application, the first batch processing task includes a first target operator and a second target operator, wherein the second target operator is a target operator concatenated after the first target operator;

[0138] Executing the first batch processing task specifically includes:

[0139] Obtain N first data streams, where N is greater than or equal to 1;

[0140] The first target operator is called to process N first data streams to obtain M second data streams, where M is an integer greater than or equal to 1;

[0141] The second objective operator is invoked to process the M second data streams to obtain Q third data streams, where Q is an integer greater than or equal to 1.

[0142] In one or more embodiments, a method for transferring data streams between target operators is described. As described in the foregoing embodiments, an intermediate operation B processes data received from the pipeline of its upstream operation A, and then outputs the processed data through the pipeline held by intermediate operation B. The downstream operation C continues to read the processed data from the pipeline held by intermediate operation B, and then continues to process that data. Thus, each operation runs in its own unique coroutine, thereby fully utilizing multi-core CPUs and accelerating the execution speed of batch tasks.

[0143] Specifically, for ease of understanding, please refer to Figure 7 , Figure 7 This is a schematic diagram illustrating the input and output relationship between target operators in an embodiment of this application. The following description, in conjunction with the diagram, will be provided:

[0144] (1) Please refer to Figure 7In Figure (A), the input to the first objective operator is a first data stream (i.e., N = 1). The first objective operator processes this first data stream (e.g., through mapping) to obtain a new data stream, the second data stream (i.e., M = 1). This second data stream is then used as the input to the second objective operator, which processes it (e.g., through mapping) to obtain a new data stream, the third data stream (i.e., Q = 1).

[0145] (2) Please refer to Figure 7 In Figure (B), the input to the first objective operator is a first data stream (i.e., N = 1). The first objective operator processes this first data stream (e.g., grouping) to obtain multiple new data streams, namely, the second data streams (i.e., M > 1). These multiple second data streams are then used as input to the second objective operator. After processing these second data streams (e.g., merging) by the second objective operator, a new data stream, namely, the third data stream (i.e., Q = 1), is obtained.

[0146] (3) Please refer to Figure 7 In Figure (C), the input to the first objective operator is multiple first data streams (i.e., N > 1). The first objective operator processes these multiple first data streams (e.g., merges them) to obtain a new data stream, the second data stream (i.e., M = 1). This second data stream is then used as the input to the second objective operator, which processes it (e.g., maps it) to obtain a new data stream, the third data stream (i.e., Q = 1).

[0147] (4) Please refer to Figure 7 In Figure (D), the input to the first objective operator is multiple first data streams (i.e., N > 1). The first objective operator processes these first data streams (e.g., through mapping) to obtain multiple new data streams, namely, second data streams (i.e., M > 1). These multiple second data streams are then used as input to the second objective operator. After processing these second data streams (e.g., through multiple processing steps), a new data stream, namely, the third data stream (i.e., Q = 1), is obtained.

[0148] (5) Please refer to Figure 7 In Figure (E), the input to the first objective operator is a first data stream (i.e., N = 1). The first objective operator processes this first data stream (e.g., mapping) to obtain a new data stream, the second data stream (i.e., M = 1). This second data stream is then used as the input to the second objective operator, which processes it (e.g., grouping) to obtain multiple new data streams, the third data stream (i.e., Q > 1).

[0149] (6) Please refer to Figure 7 In Figure (F), the input to the first objective operator is a first data stream (i.e., N = 1). The first objective operator processes this first data stream (e.g., grouping) to obtain multiple new data streams, namely, the second data streams (i.e., M > 1). These multiple second data streams are then used as input to the second objective operator. The second objective operator processes these multiple second data streams (e.g., mapping) to obtain multiple new data streams, namely, the third data streams (i.e., Q > 1).

[0150] (7) Please refer to Figure 7 In Figure (G), the input to the first objective operator is multiple first data streams (i.e., N > 1). The first objective operator processes these multiple first data streams (e.g., merging them) to obtain a new data stream, the second data stream (i.e., M = 1). This second data stream is then used as the input to the second objective operator, which processes it (e.g., grouping it) to obtain multiple new data streams, the third data stream (i.e., Q > 1).

[0151] (8) Please refer to Figure 7 In Figure (H), the input to the first objective operator is multiple first data streams (i.e., N > 1). The first objective operator processes these multiple first data streams (e.g., through mapping) to obtain multiple new data streams, namely, second data streams (i.e., M > 1). These multiple second data streams are then used as input to the second objective operator, which processes them (e.g., through mapping) to obtain multiple new data streams, namely, third data streams (i.e., Q > 1).

[0152] It should be noted that in practical applications, the first batch processing task may include more target operators, such as the third target operator, the fourth target operator, etc. The number of target operators and the number of data streams in this application are only illustrative and should not be construed as limiting this application.

[0153] Secondly, this application provides a method for transferring data streams between target operators. This method enables efficient data transmission and processing based on multiple target operators, achieving the goal of constructing batch processing tasks through collaboration between operators. This improves the feasibility and operability of the solution.

[0154] Optionally, in the above Figure 5 Based on one or more corresponding embodiments, in another optional embodiment provided by the present application, the first target operator is a target mapping operator;

[0155] The first target operator is invoked to process N first data streams to obtain M second data streams, specifically including:

[0156] Call the interface of the target mapping operator to obtain the element mapping rules;

[0157] According to the element mapping rules, each element in the first data stream is transformed to obtain the second data stream.

[0158] In one or more embodiments, a method for implementing operations based on a target Map operator is described. As can be seen from the foregoing embodiments, the first target operator can be a target Map operator. The target Map operator is obtained by implementing the interface required by the basic Map operator.

[0159] Specifically, the SDK provides the following basic Map operators:

[0160] func Map(s Stream,mapper plugin.Mapper)Stream{

[0161] / / …

[0162] }

[0163] Here, s represents the input data stream (e.g., the first data stream). mapper represents the interface variable of the plugin.Mapper interface.

[0164] Based on the tasks required by the logic unit, the `plugin.Mapper` interface required by the basic Map operator is implemented, thus obtaining the target Map operator. The target Map operator can output an element while taking one element as input. By applying the same function to each element in the data stream (e.g., the first data stream) through Map operations, a new data stream (e.g., the second data stream) is returned, where each element in the new data stream is the result of processing the elements of the original data stream using the function.

[0165] For easier understanding, please refer to Figure 8 , Figure 8 This is a schematic diagram of data processing based on the target mapping operator in an embodiment of this application. As shown in the figure, taking the first data stream as [1,2,3,4,5...] as an example, assume the element mapping rule is "input an element, return the result of multiplying that element by 2". Based on this, a mapping operation is performed on the first data stream, applying the above doubling function to each element of the first data stream. The first element of the first data stream is 1, which becomes 2 after processing by the doubling function; the second element 2 becomes 4 after processing; the third element 3 becomes 6; the fourth element 4 becomes 8; and the fifth element 5 becomes 10. Thus, after the mapping operation, a second data stream of [2,4,6,8,10...] is obtained.

[0166] Furthermore, this application provides a method for implementing operations based on a target Map operator. Through this method, the target Map operator enables data transformation and mapping, converting data into a form more suitable for subsequent processing, facilitating analysis and computation. In addition, the Map operation enhances the flexibility of data processing, allowing for the definition of element mapping rules as needed to adapt to various complex data processing requirements.

[0167] Optionally, in the above Figure 5 Based on one or more corresponding embodiments, in another optional embodiment provided by the present application, the first target operator is a target splicing mapping operator;

[0168] The first target operator is invoked to process N first data streams to obtain M second data streams, specifically including:

[0169] The target concatenation mapping operator's interface is called to obtain the data mapping rules;

[0170] According to the data mapping rules, each element in the first data stream is transformed to obtain the element sequence corresponding to each element;

[0171] Based on the element order of the first data stream, the element sequence corresponding to each element is concatenated to obtain the second data stream.

[0172] In one or more embodiments, a method for implementing operations based on the target concatenation mapping (ConcatMap) operator is described. As can be seen from the foregoing embodiments, the first target operator can be a target ConcatMap operator. The target ConcatMap operator is obtained by implementing the interface required by the basic ConcatMap operator.

[0173] Specifically, the SDK provides the following basic ConcatMap operator:

[0174] func ConcatMap(s Stream,flatMapper plugin.FlatMapper)Stream{

[0175] / / ...

[0176] }

[0177] Here, s represents the input data stream (e.g., the first data stream). flatMapper represents the interface variable of the plugin.FlatMapper interface.

[0178] Based on the tasks required by the logic unit, the `plugin.FlatMapper` interface required by the basic `ConcatMap` operator is implemented, thereby obtaining the target `ConcatMap` operator. The target `ConcatMap` operator can output zero or more elements while taking one element as input. By applying the same function to each element in a data stream (e.g., the first data stream) through the `ConcatMap` operation, a new data stream (e.g., the second data stream) is returned, where each element is the result of processing the elements of the original data stream using the function.

[0179] For easier understanding, please refer to Figure 9 , Figure 9 This is a schematic diagram illustrating data processing based on the target concatenation mapping operator in an embodiment of this application. As shown in the figure, taking the first data stream as an example: Order 1: {Order Number: 101, Product List: [Product A, Product B]}, Order 2: {Order Number: 102, Product List: [Product C]}, Order 3: {Order Number: 103, Product List: [Product D, Product E]}...], assuming the data mapping rule is "expand the product list in each order so that each product becomes an independent output." Based on this, a mapping and concatenation operation is performed on the first data stream. First, each element in the first data stream is mapped to a product list. Then, these product lists are concatenated sequentially to form a second data stream output, where each product becomes an independent element in this second data stream.

[0180] It should be noted that the basic numeric concatenation mapping (ConcatMapEager) operator is somewhat similar to the basic ConcatMap operator. The difference lies in the fact that the basic ConcatMapEager operator generates an array of elements, that is, it writes the array sequentially to the data stream, while the basic ConcatMap operator generates a pipe, continuously reading data from this pipe and writing it to the corresponding output data stream until the data pipe is closed.

[0181] Furthermore, this application provides a method for implementing operations based on the target ConcatMap operator. Through this method, the target ConcatMap operator can be used to transform and map data, and the mapped data is output in order. This not only yields a data format more suitable for subsequent processing, facilitating analysis and calculation, but also ensures the orderliness of data processing. In addition, the ConcatMap operation enhances the flexibility of data processing, allowing for the definition of element mapping rules as needed to adapt to various complex data processing requirements.

[0182] Optionally, in the above Figure 5Based on one or more corresponding embodiments, in another optional embodiment provided by the present application, the first target operator is a target filtering operator;

[0183] The first target operator is invoked to process N first data streams to obtain M second data streams, specifically including:

[0184] Call the interface of the target filtering operator to obtain the element filtering rules;

[0185] According to the element filtering rules, each element in the first data stream is filtered to obtain a Boolean value for each element. The Boolean value is used to indicate whether the element meets the element filtering rules.

[0186] Based on the Boolean value of each element, retain all elements in the first data stream that satisfy the element filtering rules to obtain the second data stream.

[0187] In one or more embodiments, a method for implementing operations based on a target filtering operator is described. As can be seen from the foregoing embodiments, the first target operator can be a target filter operator. The target filter operator is obtained by implementing the interface required by the basic filter operator.

[0188] Specifically, the SDK provides the following basic filter operators:

[0189] func Filter(s Stream,filter plugin.Filter)Stream{

[0190] return FilterOpt(nil,s,filter)

[0191] }

[0192] Here, s represents the input data stream (e.g., the first data stream). filter represents the interface variable of the plugin.Filter interface.

[0193] Based on the tasks required by the logic unit, the `plugin.Filter` interface required by the basic `Filter` operator is implemented, thereby obtaining the target `Filter` operator. The target `Filter` operator executes a Boolean function on each element in the data stream (e.g., the first data stream), obtaining the corresponding Boolean value for each element, and retaining elements whose Boolean function output (i.e., Boolean value) is "true". The Boolean value indicates whether an element meets the element filtering rules, that is, whether it will be submitted to the downstream data stream (e.g., the second data stream), thus achieving the filtering purpose.

[0194] For easier understanding, please refer to Figure 10 , Figure 10 This is a schematic diagram of data processing based on a target filtering operator in an embodiment of this application. As shown in the figure, taking the first data stream as [1,2,3,4,5,6,7,8,9,10...] as an example, assume the element filtering rule is "filter out all even numbers". Based on this, a filtering operation is performed on the first data stream, retaining the elements that meet the condition (i.e., even numbers) and removing the elements that do not meet the condition (i.e., odd numbers). Thus, after the filtering operation, a second data stream of [2,4,6,8,10...] is obtained.

[0195] Furthermore, this application provides a method for implementing operations based on a target filter operator. Using this method, the target filter operator can extract data that meets the requirements from the data stream according to specific conditions, and remove irrelevant or unsuitable data, thus purifying the data and making subsequent processing more targeted. In addition, by filtering out unnecessary data in advance, the amount of data processed subsequently is reduced, thereby reducing the consumption of computing resources and improving data processing efficiency. Especially when processing large-scale data, it can significantly improve system performance.

[0196] Optionally, in the above Figure 5 Based on one or more corresponding embodiments, in another optional embodiment provided by the present application, the first target operator is a target reduction operator;

[0197] The first target operator is invoked to process N first data streams to obtain M second data streams, specifically including:

[0198] Call the interface of the target reduction operator to obtain the element calculation rules and batch output rules;

[0199] According to the element calculation rules, at least two elements in the first data stream are calculated step by step. Under the condition of satisfying the batch output rules, the target element corresponding to a batch is obtained, wherein the batch includes at least two elements in the first data stream.

[0200] A second data stream is generated based on the target element corresponding to each batch.

[0201] In one or more embodiments, a method for implementing operations based on a target reduction operator is described. As can be seen from the foregoing embodiments, the first target operator can be a target reduce operator. The target reduce operator is obtained by implementing the interface required by the basic reduce operator.

[0202] Specifically, the SDK provides the following basic Reduce operators:

[0203] func Reduce(s Stream,reducer plugin.Reducer)Stream{

[0204] / / ...

[0205] }

[0206] Here, s represents the input data stream (e.g., the first data stream). reducer represents the interface variable of the plugin.Reducer interface.

[0207] Based on the tasks required by the logical unit, the `plugin.Reducer` interface required by the basic Reduce operator is implemented, thus obtaining the target Reduce operator. The target Reduce operator is used to perform reduction operations "rolling" on the data stream (e.g., the first data stream). It combines the current element with the value obtained from the last reduction and then outputs it as the element in the data stream (e.g., the second data stream). The specific implementation of the `plugin.Reducer` interface determines how the elements are reduced. Its returned boolean value indicates whether the current batch of reduction transformations has ended; that is, a boolean value of "true" indicates that the batch output rules are satisfied. The other return value represents the result of the reduction transformation.

[0208] For easier understanding, please refer to Figure 11 , Figure 11 This is a schematic diagram of data processing based on the target reduction operator in an embodiment of this application. As shown in the figure, taking the first data stream [1,2,3,4,5,6,7,8,9,10,11,12...] as an example, assume the element calculation rule is "add each element sequentially", and the batch output rule is "every three elements form a batch". Based on this, a reduction operation is performed on the first data stream. First, the first element 1 in the first data stream is added to the second element 2 to get 3. Then, 3 is added to the third element 3 to get 6. At this time, the batch output rule is satisfied, that is, 6 is output as the first element in the second data stream. And so on. Thus, after the reduction operation, a second data stream [6,15,24,33...] is obtained.

[0209] Furthermore, this application provides a method for implementing operations based on a target Reduce operator. Through this method, the elements in the data stream are aggregated according to specified logic using the target Reduce operator, thereby enabling the rapid extraction of representative statistical information from large amounts of data. In addition, the Reduce operation can transform complex data structures into simpler, more understandable, and easier-to-process forms, reducing the complexity of data processing and facilitating subsequent analysis and operations.

[0210] Optionally, in the above Figure 5 Based on one or more corresponding embodiments, in another optional embodiment provided by the present application, the first target operator is a target viewing operator;

[0211] The first target operator is invoked to process N first data streams to obtain M second data streams, specifically including:

[0212] Call the interface of the target viewing operator to obtain the element viewing rules;

[0213] Based on the element viewing rules, the first data stream is bypassed, and a second data stream is generated, wherein the elements in the second data stream are consistent with the elements in the first data stream.

[0214] In one or more embodiments, a method for implementing operations based on a target Peek operator is described. As can be seen from the foregoing embodiments, the first target operator can be a target Peek operator. The target Peek operator is obtained by implementing the interface required by the basic Peek operator.

[0215] Specifically, the SDK provides the following basic Peek operators:

[0216] func Peek(opt*Options,s Stream,peeper plugin.Peeper)Stream{

[0217] / / ...

[0218] }

[0219] Here, s represents the input data stream (e.g., the first data stream). peeper represents the interface variable of the plugin.Peeper interface. opt represents the control parameter.

[0220] Based on the tasks required by the logic unit, the `plugin.Peeper` interface required by the basic Peek operator is implemented, thereby obtaining the target Peek operator. The target Peek operator is used to bypass data from the data stream (e.g., the first data stream) in the form of a probe. It is important to note that the data stream does not change after passing through the target Peek operator; that is, the new data stream (e.g., the second data stream) is consistent with the original data stream (e.g., the first data stream).

[0221] For easier understanding, please refer to Figure 12 , Figure 12 This is a schematic diagram illustrating data processing based on the target viewing operator in an embodiment of this application. As shown in the figure, taking the first data stream as [{"Zhang San, 25"}, {"Li Si, 30"}, {"Wang Wu, 28"}...] as an example, assume the element viewing rule is "extract the name of each person". Based on this, extracting is performed on the first data stream, which is equivalent to setting an "extraction point" on the path traversed by the first data stream. When the first data stream passes through this "extraction point", the name portion will be extracted, outputting "Zhang San", "Li Si", and "Wang Wu" in sequence. The second data stream remains [{"Zhang San, 25"}, {"Li Si, 30"}, {"Wang Wu, 28"}...], unaffected by the element extraction.

[0222] Furthermore, this application provides a method for implementing operations based on a target peek operator. This method allows users to view portions or specific characteristics of data (e.g., data format, range, presence of outliers, etc.) without altering the data or processing operations. This helps in quickly understanding the general state of the data. It not only provides a basis for formulating data processing strategies but also helps developers promptly identify errors or deviations in the data processing flow, improving the accuracy and reliability of data processing.

[0223] Optionally, in the above Figure 5 Based on one or more corresponding embodiments, in another optional embodiment provided by the present application, the first target operator is a target merging operator;

[0224] The first target operator is invoked to process N first data streams to obtain M second data streams, specifically including:

[0225] The target merging operator is used to merge at least two first data streams to obtain a second data stream, wherein all elements in the second data stream originate from at least two first data streams.

[0226] In one or more embodiments, a method for implementing operations based on a target merge operator is described. As can be seen from the foregoing embodiments, the first target operator can be a target merge operator. The target merge operator is obtained based on the base merge operator.

[0227] Specifically, the SDK provides the following basic Merge operators:

[0228] func Merge(s Stream,streams...Stream)Stream{

[0229] / / ...

[0230] }

[0231] Where 's' represents the input data stream (e.g., a first data stream). 'Stream' represents other input data streams (e.g., other first data streams).

[0232] Based on the task required by the logical unit, a target merge operator can be constructed based on the basic merge operator. The target merge operator is used to merge at least two data streams (e.g., the first data stream) to obtain a new data stream (e.g., the second data stream). The order of elements in the at least two data streams is not guaranteed during the merging process.

[0233] For easier understanding, please refer to Figure 13 , Figure 13 This is a schematic diagram of data processing based on the target merging operator in an embodiment of this application. As shown in the figure, taking two first data streams as an example, one first data stream is [15,16,18,17,19,20,21...] and the other first data stream is [18,19,21,20,22,23,24...]. Based on this, a merging operation is performed on these two first data streams. First, the first element can be taken from one first data stream, and then the first element can be taken from the other first data stream, and so on. The final output merged second data stream is [15,18,16,19,18,21,17,20,19,22,20,23,21,24...].

[0234] Furthermore, this application provides a method for implementing operations based on a target merge operator. Using this method, multiple data streams can be merged into a unified data stream using the target merge operator, facilitating unified management, analysis, and processing. This avoids the hassle of frequent switching and operations between different data sources, improving the efficiency and convenience of data processing. In addition, by merging data streams from different dimensions or perspectives, more comprehensive information can be provided for data analysis, making the analysis results more in-depth and accurate.

[0235] Optionally, in the above Figure 5 Based on one or more corresponding embodiments, in another optional embodiment provided by the present application, the first target operator is a target grouping operator;

[0236] The first target operator is invoked to process N first data streams to obtain M second data streams, specifically including:

[0237] Call the interface of the target grouping operator to obtain the element grouping rules;

[0238] Based on the element grouping rules, each element in the first data stream is grouped to obtain the group identifier corresponding to each element;

[0239] Based on the group identifier corresponding to each element, elements belonging to the same group identifier are added to the same data stream to obtain at least one second data stream.

[0240] In one or more embodiments, a method for implementing operations based on a target GroupBy operator is described. As can be seen from the foregoing embodiments, the first target operator can be a target GroupBy operator. The target GroupBy operator is obtained by implementing the interface required by the basic GroupBy operator.

[0241] Specifically, the SDK provides the following basic GroupBy operator:

[0242] func GroupBy(s Stream,keySelector plugin.Mapper)Stream{

[0243] / / …

[0244] }

[0245] Here, s represents the input data stream (e.g., the first data stream). keySelector represents the interface variable of the plugin.Mapper interface.

[0246] Based on the tasks required by the logical unit, the `plugin.Mapper` interface required by the basic `GroupBy` operator is implemented, thereby obtaining the target `GroupBy` operator. The target `GroupBy` operator can map each element in the data stream (e.g., the first data stream) to the corresponding group identifier. Elements belonging to the same group identifier will be added to the same data stream (i.e., the second data stream).

[0247] For easier understanding, please refer to Figure 14 , Figure 14 This is a schematic diagram of data processing based on a target grouping operator in an embodiment of this application. As shown in the figure, taking the first data stream as [(apple, 5), (banana, 3), (apple, 7), (orange, 4), (banana, 6)...] as an example, each element represents "(fruit name, sales quantity)". Assume the element grouping rule is "grouping based on fruit name". Based on this, a grouping operation is performed on the first data stream to obtain three second data streams: one second data stream is [(apple, 5), (apple, 7)...], one second data stream is [(banana, 3), (banana, 6)...], and one second data stream is [(orange, 4)...].

[0248] Furthermore, this application provides a method for implementing operations based on a target GroupBy operator. Using this method, the target GroupBy operator can divide the data stream into different groups according to a specified key or condition, giving the data a clear structure and facilitating unified analysis and processing of data with similar characteristics. In addition, after grouping, various aggregation operations can be performed on each group of data, such as calculating the average, sum, and count, thereby quickly obtaining statistical information from different groups.

[0249] Optionally, in the above Figure 5 Based on one or more corresponding embodiments, in another optional embodiment provided by the present application, the first target operator is a target identity operator;

[0250] The first target operator is invoked to process N first data streams to obtain M second data streams, specifically including:

[0251] Based on the target identity operator, a second data stream is generated from the first data stream, wherein the elements in the second data stream are consistent with the elements in the first data stream.

[0252] In one or more embodiments, a method for implementing operations based on a target identity operator is described. As can be seen from the foregoing embodiments, the first target operator can be a target identity operator. The target identity operator is obtained based on the base identity operator.

[0253] Specifically, the SDK provides the following basic Identity operators:

[0254] func Identity(s Stream)Stream{

[0255] return s

[0256] }

[0257] Here, s represents the input data stream (e.g., the first data stream).

[0258] Based on the tasks required by the logical unit, the basic Identity operator can be directly used as the target Identity operator; that is, the basic Identity operator does not require the user to implement an interface. It is important to note that the data flow (e.g., the first data flow) will not change after passing through the target Identity operator; that is, the new data flow (e.g., the second data flow) will remain consistent with the original data flow (e.g., the first data flow).

[0259] For easier understanding, please refer to Figure 15 , Figure 15 This is a schematic diagram of data processing based on the target identity operator in an embodiment of this application. As shown in the figure, taking the first data stream as [1,2,3,4,5,6,7,8,9,10,11,12...] as an example, after the identity operation, a second data stream is obtained as [1,2,3,4,5,6,7,8,9,10,11,12...], that is, the second data stream is consistent with the first data stream.

[0260] Furthermore, this application provides a method for implementing operations based on a target Identity operator. By utilizing the target Identity operator, it is ensured that the data remains unchanged during processing, completely transmitting input data as output data and guaranteeing the originality and integrity of the data. In addition, in complex data processing pipelines, using Identity operations to connect different processing stages makes the entire data processing flow clearer, easier to understand, and facilitates workflow construction and debugging.

[0261] Optionally, in the above Figure 5Based on one or more corresponding embodiments, in another optional embodiment provided by the present application, the execution of the first batch task in response to a subscription operation for the first batch task specifically includes:

[0262] In response to the subscription operation for the first batch task, the interface of the target observation operator is called to obtain the data processing rules;

[0263] According to the data processing rules, execute the first batch processing task.

[0264] In one or more embodiments, a method for implementing operations based on the target Observer operator is described. As shown in the foregoing embodiments, the batch processing task requires at least one subscription operation at the end to consume data from the entire topology. The subscription operation involves the target Observer operator, which will be explained below with examples.

[0265] Specifically, triggering a subscription operation will drive the batch task to run, that is:

[0266]

[0267] Here, s represents the input data stream, and ob represents an interface variable of type Observer.

[0268] As we can see, the `Subscribe` function acts as a subscription mechanism. It receives a data stream and applies Observer operations to that stream. Specifically, if normal data is received, the `OnNext` function is called. If an error is encountered, the `OnError` function is called for handling. If the batch data processing task has finished, the `OnDone` function is called for processing. The current status of the subscription operation can also be obtained by calling `GetStatus`.

[0269] It's important to note that the entire batch processing task's execution, pause, and termination are controlled via Go's pipeline. If subscribers stop consuming data from the upstream data stream pipeline, the entire batch processing task pauses. If a data stream pipeline is closed, the entire batch processing task terminates.

[0270] Secondly, this application provides a method for implementing operations based on a target Observer operator. Using this method, changes in data can be observed in real time using the target Observer operator, and corresponding processing operations can be triggered immediately when the data changes, ensuring the timeliness and accuracy of data processing, thereby achieving the subscription operation.

[0271] Optionally, in the above Figure 5Based on one or more corresponding embodiments, in another optional embodiment provided by this application, for each logic unit in the first logic unit set, the basic operator corresponding to the logic unit is obtained, specifically including:

[0272] For each logical unit in the first set of logical units, if the Software Development Kit (SDK) provides the basic operator corresponding to the logical unit, obtain the basic operator corresponding to the logical unit.

[0273] For each logical unit in the first set of logical units, if the SDK does not provide the basic operator corresponding to the logical unit, the basic operator corresponding to the logical unit is generated in response to the operator construction operation.

[0274] In one or more embodiments, a method for obtaining basic operators is described. As can be seen from the foregoing embodiments, the SDK can provide documentation, code examples, and callback functions corresponding to the basic operators for users to use. Based on this, the basic operators provided by the SDK can be used directly.

[0275] Specifically, if the SDK does not provide the required basic operators, users can also construct their own basic operators to meet their specific needs. For example, suppose a basic operator needs to be constructed for switching the Map operator on and off, namely:

[0276]

[0277]

[0278] Therefore, by calling the Disable() and Enable() methods of the SwitchableStream interface, the functionality of the Map operator can be turned on or off at any time.

[0279] Secondly, this application provides a method for obtaining basic operators. Using this method, if the SDK already provides the required basic operators, they can be used directly. If the SDK does not provide the required basic operators, existing basic operators can be quickly extended to add other basic operators. It is evident that this method not only has low operational difficulty, but the extended operators can also be directly embedded into the existing topology without modifying the old logic, thereby improving the convenience of functional expansion.

[0280] Optionally, in the above Figure 5 In addition to one or more corresponding embodiments, another optional embodiment provided in this application may further include:

[0281] Obtain a second set of logical units, wherein the second set of logical units includes X logical units required to execute the same batch task, each logical unit is used to indicate an operation, and X is an integer greater than or equal to 1;

[0282] In the case where there are duplicate logic units in the second logic unit set and the first logic unit set, Y logic units are obtained from the second logic unit set. The Y logic units include logic units that are not duplicated in the second logic unit set and the first logic unit set. Y is an integer greater than or equal to 1 and less than X.

[0283] Obtain the basic operators corresponding to the logic units among the Y logic units;

[0284] For each of the Y logic units, obtain the target operator corresponding to the logic unit based on the operation indicated by the logic unit and the corresponding basic operator;

[0285] The X target operators are concatenated to obtain the second batch task, wherein the X target operators include the target operators corresponding to each logical unit in the second set of logical units;

[0286] In response to the subscription operation for the second batch task, execute the second batch task.

[0287] In one or more embodiments, a method for constructing multiple batch execution tasks is described. As can be seen from the foregoing embodiments, multiple batch execution tasks can be constructed based on configuration parameters. That is, the configuration parameters control which operations constitute the batch execution task, and an operation can be plugged in or removed at any time during runtime, thereby changing the execution logic of the batch execution task.

[0288] Specifically, for ease of understanding, please refer to Figure 16 , Figure 16 This is a schematic diagram illustrating the execution of multiple batch processing tasks in an embodiment of this application. The figure shows 12 batch processing tasks, each comprising 9 logical units. These 12 tasks involve reading data from different data sources and pushing it to three channels: Application A, Application B, and Application C. It is evident that the only difference between the 12 batch processing tasks is the target operator for reading files and the target operator for pushing data; the code for other target operators is completely reusable.

[0289] For example, constructing the first batch processing task requires the following logical units: "Read Task Parameters", "Read Task Cursor", "Read ES", "Query File Metadata", "Filter Documents Meeting Requirements", "Trace Reporting", "Get Text", "Push to Application A", and "Persist Task Cursor". Therefore, the first set of logical units corresponding to the first batch processing task includes nine logical units.

[0290] For example, constructing the second batch task requires the following logical units: "Read Task Parameters", "Read Task Cursor", "Read ES", "Query File Metadata", "Filter Documents Meeting Requirements", "Trace Reporting", "Get Text", "Push to Application B", and "Persist Task Cursor". Therefore, the second set of logical units corresponding to the second batch task includes 9 logical units (X = 9 in this case).

[0291] The only difference between the first and second batch processing tasks is that the first task requires the logical unit corresponding to "push to application A," while the second task requires the logical unit corresponding to "push to application B." All other logical units are the same. Since the second set of logical units overlaps with the first set by eight units, the corresponding eight target operators can be reused. For example, if both the first and second batch processing tasks need to extract data from a Structured Query Language Database Management System (MySQL) for subsequent processing, and the first task has already implemented the target operator for extracting data from MySQL, the second task only needs to specify the same target through configuration parameters, eliminating the need for redundant development.

[0292] The second set of logical units contains only one non-overlapping logical unit (Y = 1 at this point). Therefore, it is necessary to obtain the corresponding target operator based on the operation indicated by the logical unit corresponding to "Push to Application B" and the corresponding basic operator. Then, the existing eight target operators are concatenated to construct the second batch task. Based on this, the second batch task is executed in response to a subscription operation for it.

[0293] It's important to note that data extraction is abstracted into a class of operators. Data can be extracted from MySQL, local files, or input parameters. If only the data extraction source changes, while the subsequent processing logic remains the same, then when creating a new batch task, only the target operator corresponding to the data extraction needs to be switched. Furthermore, the target operator can be reused.

[0294] Understandably, batch processing tasks are composed of multiple target operators. An implementation of a particular target operator (i.e., an operation with a specific function) can be used by both batch processing task A and batch processing task B, thus offering high code reusability. If a new batch processing task C needs to be designed, and batch processing task C differs only slightly from existing batch processing tasks B or A (i.e., some target operator implementations are different), then only the differing target operators need to be reimplemented.

[0295] Secondly, this application provides a method for constructing multiple batch processing tasks. Through this method, different batch processing tasks can reuse operations; that is, existing target operators can be reused among different batch processing tasks. This avoids code redundancy and facilitates the rapid deployment of new batch processing tasks. Each operation only needs to focus on the output of the upstream operation and the input provided to the downstream operation; each operation is clearly defined and does not interfere with others, thus achieving a high cohesion and low coupling effect.

[0296] In summary, the following section, with illustrations, describes the process of writing data in batches to the Elasticsearch (ES) search engine. For easier understanding, please refer to [link to documentation / reference]. Figure 17 , Figure 17 This is a schematic diagram of a batch processing task topology in an embodiment of this application. As shown in the figure, different operations correspond to specific operators, exemplarily:

[0297] (1) The operation of receiving data (ReceiveData) uses the ConcatMap operator;

[0298] (2) The ParseEvent operation uses the Map operator;

[0299] (3) The operation of filtering illegal events (FilterlllegalEvent) uses the Filter operator;

[0300] (4) The operation of recording the active document (RecordActiveDoc) uses the Peek operator;

[0301] (5) The operation of splitting the data stream (SplitStream) uses the GroupBy operator;

[0302] (6) The operation of fetching data blocks (FetchBlockData) uses the ConcatMapEager operator;

[0303] (7) Operations that return a data stream (Identity) use the Identity operator;

[0304] (8) The operation of normalizing data (NormalizeData) uses the Map operator;

[0305] (9) The Merge Data operation uses the Merge operator;

[0306] (10) The DispatchMessage operation uses the GroupBy operator;

[0307] (11) The ReduceEvent operation uses the ConcatMapEager operator;

[0308] (12) The MergeEvent operation uses the ConcatMapEager operator;

[0309] (13) The operation of saving data to ES (SaveToES) uses the Observer operator.

[0310] When implementing specific operations, business logic involves implementing the interfaces required by the corresponding basic operator definitions. Taking the Map operator as an example, this means implementing the `plugin.Mapper` interface, as follows:

[0311] type Mapper interface{

[0312] Apply(context.Context,interface{})interface{}

[0313] InputType()reflect.Type

[0314] OutputType() reflects.Type

[0315] }

[0316] The Map operator receives a data stream (e.g., the first data stream), applies the Apply operation of the Mapper interface, and outputs the result as a new data stream (e.g., the second data stream). Downstream applications can then continue to receive the output streams of the Map operator and apply new operations. Therefore, business logic can obtain Map operators with different functionalities through different implementations of the plugin.Mapper interface.

[0317] The batch task execution device in this application is described in detail below. Please refer to [link / reference]. Figure 18 , Figure 18 This is a schematic diagram of one embodiment of the batch task execution device in this application. The batch task execution device 180 includes:

[0318] The acquisition module 1801 is used to acquire a first set of logical units, wherein the first set of logical units includes K logical units required to execute the same batch task, each logical unit is used to indicate an operation, and K is an integer greater than or equal to 1;

[0319] The acquisition module 1801 is also used to acquire the basic operator corresponding to each logical unit in the first set of logical units, wherein the basic operator is written based on the target language;

[0320] The acquisition module 1801 is also used to acquire the target operator corresponding to each logic unit in the first logic unit set, based on the operation indicated by the logic unit and the corresponding basic operator.

[0321] The processing module 1802 is used to concatenate K target operators to obtain the first batch task developed based on the target language, wherein the K target operators include the target operators corresponding to each logical unit in the first set of logical units;

[0322] Execution module 1803 is used to execute the first batch task in response to a subscription operation for the first batch task.

[0323] This application provides a batch task execution device. Using this device, a tool for other language ecosystems is designed, meaning that businesses developed in other languages ​​can use this tool to execute batch tasks. On the one hand, it eliminates the need for cross-language calls by the business side, reducing the difficulty of business development and maintenance. On the other hand, the tool designed in this application enables lightweight deployment and has a low barrier to entry.

[0324] Optionally, in the above Figure 18Based on the corresponding embodiments, in another embodiment of the batch task execution device 180 provided in this application, the first batch task includes a first target operator and a second target operator, wherein the second target operator is a target operator concatenated after the first target operator;

[0325] Execution module 1803 is specifically used to obtain N first data streams, where N is greater than or equal to 1;

[0326] The first target operator is called to process N first data streams to obtain M second data streams, where M is an integer greater than or equal to 1;

[0327] The second objective operator is invoked to process the M second data streams to obtain Q third data streams, where Q is an integer greater than or equal to 1.

[0328] This application provides a batch task execution device. Using this device, efficient data transmission and processing can be achieved based on multiple target operators, and the purpose of constructing batch tasks can be achieved through cooperation between operators. This improves the feasibility and operability of the solution.

[0329] Optionally, in the above Figure 18 Based on the corresponding embodiments, in another embodiment of the batch task execution device 180 provided in this application, the first target operator is a target mapping operator;

[0330] Execution module 1803 is specifically used to call the interface of the target mapping operator to obtain the element mapping rules;

[0331] According to the element mapping rules, each element in the first data stream is transformed to obtain the second data stream.

[0332] This application provides a batch task execution device. Using this device, the target Map operator can be used to transform and map data, converting it into a form more suitable for subsequent processing, facilitating analysis and computation. Furthermore, the Map operation enhances the flexibility of data processing, allowing data mapping rules to be defined as needed to adapt to various complex data processing requirements.

[0333] Optionally, in the above Figure 18 Based on the corresponding embodiments, in another embodiment of the batch task execution device 180 provided in this application, the first target operator is a target splicing mapping operator;

[0334] Execution module 1803 is specifically used to call the interface of the target splicing mapping operator to obtain the data mapping rules;

[0335] According to the data mapping rules, each element in the first data stream is transformed to obtain the element sequence corresponding to each element;

[0336] Based on the element order of the first data stream, the element sequence corresponding to each element is concatenated to obtain the second data stream.

[0337] This application provides a batch task execution apparatus. Using this apparatus, the target ConcatMap operator can be used to transform and map data, and output the mapped data in sequence. This not only yields a data format more suitable for subsequent processing, facilitating analysis and calculation, but also ensures the orderliness of data processing. Furthermore, the ConcatMap operation enhances the flexibility of data processing, allowing for the definition of element mapping rules as needed to adapt to various complex data processing requirements.

[0338] Optionally, in the above Figure 18 Based on the corresponding embodiments, in another embodiment of the batch task execution device 180 provided in this application, the first target operator is a target filtering operator;

[0339] Execution module 1803 is specifically used to call the interface of the target filtering operator to obtain the element filtering rules;

[0340] According to the element filtering rules, each element in the first data stream is filtered to obtain a Boolean value for each element. The Boolean value is used to indicate whether the element meets the element filtering rules.

[0341] Based on the Boolean value of each element, retain all elements in the first data stream that satisfy the element filtering rules to obtain the second data stream.

[0342] This application provides a batch processing task execution apparatus. Using this apparatus, a target filter operator can extract data that meets requirements from the data stream based on specific conditions, and remove irrelevant or unsuitable data, thus purifying the data and making subsequent processing more targeted. Furthermore, by filtering out unnecessary data in advance, the amount of data processed subsequently is reduced, thereby lowering the consumption of computing resources and improving data processing efficiency. This is particularly effective when processing large-scale data, significantly enhancing system performance.

[0343] Optionally, in the above Figure 18 Based on the corresponding embodiments, in another embodiment of the batch task execution device 180 provided in this application, the first target operator is a target reduction operator;

[0344] The execution module 1803 is specifically used to call the interface of the target reduction operator to obtain the element calculation rules and batch output rules;

[0345] According to the element calculation rules, at least two elements in the first data stream are calculated step by step. Under the condition of satisfying the batch output rules, the target element corresponding to a batch is obtained, wherein the batch includes at least two elements in the first data stream.

[0346] A second data stream is generated based on the target element corresponding to each batch.

[0347] This application provides a batch task execution apparatus. Using this apparatus, the elements in the data stream are aggregated according to specified logic using a target Reduce operator, thereby quickly extracting representative statistical information from a large amount of data. Furthermore, the Reduce operation can transform complex data structures into a simpler, more understandable, and easier-to-process form, reducing the complexity of data processing and facilitating subsequent analysis and operations.

[0348] Optionally, in the above Figure 18 Based on the corresponding embodiments, in another embodiment of the batch task execution device 180 provided in this application, the first target operator is a target viewing operator;

[0349] Execution module 1803 is specifically used to call the interface of the target viewing operator to obtain the element viewing rules;

[0350] Based on the element viewing rules, the first data stream is bypassed, and a second data stream is generated, wherein the elements in the second data stream are consistent with the elements in the first data stream.

[0351] This application provides a batch task execution apparatus. Using this apparatus, a target peek operator allows users to view portions or specific characteristics of the data (e.g., data format, range, presence of outliers, etc.) without altering the data or processing operations. This helps to quickly understand the general state of the data. It not only provides a basis for formulating data processing strategies but also helps developers promptly identify errors or deviations in the data processing flow, improving the accuracy and reliability of data processing.

[0352] Optionally, in the above Figure 18 Based on the corresponding embodiments, in another embodiment of the batch task execution device 180 provided in this application, the first target operator is a target merging operator;

[0353] The execution module 1803 is specifically used to merge at least two first data streams using a target merging operator to obtain a second data stream, wherein all elements in the second data stream originate from at least two first data streams.

[0354] This application provides a batch task execution device. Using this device, multiple data streams can be merged into a unified data stream using the target merge operator, facilitating unified management, analysis, and processing. This avoids the hassle of frequent switching and operations between different data sources, improving the efficiency and convenience of data processing. Furthermore, by merging data streams from different dimensions or perspectives, more comprehensive information can be provided for data analysis, resulting in more in-depth and accurate analysis results.

[0355] Optionally, in the above Figure 18 Based on the corresponding embodiments, in another embodiment of the batch task execution device 180 provided in this application, the first target operator is a target grouping operator;

[0356] Execution module 1803 is specifically used to call the interface of the target grouping operator to obtain the element grouping rules;

[0357] Based on the element grouping rules, each element in the first data stream is grouped to obtain the group identifier corresponding to each element;

[0358] Based on the group identifier corresponding to each element, elements belonging to the same group identifier are added to the same data stream to obtain at least one second data stream.

[0359] This application provides a batch task execution device. Using this device, the target GroupBy operator can divide a data stream into different groups based on a specified key or condition, giving the data a clear structure and facilitating unified analysis and processing of data with similar characteristics. Furthermore, after grouping, various aggregation operations can be performed on each group of data, such as calculating averages, sums, and counts, thereby quickly obtaining statistical information from different groups.

[0360] Optionally, in the above Figure 18 Based on the corresponding embodiments, in another embodiment of the batch task execution device 180 provided in this application, the first target operator is a target identity operator;

[0361] The execution module 1803 is specifically used to generate a second data stream based on the first data stream and the target identity operator, wherein the elements in the second data stream are consistent with the elements in the first data stream.

[0362] This application provides a batch task execution device. Using this device, the target Identity operator ensures that the data remains unchanged during processing, completely transmitting input data to output data and guaranteeing the originality and integrity of the data. Furthermore, in complex data processing pipelines, the Identity operation connects different processing stages, making the entire data processing flow clearer, easier to understand, and facilitating workflow construction and debugging.

[0363] Optionally, in the above Figure 18 Based on the corresponding embodiments, in another embodiment of the batch task execution device 180 provided in this application,

[0364] Execution module 1803 is specifically used to respond to the subscription operation for the first batch task by calling the interface of the target observation operator to obtain the data processing rules;

[0365] According to the data processing rules, execute the first batch processing task.

[0366] This application provides a batch task execution device. Using this device, the target Observer operator can observe data changes in real time and immediately trigger corresponding processing operations when data changes, ensuring the timeliness and accuracy of data processing, thereby achieving subscription operations.

[0367] Optionally, in the above Figure 18 Based on the corresponding embodiments, in another embodiment of the batch task execution device 180 provided in this application,

[0368] The acquisition module 1801 is specifically used to acquire the basic operator corresponding to each logical unit in the first set of logical units, provided that the software development kit (SDK) provides the basic operator corresponding to the logical unit.

[0369] For each logical unit in the first set of logical units, if the SDK does not provide the basic operator corresponding to the logical unit, the basic operator corresponding to the logical unit is generated in response to the operator construction operation.

[0370] This application provides a batch task execution device. Using this device, if the SDK already provides the required basic operators, they can be used directly. If the SDK does not provide the required basic operators, existing basic operators can be quickly extended to add other basic operators. Therefore, not only is the operation relatively simple, but the extended operators can also be directly embedded into the existing topology without modifying the old logic, thereby improving the convenience of functional expansion.

[0371] Optionally, in the above Figure 18 Based on the corresponding embodiments, in another embodiment of the batch task execution device 180 provided in this application,

[0372] The acquisition module 1801 is also used to acquire a second set of logical units, wherein the second set of logical units includes X logical units required to execute the same batch task, each logical unit is used to indicate an operation, and X is an integer greater than or equal to 1;

[0373] The acquisition module 1801 is further configured to acquire Y logical units from the second logical unit set when there are duplicate logical units in the second logical unit set and the first logical unit set, wherein the Y logical units include logical units that are not duplicated in the second logical unit set and the first logical unit set, and Y is an integer greater than or equal to 1 and less than X.

[0374] The acquisition module 1801 is also used to acquire the basic operators corresponding to the logic units among the Y logic units;

[0375] The acquisition module 1801 is also used to acquire the target operator corresponding to each of the Y logic units, based on the operation indicated by the logic unit and the corresponding basic operator.

[0376] The processing module 1802 is also used to concatenate the X target operators to obtain the second batch task, wherein the X target operators include the target operators corresponding to each logical unit in the second set of logical units;

[0377] Execution module 1803 is also used to execute the second batch task in response to a subscription operation for the second batch task.

[0378] This application provides a batch task execution apparatus. Using this apparatus, different batch tasks can reuse operations; that is, existing target operators can be reused among different batch tasks. This avoids code redundancy and facilitates the rapid deployment of new batch tasks.

[0379] This application also provides a computer device, such as... Figure 19 As shown, for ease of explanation, only the parts related to the embodiments of this application are shown. For specific technical details not disclosed, please refer to the method section of the embodiments of this application. In the embodiments of this application, a personal computer is used as an example for illustration:

[0380] Figure 19 This is a block diagram illustrating a portion of the structure of a personal computer related to the computer device provided in an embodiment of this application. (Reference) Figure 19A personal computer includes components such as: radio frequency (RF) circuitry 1910, memory 1920, input unit 1930, display unit 1940, sensor 1950, audio circuitry 1960, wireless fidelity (WiFi) module 1970, processor 1980, and power supply 1990. Those skilled in the art will understand that... Figure 19 The personal computer architecture shown does not constitute a limitation on a personal computer and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0381] The following is combined with Figure 19 A detailed introduction to the various components of a personal computer:

[0382] RF circuit 1910 can be used for receiving and transmitting signals during information transmission or calls. Specifically, it receives downlink information from the base station and processes it with processor 1980; additionally, it transmits uplink data to the base station. Typically, RF circuit 1910 includes, but is not limited to, an antenna, at least one amplifier, a transceiver, a coupler, a low-noise amplifier (LNA), and a duplexer. Furthermore, RF circuit 1910 can also communicate wirelessly with networks and other devices. The aforementioned wireless communication can use any communication standard or protocol, including but not limited to Global System for Mobile Communication (GSM), General Packet Radio Service (GPRS), Code Division Multiple Access (CDMA), Wideband Code Division Multiple Access (WCDMA), Long Term Evolution (LTE), email, and Short Message Service (SMS).

[0383] The memory 1920 can be used to store software programs and modules. The processor 1980 executes various functions and data processing of the personal computer by running the software programs and modules stored in the memory 1920. The memory 1920 may mainly include a program storage area and a data storage area. The program storage area may store the operating system, application programs required for at least one function (such as sound playback function, image playback function, etc.), etc.; the data storage area may store data created according to the use of the personal computer (such as audio data, telephone directory, etc.). In addition, the memory 1920 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device.

[0384] Input unit 1930 can be used to receive input numerical or character information, and to generate key signal inputs related to user settings and function control of the personal computer. Specifically, input unit 1930 may include touch panel 1931 and other input devices 1932. Touch panel 1931, also known as a touch screen, can collect touch operations performed by the user on or near it (such as operations performed by the user using a finger, stylus, or any suitable object or accessory on or near touch panel 1931), and drive corresponding connection devices according to a pre-set program. Optionally, touch panel 1931 may include two parts: a touch detection device and a touch controller. The touch detection device detects the user's touch position and the signal generated by the touch operation, and transmits the signal to the touch controller; the touch controller receives touch information from the touch detection device, converts it into touch point coordinates, and sends it to processor 1980, and can also receive and execute commands from processor 1980. In addition, touch panel 1931 can be implemented using various types such as resistive, capacitive, infrared, and surface acoustic wave. In addition to the touch panel 1931, the input unit 1930 may also include other input devices 1932. Specifically, other input devices 1932 may include, but are not limited to, one or more of the following: a physical keyboard, function keys (such as volume control buttons, power buttons, etc.), a mouse, and a joystick.

[0385] Display unit 1940 can be used to display information input by the user or information provided to the user, as well as various menus of a personal computer. Display unit 1940 may include a display panel 1941, optionally configured as a liquid crystal display (LCD), organic light-emitting diode (OLED), or similar device. Further, touch panel 1931 may cover display panel 1941. When touch panel 1931 detects a touch operation on or near it, it transmits the information to processor 1980 to determine the type of touch event. Subsequently, processor 1980 provides corresponding visual output on display panel 1941 based on the type of touch event. Although in Figure 19 In this embodiment, the touch panel 1931 and the display panel 1941 are two separate components to realize the input and output functions of the personal computer. However, in some embodiments, the touch panel 1931 and the display panel 1941 can be integrated to realize the input and output functions of the personal computer.

[0386] The personal computer may also include at least one sensor 1950, such as a light sensor, a motion sensor, and other sensors. Specifically, the light sensor may include an ambient light sensor and a proximity sensor. The ambient light sensor can adjust the brightness of the display panel 1941 according to the ambient light level, and the proximity sensor can turn off the display panel 1941 and / or the backlight when the personal computer is moved to the ear. As a type of motion sensor, an accelerometer sensor can detect the magnitude of acceleration in various directions (generally three axes). When stationary, it can detect the magnitude and direction of gravity and can be used for applications that recognize the posture of the personal computer (such as landscape / portrait switching, related games, magnetometer posture calibration), vibration recognition-related functions (such as pedometers, taps), etc. Other sensors that may be configured in the personal computer, such as gyroscopes, barometers, hygrometers, thermometers, and infrared sensors, will not be described in detail here.

[0387] Audio circuitry 1960, speaker 1961, and microphone 1962 provide an audio interface between the user and a personal computer. Audio circuitry 1960 converts received audio data into electrical signals and transmits them to speaker 1961, where speaker 1961 converts them into sound signals for output. On the other hand, microphone 1962 converts collected sound signals into electrical signals, which are received by audio circuitry 1960, converted into audio data, and then processed by processor 1980 before being transmitted via RF circuitry 1910 to, for example, another personal computer, or output to memory 1920 for further processing.

[0388] WiFi is a short-range wireless transmission technology. Personal computers using WiFi modules (1970) can help users send and receive emails, browse web pages, and access streaming media, providing wireless broadband internet access. Although Figure 19 WiFi module 1970 is shown, but it is understood that it is not an essential component of a personal computer and can be omitted as needed without changing the nature of the invention.

[0389] Processor 1980 is the control center of the personal computer, connecting various parts of the personal computer through various interfaces and lines. It performs various functions and processes data by running or executing software programs and / or modules stored in memory 1920 and calling data stored in memory 1920. Optionally, processor 1980 may include one or more processing units; optionally, processor 1980 may integrate an application processor and a modem processor, wherein the application processor mainly handles the operating system, user interface, and applications, and the modem processor mainly handles wireless communication. It is understood that the aforementioned modem processor may also not be integrated into processor 1980.

[0390] The personal computer also includes a power supply 1990 (such as a battery) that supplies power to various components. Optionally, the power supply can be logically connected to the processor 1980 through a power management system, thereby enabling functions such as charging, discharging, and power consumption management through the power management system.

[0391] Although not shown, personal computers may also include a camera, Bluetooth module, etc., which will not be described in detail here.

[0392] The steps performed by the computer device in the above embodiments can be based on this Figure 19 The computer device structure shown.

[0393] This application also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the methods described in the foregoing embodiments.

[0394] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the methods described in the foregoing embodiments.

[0395] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0396] In this application embodiment, the terms "module" or "unit" refer to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.

[0397] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection between apparatuses or units through some interfaces, and may be electrical, mechanical, or other forms.

[0398] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0399] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0400] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a server or terminal device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing computer programs, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0401] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A method for executing batch tasks, characterized in that, include: Obtain a first set of logical units, wherein the first set of logical units includes K logical units required to execute the same batch task, each logical unit is used to indicate an operation, and K is an integer greater than or equal to 1; For each logical unit in the first set of logical units, obtain the basic operator corresponding to the logical unit, wherein the basic operator is written based on the target language; For each logical unit in the first set of logical units, the target operator corresponding to the logical unit is obtained according to the operation indicated by the logical unit and the corresponding basic operator; The K target operators are concatenated to obtain the first batch processing task developed based on the target language, wherein the K target operators include the target operators corresponding to each logical unit in the first set of logical units; In response to the subscription operation for the first batch task, the first batch task is executed.

2. The execution method according to claim 1, characterized in that, The first batch processing task includes a first target operator and a second target operator, wherein the second target operator is the target operator concatenated after the first target operator; The execution of the first batch processing task includes: Obtain N first data streams, where N is greater than or equal to 1; The first target operator is invoked to process the N first data streams to obtain M second data streams, where M is an integer greater than or equal to 1; The second target operator is invoked to process the M second data streams to obtain Q third data streams, where Q is an integer greater than or equal to 1.

3. The execution method according to claim 2, characterized in that, The first target operator is a target mapping operator; The process of calling the first target operator to process the N first data streams to obtain M second data streams includes: The interface of the target mapping operator is called to obtain the element mapping rules; According to the element mapping rules, each element in the first data stream is transformed to obtain the second data stream.

4. The execution method according to claim 2, characterized in that, The first target operator is a target concatenation mapping operator; The process of calling the first target operator to process the N first data streams to obtain M second data streams includes: The interface of the target splicing mapping operator is called to obtain the data mapping rules; According to the data mapping rules, each element in the first data stream is transformed to obtain the element sequence corresponding to each element; Based on the element order of the first data stream, the element sequence corresponding to each element is concatenated to obtain the second data stream.

5. The execution method according to claim 2, characterized in that, The first target operator is a target filtering operator; The process of calling the first target operator to process the N first data streams to obtain M second data streams includes: The interface of the target filtering operator is called to obtain the element filtering rules; According to the element filtering rules, each element in the first data stream is filtered to obtain a Boolean value for each element. The Boolean value is used to indicate whether the element satisfies the element filtering rules. Based on the Boolean value of each element, retain all elements in the first data stream that satisfy the element filtering rules to obtain the second data stream.

6. The execution method according to claim 2, characterized in that, The first objective operator is an objective reduction operator; The process of calling the first target operator to process the N first data streams to obtain M second data streams includes: The interface of the target reduction operator is called to obtain the element calculation rules and batch output rules; According to the element calculation rules, at least two elements in the first data stream are calculated step by step. Under the condition of satisfying the batch output rules, a target element corresponding to a batch is obtained, wherein the batch includes at least two elements in the first data stream. A second data stream is generated based on the target element corresponding to each batch.

7. The execution method according to claim 2, characterized in that, The first target operator is the target viewing operator; The process of calling the first target operator to process the N first data streams to obtain M second data streams includes: The interface of the target viewing operator is called to obtain the element viewing rules; According to the element viewing rules, the first data stream is bypassed and a second data stream is generated, wherein the elements in the second data stream are consistent with the elements in the first data stream.

8. The execution method according to claim 2, characterized in that, The first target operator is the target merging operator; The process of calling the first target operator to process the N first data streams to obtain M second data streams includes: The target merging operator is used to merge at least two first data streams to obtain a second data stream, wherein all elements in the second data stream originate from the at least two first data streams.

9. The execution method according to claim 2, characterized in that, The first target operator is a target grouping operator; The process of calling the first target operator to process the N first data streams to obtain M second data streams includes: The interface of the target grouping operator is called to obtain the element grouping rules; According to the element grouping rules, each element in the first data stream is grouped to obtain the group identifier corresponding to each element; Based on the group identifier corresponding to each element, elements belonging to the same group identifier are added to the same data stream to obtain at least one second data stream.

10. The execution method according to claim 2, characterized in that, The first objective operator is the objective identity operator; The process of calling the first target operator to process the N first data streams to obtain M second data streams includes: Based on the target identity operator, a second data stream is generated from the first data stream, wherein the elements in the second data stream are consistent with the elements in the first data stream.

11. The execution method according to any one of claims 1 to 10, characterized in that, The step of executing the first batch task in response to a subscription operation for the first batch task includes: In response to the subscription operation for the first batch task, the interface of the target observation operator is invoked to obtain the data processing rules; The first batch processing task is executed according to the data processing rules.

12. The execution method according to any one of claims 1 to 11, characterized in that, The step of obtaining the basic operator corresponding to each logical unit in the first set of logical units includes: For each logical unit in the first set of logical units, if the Software Development Kit (SDK) provides the basic operator corresponding to the logical unit, obtain the basic operator corresponding to the logical unit; For each logical unit in the first set of logical units, if the SDK does not provide the basic operator corresponding to the logical unit, the basic operator corresponding to the logical unit is generated in response to the operator construction operation.

13. The execution method according to any one of claims 1 to 12, characterized in that, The method further includes: Obtain a second set of logical units, wherein the second set of logical units includes X logical units required to execute the same batch task, each logical unit being used to indicate an operation, and X being an integer greater than or equal to 1; In the case where there are duplicate logic units in the second logic unit set and the first logic unit set, Y logic units are obtained from the second logic unit set, wherein the Y logic units include logic units that are not duplicated in the second logic unit set and the first logic unit set, and Y is an integer greater than or equal to 1 and less than X; Obtain the basic operators corresponding to the logic units among the Y logic units; For each of the Y logic units, the target operator corresponding to the logic unit is obtained according to the operation indicated by the logic unit and the corresponding basic operator; The X target operators are concatenated to obtain the second batch task, wherein the X target operators include the target operators corresponding to each logical unit in the second set of logical units; In response to the subscription operation for the second batch task, the second batch task is executed.

14. A batch task execution device, characterized in that, include: The acquisition module is used to acquire a first set of logical units, wherein the first set of logical units includes K logical units required to execute the same batch task, each logical unit is used to indicate an operation, and K is an integer greater than or equal to 1; The acquisition module is further configured to acquire the basic operator corresponding to each logical unit in the first set of logical units, wherein the basic operator is written based on the target language; The acquisition module is further configured to acquire the target operator corresponding to each logic unit in the first logic unit set, based on the operation indicated by the logic unit and the corresponding basic operator. The processing module is used to concatenate K target operators to obtain a first batch task developed based on the target language, wherein the K target operators include the target operators corresponding to each logical unit in the first set of logical units; The execution module is used to execute the first batch task in response to a subscription operation for the first batch task.

15. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the batch task execution method according to any one of claims 1 to 13.

16. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the batch task execution method according to any one of claims 1 to 13.

17. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the batch task execution method according to any one of claims 1 to 13.

Images