Data stream processing method and apparatus, electronic device, and storage medium

Abstract

This disclosure relates to a data stream processing method, apparatus, electronic device, and storage medium. The method includes: responding to a data stream processing instruction, acquiring business requirement information corresponding to the data stream processing instruction, and creating a corresponding task information table for the data stream to be processed; configuring the task information table for the data stream to be processed based on the business requirement information; acquiring a first verification result of the task information table in a test stream; if the first verification result is successful, acquiring a second verification result of the task information table in a grayscale stream; and if the second verification result is successful, publishing the task information table to the official stream. This method can improve the iteration efficiency of data stream maintenance and creation.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and in particular to a data stream processing method, apparatus, electronic device, storage medium, and program product. Background Technology

[0002] The daily business needs of media resource transmission systems iterate very frequently, thus requiring the frequent creation or updating of data streams.

[0003] Currently, the way to create or update data streams is to manually manage the configuration of service nodes and business data streams through code and configuration change platforms. That is, when creating or updating data streams, code is modified according to business needs, submitted for testing, quality assurance verification, code merging, and finally manually deployed and launched to make changes. This method has low iteration efficiency. Summary of the Invention

[0004] This disclosure provides a data stream processing method, apparatus, electronic device, storage medium, and program product to at least solve the problem of low iteration efficiency in the methods of creating or updating data streams in related technologies. The technical solution of this disclosure is as follows:

[0005] According to a first aspect of the present disclosure, a data stream processing method is provided, comprising:

[0006] In response to a data stream processing instruction, the system obtains the business requirement information corresponding to the data stream processing instruction and creates a new task information table for the corresponding data stream to be processed.

[0007] Based on the aforementioned business requirements information, configure the task information table for the data stream to be processed;

[0008] Obtain the first verification result of the task information table in the test stream. If the first verification result is successful, obtain the second verification result of the task information table in the grayscale stream.

[0009] If the second verification process passes, the task information table is published to the formal process.

[0010] In one exemplary embodiment, the step of responding to a data stream processing instruction by obtaining business requirement information corresponding to the data stream processing instruction and creating a corresponding task information table for the data stream to be processed includes:

[0011] When the data stream processing instruction is a data stream change instruction, the change field information is obtained as the business requirement information corresponding to the data stream change instruction, and a corresponding task information table for the data stream to be changed is created.

[0012] In one exemplary embodiment, the step of obtaining business requirement information corresponding to the data stream processing instruction and creating a corresponding task information table for the data stream to be processed in response to the data stream processing instruction further includes:

[0013] When the data stream processing instruction is a data stream creation instruction, the data stream feature information is obtained as the business requirement information corresponding to the data stream creation instruction, and a corresponding task information table for the data stream to be created is created.

[0014] In one exemplary embodiment, the method further includes:

[0015] When a new data stream is obtained, determine the business type corresponding to the new data stream;

[0016] Based on the business type, the newly created data stream is added to the corresponding data stream group.

[0017] In one exemplary embodiment, the method further includes a data stream creation method, the data stream creation method comprising:

[0018] Generate the main task corresponding to the data stream to be created; the main task includes the baseline stream creation main task and the incremental stream creation main task, and both the baseline stream creation main task and the incremental stream creation main task include multiple sub-tasks;

[0019] The subtasks included in the main task of creating the benchmark stream are executed sequentially. When all the subtasks included in the main task of creating the benchmark stream have been completed, the creation of the benchmark stream is determined to be complete.

[0020] The incremental stream creation main task is executed sequentially, and the incremental stream creation is determined to be complete when all the subtasks included in the incremental stream creation main task have been executed.

[0021] In an exemplary embodiment, the sequential execution of the various subtasks included in the main task of creating the baseline stream includes:

[0022] Obtain the first priority order of each subtask included in the main task of creating the baseline stream;

[0023] The subtasks included in the main task of creating the baseline stream are executed sequentially according to the first priority order.

[0024] In an exemplary embodiment, the sequential execution of the various subtasks included in the main task of incremental stream creation includes:

[0025] Obtain the second priority order of each subtask included in the main task of creating the incremental stream;

[0026] The subtasks included in the incremental stream creation main task are executed sequentially according to the second priority order.

[0027] In one exemplary embodiment, the method further includes:

[0028] In response to a display command for a data stream, the data stream relationship diagram of the data stream is invoked; the data stream relationship diagram is used to provide a graphical representation of the input, intermediate services, output, and objects used in the data stream.

[0029] The data flow relationship diagram is shown.

[0030] According to a second aspect of the present disclosure, a data stream processing apparatus is provided, comprising:

[0031] The acquisition unit is configured to respond to a data stream processing instruction by acquiring the business requirement information corresponding to the data stream processing instruction and creating a task information table for the corresponding data stream to be processed.

[0032] The configuration unit is configured to execute the task information table of the data stream to be processed based on the business requirement information;

[0033] The verification unit is configured to perform a first verification process to obtain the task information table in the test stream, and if the first verification process result is passed, to obtain a second verification process result to obtain the task information table in the grayscale stream.

[0034] The publishing unit is configured to publish the task information table to the formal flow if the second verification process passes.

[0035] In an exemplary embodiment, the acquisition unit is configured to, when the data stream processing instruction is a data stream change instruction, acquire change field information as the business requirement information corresponding to the data stream change instruction, and create a corresponding task information table for the data stream to be changed.

[0036] In an exemplary embodiment, the acquisition unit is configured to acquire data stream feature information as business requirement information corresponding to the data stream creation instruction when the data stream processing instruction is a data stream creation instruction, and to create a corresponding task information table for the data stream to be created.

[0037] In one exemplary embodiment, the apparatus further includes an adding unit configured to perform the following actions when a new data stream is obtained: determining the service type corresponding to the new data stream; and adding the new data stream to the corresponding data stream group according to the service type.

[0038] In one exemplary embodiment, the apparatus further includes a data stream creation unit configured to execute a main task corresponding to generating a new data stream; the main task includes a baseline stream creation main task and an incremental stream creation main task, each of which includes multiple subtasks; the subtasks included in the baseline stream creation main task are executed sequentially, and when all subtasks included in the baseline stream creation main task are completed, the baseline stream creation is determined to be complete; the subtasks included in the incremental stream creation main task are executed sequentially, and when all subtasks included in the incremental stream creation main task are completed, the incremental stream creation is determined to be complete.

[0039] In an exemplary embodiment, the data stream creation unit is further configured to execute a first priority order of obtaining the subtasks included in the main task of creating the reference stream; and to execute the subtasks included in the main task of creating the reference stream in sequence according to the first priority order.

[0040] In an exemplary embodiment, the data stream creation unit is further configured to execute a second priority order of acquiring the subtasks included in the incremental stream creation main task; and to execute the subtasks included in the incremental stream creation main task sequentially according to the second priority order.

[0041] In one exemplary embodiment, the apparatus further includes a display unit configured to execute a display instruction for a data stream, invoking a data stream relationship diagram of the data stream; the data stream relationship diagram is used to provide a graphical representation of the inputs, intermediate services, outputs, and objects used in the data stream; and to display the data stream relationship diagram.

[0042] According to a third aspect of the present disclosure, an electronic device is provided, comprising:

[0043] processor;

[0044] Memory used to store the processor's executable instructions;

[0045] The processor is configured to execute the instructions to implement the method as described in any of the preceding methods.

[0046] According to a fourth aspect of the present disclosure, a computer-readable storage medium is provided that, when instructions in the computer-readable storage medium are executed by a processor of an electronic device, enables the electronic device to perform the method described in any of the preceding claims.

[0047] According to a fifth aspect of the present disclosure, a computer program product is provided, the computer program product including instructions that, when executed by a processor of an electronic device, enable the electronic device to perform the method as described in any of the preceding claims.

[0048] The technical solutions provided by the embodiments of this disclosure bring at least the following beneficial effects:

[0049] By configuring business requirements in the task information table, changes or creations of data streams can be achieved simply by testing and verifying the task information table. This eliminates the need to test, verify, and deploy the entire code after modification, greatly improving the iteration efficiency of data stream maintenance and creation.

[0050] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0051] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure, and are not intended to unduly limit this disclosure.

[0052] Figure 1 This is a flowchart illustrating a data stream processing method according to an exemplary embodiment.

[0053] Figure 2 This is a schematic diagram illustrating a process for processing business iteration work orders according to an exemplary embodiment.

[0054] Figure 3 This is a schematic diagram illustrating the display effect of a data stream group according to an exemplary embodiment.

[0055] Figure 4 This is a schematic diagram illustrating the front-end display effect of the management interface of a data flow group according to an exemplary embodiment.

[0056] Figure 5.1 This is a schematic diagram of the management interface of a reference stream according to an exemplary embodiment.

[0057] Figure 5.2 This is a schematic diagram of the management interface for incremental streams, illustrated according to an exemplary embodiment.

[0058] Figure 6 This is a schematic diagram illustrating the automated creation of incremental stream services according to an exemplary embodiment.

[0059] Figure 7 This is a flowchart illustrating a data stream creation method according to an exemplary embodiment.

[0060] Figure 8.1 This is a schematic diagram illustrating the data flow relationship of a reference stream according to an exemplary embodiment.

[0061] Figure 8.2This is a schematic diagram illustrating the data flow relationship of a reference stream according to an exemplary embodiment.

[0062] Figure 9 This is a structural block diagram of a data stream processing apparatus according to an exemplary embodiment.

[0063] Figure 10 This is a block diagram illustrating an electronic device according to an exemplary embodiment. Detailed Implementation

[0064] To enable those skilled in the art to better understand the technical solutions of this disclosure, the technical solutions in the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings.

[0065] It should be noted that the embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims.

[0066] It should be noted that the terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this disclosure are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this disclosure described herein can be implemented in orders other than those illustrated or described herein. It should also be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for display, data used for analysis, etc.) involved in this disclosure are all information and data authorized by the user or fully authorized by all parties.

[0067] In one exemplary embodiment, such as Figure 1As shown, a data stream processing method is provided. This embodiment illustrates the method by applying it to a terminal. Specifically, a data stream management platform can be built to handle data stream services for various businesses. The data stream management platform can be installed on the terminal. When the data stream management platform receives a data stream processing instruction, it obtains the business requirement information corresponding to the instruction and creates a corresponding task information table for the data stream to be processed. Based on the business requirement information, the task information table is configured and then published to the test stream for initial verification. If the initial verification passes, the task information table is published to the grayscale stream for a second verification. If the second verification passes, the task information table is deployed online. This disclosure aims to manage various business data stream services, visualize business requirement configuration, and support rapid iteration and deployment of requirements by building a data stream management platform, thereby solving problems such as complex business configuration, low iteration efficiency, and non-standard iteration processes.

[0068] It is understood that this method can also be applied to servers, and to systems including terminals and servers, and is implemented through the interaction between the terminals and servers. The terminals can be, but are not limited to, various personal computers, laptops, smartphones, tablets, IoT devices, and portable wearable devices. IoT devices can include smart speakers, smart TVs, smart air conditioners, smart in-vehicle devices, etc. Portable wearable devices can include smartwatches, smart bracelets, head-mounted devices, etc. The server can be implemented using a standalone server or a server cluster consisting of multiple servers. In this embodiment, the method includes the following steps:

[0069] In step S110, in response to the data stream processing instruction, the business requirement information corresponding to the data stream processing instruction is obtained, and a task information table for the corresponding data stream to be processed is created.

[0070] The data stream processing instructions include data stream creation instructions and data stream modification instructions. The data stream creation instructions are used to create new data streams, and the data stream modification instructions are used to modify data streams.

[0071] Among them, business requirement information can represent the business requirements put forward by the business party, such as changing the value of a certain field, adding a new field, etc.

[0072] In practice, when the data stream management platform installed on the terminal receives a data stream processing instruction, it can obtain the business requirement information corresponding to the data stream processing instruction and create a corresponding task information table for the data stream to be processed. This allows the newly created task information table to be configured as the task information table for the data stream to be processed, based on the business requirement information corresponding to the data stream processing instruction.

[0073] In step S120, a task information table for the data stream to be processed is configured based on business requirement information.

[0074] In practical implementation, when the data stream processing instruction is a data stream creation instruction, data stream characteristic information can be obtained as the business requirement information corresponding to the data stream creation instruction. Then, based on the data stream characteristic information, the corresponding task information table for the data stream to be created is configured. When the data stream processing instruction is a data stream modification instruction, i.e., a requirement to modify the data stream, the modification field information can be obtained as the business requirement information corresponding to the data stream modification instruction. Based on the modification field information, the corresponding task information table for the data stream to be modified is configured.

[0075] In step S130, the first verification result of the task information table in the test stream is obtained. If the first verification result is passed, the second verification result of the task information table in the grayscale stream is obtained.

[0076] In the specific implementation, refer to Figure 2 The flowchart shown illustrates the process of handling business iteration work orders. After configuring the task information table, it contains the business requirement information corresponding to the data stream being processed. The task information table can then be submitted to the approval terminal for review. Upon approval, the status of the task information table is changed to "approved" or "rejected." When the task information table is detected as "approved," it can be published to the test stream. In the test stream, the task information table undergoes its first verification process, and its status is changed to "verifying," while a message is sent to the verification terminal for verification. If the first verification process is successful, the verification terminal sends a verification success message to the requesting terminal, which then publishes the task information table to the grayscale stream (grayscale environment). In the grayscale stream, the approval, verification, and publication process is repeated to obtain the second verification result.

[0077] In step S140, if the second verification process passes, the task information table is uploaded.

[0078] Specifically, this disclosure also includes a connection interface between the test environment and the online environment, such as... Figure 2 As shown, the test environment includes test streams and grayscale streams. The online environment represents the environment where the official stream resides, also known as the production environment. Through this connection interface, the task information table of the test environment can be copied to the online environment. Thus, if the second verification process passes, the task information table can be copied to the online environment through the connection interface for approval and verification. Once the verification in the online environment passes, the task information table can be published to the official stream, i.e., the online task information table is deployed for practical application.

[0079] In the above data stream processing method, by configuring business requirement information in the task information table, changes or creation of the data stream can be achieved simply by testing and verifying the task information table. This eliminates the need to test, verify, and deploy the entire code after modification, greatly improving the iteration efficiency of data stream maintenance and creation.

[0080] In an exemplary embodiment, in step S110, in response to a data stream processing instruction, business requirement information corresponding to the data stream processing instruction is obtained, and a corresponding task information table for the data stream to be processed is created, including: when the data stream processing instruction is a data stream change instruction, obtaining change field information as the business requirement information corresponding to the data stream change instruction, and creating a corresponding task information table for the data stream to be changed.

[0081] In this embodiment, when the data stream processing instruction is a data stream change instruction, the change field information can be obtained as the business requirement information corresponding to the data stream change instruction, and then the task information table of the data stream to be changed can be configured so that the task information table can contain business requirement information, so that the data stream can be tested and verified through the flow of the task information table, replacing the complex operations such as modifying code and deploying online in the traditional method.

[0082] In practical applications, when the data stream processing instruction is a data stream change instruction, in addition to obtaining the change field information, the information of the application terminal, approval terminal, and verification terminal can also be obtained. At the same time, the task status is set, and the change field information, application terminal, approval terminal, verification terminal, and task status are all written into the task information table to obtain the task information table of the data stream to be changed. This allows for quick determination of the objects to which the task information table needs to be sent at each step based on the application terminal, approval terminal, and verification terminal information during the flow of the task information table, and determination of the next action based on the task status.

[0083] In an exemplary embodiment, in step S110, in response to a data stream processing instruction, business requirement information corresponding to the data stream processing instruction is obtained, and a corresponding task information table for the data stream to be processed is created. The method further includes: if the data stream processing instruction is a data stream creation instruction, obtaining data stream feature information as business requirement information corresponding to the data stream creation instruction, and creating a corresponding task information table for the data stream to be created.

[0084] The data stream characteristic information may include the data stream name, data stream export path, or the topic to be issued.

[0085] In this embodiment, when the data stream processing instruction is a data stream creation instruction, a task information table can be created in the data stream management platform. Information such as the data stream name, data stream export path, or topic to be issued for the data stream to be created can be written into the task information table to obtain the task information table of the data stream to be created. This allows the task information table to contain the business requirement information of the data stream to be created, so that the data stream can be automatically created, tested, and verified in the future, replacing the traditional method of R&D and code addition, and improving iteration efficiency.

[0086] In practical applications, in addition to obtaining data flow characteristic information such as the data flow name, data flow export path, or topic of the data flow to be created, information such as the application terminal, approval terminal, and verification terminal can also be obtained. At the same time, the task status is set, and the data flow characteristic information, application terminal, approval terminal, verification terminal, and task status are all written into the task information table to obtain the task information table of the data flow to be created. This allows for quick determination of the objects to which the task information table needs to be sent at each step when testing and verifying the new data flow, based on the application terminal, approval terminal, and verification terminal information, and determination of the next action based on the task status.

[0087] In one exemplary embodiment, the method further includes: when a newly created data stream is obtained, determining the service type corresponding to the newly created data stream, and adding the newly created data stream to the corresponding data stream group according to the service type.

[0088] For example, if a data stream comprises two types: a base stream and an incremental stream, then a data stream group can represent a collection of base streams and incremental streams. Considering the increasing business demands of media resource transmission systems, the number of data stream nodes also grows, leading to increased management difficulty. This disclosure also proposes a method of classifying numerous data streams according to business types and implementing group maintenance to address the problem of numerous data stream nodes and increased management difficulty as business develops. Furthermore, when a new data stream is obtained, the corresponding business type can be determined, and the new data stream can be added to the corresponding data stream group based on the business type, facilitating subsequent group maintenance.

[0089] Grouping and maintaining data streams allows for the encapsulation and maintenance of information related to data stream group dimensions, such as data stream name (categorized by business type or business line), data stream description, and interface personnel, as follows:

[0090] On the data flow management platform, the data flow group is displayed as follows: Figure 3 As shown, the information displayed for each data stream group includes: data stream group name, number of data streams, number of data sources, person in charge, creation time, update time, and corresponding operation controls.

[0091] The management of data flow groups mainly includes the management of the data flow group itself and the management of each data flow within the data flow group.

[0092] (1) Management of the data flow group itself includes adding, querying, and updating data flow groups. To further facilitate the management of data flow groups, different permission levels can be configured for each data flow group, and operations such as adding, deleting, and modifying administrators can be performed, facilitating long-term maintenance. The front-end display effect of the data flow group management interface is as follows: Figure 4 As shown, Figure 4 Taking data stream group A as an example, users can be added as administrators, existing administrators can be deleted, and the roles of administrators can be modified to manage the data stream group.

[0093] (2) Management of each data stream within a data stream group includes the management of the baseline stream and the management of incremental streams. The identifier of the data stream group (e.g., `id`) will exist as a foreign key in the baseline stream and incremental stream tables. A baseline stream or incremental stream can only uniquely bind to one data stream group, but a single data stream group can support multiple baseline streams or incremental streams. To adapt to business needs, this disclosure also provides operations for changing and adding baseline streams or incremental streams. Furthermore, to improve platform usability, the function of querying baseline streams or incremental streams has been added. For example... Figure 5.1 The diagram shows the management interface for a benchmark stream. In this interface, users can query benchmark streams by data source, data table, and storage path; add benchmark streams using the "Add" button; and modify benchmark streams by changing their data source, data table, and storage path. Figure 5.2 The diagram shows the management interface for incremental streams. In the management interface, users can query incremental streams through data sources, data tables, and data topics. Users can add incremental streams by clicking the "Add" button and modify incremental streams by modifying data sources, data tables, and data topics.

[0094] The aforementioned data flow management platform provides data flow group management and management services for each data flow within a data flow group, making it easy to view the relevant configurations of the data flow.

[0095] Traditionally, creating new data streams requires developers to add code, which is costly and inefficient. Therefore, this disclosure proposes a task information table for creating new streams. This automated service integrates version control systems, pipelines, container clouds, and other third-party services. A single work order-style task information table can generate baseline and incremental stream services, which are then automatically added to the data stream management platform for management and maintenance. (Reference) Figure 6 This is a schematic diagram illustrating an automated incremental stream creation service as an exemplary embodiment. The automated data stream creation functions include: integrating with a version control system to generate code, integrating with a pipeline to package and build code, integrating with a pipeline to deploy to a test environment, integrating with a container cloud to deploy to a production environment, integrating with a data bus to request consumption tasks, and integrating with the container cloud to start services. These functions all need to be completed asynchronously, and there are dependencies between them. Therefore, this disclosure abstracts the generation logic into main tasks and sub-tasks, which are organized and implemented by a unified automatic scheduling framework. The automated tasks are divided into baseline stream automated tasks and incremental stream automated tasks.

[0096] like Figure 7 As shown, in an exemplary embodiment, the data stream creation method includes:

[0097] Step S710: Generate the main task corresponding to the data stream to be created; the main task includes the baseline stream creation main task and the incremental stream creation main task, and both the baseline stream creation main task and the incremental stream creation main task include multiple sub-tasks.

[0098] Step S720: Execute each subtask included in the main task of creating the benchmark stream in sequence. When all subtasks included in the main task of creating the benchmark stream have been executed, the benchmark stream creation is determined to be complete.

[0099] Step S730: Execute each subtask included in the main task of incremental stream creation in sequence. When all subtasks included in the main task of incremental stream creation have been completed, the incremental stream creation is determined to be complete.

[0100] Among them, the baseline stream is a data processing tool downstream of the database, and the baseline can be understood as the existing data information in the database.

[0101] Incremental streams can represent newly added data in the database.

[0102] The main task of creating the baseline stream includes subtasks such as: building code, compiling, packaging and mirroring the code through a pipeline, and creating an environment, deploying code and starting services through a pipeline or container cloud.

[0103] The main task of incremental stream creation includes the following subtasks: building code, compiling, packaging and mirroring code through a pipeline, creating an environment through a pipeline or container cloud, applying for a data source through a data bus, and deploying code and starting services through a pipeline / container cloud.

[0104] In the specific implementation, when creating a new data stream, a task information table is first created, writing relevant information about the data stream to be created, such as the data stream name, data stream export path, and topic, and then submitted to the management terminal for approval. When approval is granted, the baseline stream creation master task and the incremental stream creation master task for the data stream to be created are generated, along with the various subtasks included in the baseline stream creation master task and the incremental stream creation master task. Under each master task, each subtask is executed sequentially according to priority. The next subtask begins execution after the previous one has finished. The execution of a subtask includes three main steps: obtaining subtask execution rights; executing the subtask logic; determining the subtask execution status and making subsequent status changes; and setting the master task to a completed state after all subtasks have been executed.

[0105] Furthermore, in an exemplary embodiment, step S720 involves sequentially executing each subtask included in the main task of creating the reference stream, including: obtaining a first priority order of each subtask included in the main task of creating the reference stream; and sequentially executing each subtask included in the main task of creating the reference stream according to the first priority order.

[0106] Specifically, the primary task of creating the baseline stream comprises the following subtasks, with the following priority order: building the code, compiling, packaging, and mirroring the code via a pipeline, and creating the environment, deploying the code, and starting the service via a pipeline or container cloud. Each subtask is executed sequentially according to this priority order, with the next subtask only executing after the previous one is completed. For example, the subtask "compiling, packaging, and mirroring the code via a pipeline" can only be executed after the subtask "compiling, packaging, and mirroring the code via a pipeline" is completed; similarly, the subtask "creating the environment, deploying the code, and starting the service via a pipeline or container cloud" can only be executed after the subtask "compiling, packaging, and mirroring the code via a pipeline" is completed. The baseline stream creation is considered complete when all subtasks are finished.

[0107] In an exemplary embodiment, step S730 involves sequentially executing each subtask included in the incremental stream creation main task, including: obtaining a second priority order of each subtask included in the incremental stream creation main task; and sequentially executing each subtask included in the incremental stream creation main task according to the second priority order.

[0108] Specifically, the second priority order of the subtasks included in the incremental stream creation main task is as follows: building the code, compiling, packaging, and mirroring the code through a pipeline, creating the environment through a pipeline or container cloud, requesting a data source through a data bus, and deploying the code and starting the service through a pipeline / container cloud. Each subtask is executed sequentially according to this second priority order, with the next subtask only executing after the previous one is completed. The incremental stream creation is considered complete when all subtasks are finished.

[0109] The above embodiments, by integrating services such as automatic code submission and container cloud deployment, and combining them with task information tables to automate the creation of data streams, can greatly improve iteration efficiency.

[0110] In an exemplary embodiment, the method further includes: in response to a display instruction for the data stream, invoking a data stream relationship diagram of the data stream; the data stream relationship diagram is used to provide a graphical representation of the input, intermediate services, output, and objects used in the data stream; and displaying the data stream relationship diagram.

[0111] The different data streams contain configuration information such as data source, data table, storage path, topic, limit on the number of rows returned by the query, and horizontal cascading.

[0112] Specifically, this embodiment provides a method for visually displaying data stream configuration information. When a display instruction for a data stream is received, the data stream relationship diagram of the data stream is invoked, and the configuration information of the data stream is displayed through the data stream relationship diagram.

[0113] refer to Figure 8.1 This is a schematic diagram of the data flow relationship of the baseline stream. The diagram shows information such as data tables, data sources, baseline storage paths, and baseline users. The server stores the data flow configuration information in the data tables, and the terminal calls the backend interface via an HTTP interface to display the data. (Reference) Figure 8.2 This is a schematic diagram of the data flow relationship of incremental streams. The data flow relationship diagram of incremental streams shows the data table monitored by incremental streams, the data source, and the topic (topic) to which incremental streams are issued. The server stores the above detailed information in the incremental stream data table, and the terminal calls the backend interface through the HTTP interface and displays it.

[0114] Understandably, traditional data flow configurations are maintained through code and configuration change platforms. Developers and testers need to have a good understanding of the code structure to understand the data flow configuration. After platformization, a data flow lineage diagram is used to graphically display data inputs, intermediate services, outputs, and users. The visual graph not only intuitively displays the flow information but also makes it easy for developers and testers to operate directly.

[0115] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0116] It is understood that the same / similar parts between the various embodiments of the methods described above in this specification can be referred to each other. Each embodiment focuses on the differences from other embodiments, and relevant parts can be referred to the description of other method embodiments.

[0117] Based on the same inventive concept, this disclosure also provides a data stream processing apparatus for implementing the data stream processing method described above.

[0118] Figure 9 This is a structural block diagram of a data stream processing apparatus according to an exemplary embodiment. (Refer to...) Figure 9 The device includes:

[0119] The acquisition unit 910 is configured to execute a response to a data stream processing instruction, acquire the business requirement information corresponding to the data stream processing instruction, and create a new task information table for the corresponding data stream to be processed.

[0120] Configuration unit 920 is configured to execute a task information table that configures the data stream to be processed based on business requirement information.

[0121] Verification unit 930 is configured to perform the following: obtain the first verification result of the task information table in the test stream; if the first verification result is passed, obtain the second verification result of the task information table in the grayscale stream.

[0122] Release unit 940 is configured to release the task information table to the formal flow if the second verification process passes.

[0123] In an exemplary embodiment, the acquisition unit 910 is configured to acquire change field information as business requirement information corresponding to the data flow change instruction when the data flow processing instruction is a data flow change instruction, and to create a corresponding task information table for the data flow to be changed.

[0124] In an exemplary embodiment, the acquisition unit 910 is configured to acquire data stream feature information as business requirement information corresponding to the data stream creation instruction when the data stream processing instruction is a data stream creation instruction, and to create a corresponding task information table for the data stream to be created.

[0125] In one exemplary embodiment, the apparatus further includes an adding unit configured to perform the following actions when a new data stream is obtained: determining the service type corresponding to the new data stream; and adding the new data stream to the corresponding data stream group according to the service type.

[0126] In one exemplary embodiment, the apparatus further includes a data stream creation unit configured to execute a main task corresponding to generating a new data stream; the main task includes a baseline stream creation main task and an incremental stream creation main task, both of which include multiple subtasks; each subtask of the baseline stream creation main task is executed sequentially, and when all subtasks of the baseline stream creation main task are completed, the baseline stream creation is determined to be complete; each subtask of the incremental stream creation main task is executed sequentially, and when all subtasks of the incremental stream creation main task are completed, the incremental stream creation is determined to be complete.

[0127] In an exemplary embodiment, the data stream creation unit is further configured to execute a first priority order of the subtasks included in the main task of obtaining the reference stream creation; and to execute the subtasks included in the main task of obtaining the reference stream creation in sequence according to the first priority order.

[0128] In one exemplary embodiment, the data stream creation unit is further configured to execute a second priority order of the subtasks included in the incremental stream creation main task; and to execute the subtasks included in the incremental stream creation main task in sequence according to the second priority order.

[0129] In one exemplary embodiment, the apparatus further includes a display unit configured to execute a display instruction for the data stream, invoke a data stream relationship diagram of the data stream; the data stream relationship diagram is used to provide a graphical representation of the input, intermediate services, output, and objects used in the data stream; and display the data stream relationship diagram.

[0130] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0131] Figure 10This is a block diagram illustrating an electronic device 1000 for implementing a data stream processing method according to an exemplary embodiment. For example, the electronic device 1000 may be a mobile phone, computer, digital broadcasting terminal, messaging device, game console, tablet device, medical device, fitness equipment, personal digital assistant, etc.

[0132] Reference Figure 10 The electronic device 1000 may include one or more of the following components: processing component 1002, memory 1004, power supply component 1006, multimedia component 1008, audio component 1010, input / output (I / O) interface 1012, sensor component 1014, and communication component 1016.

[0133] Processing component 1002 typically controls the overall operation of electronic device 1000, such as operations associated with display, telephone calls, data communication, camera operation, and recording operations. Processing component 1002 may include one or more processors 1020 to execute instructions to perform all or part of the steps of the methods described above. Furthermore, processing component 1002 may include one or more modules to facilitate interaction between processing component 1002 and other components. For example, processing component 1002 may include a multimedia module to facilitate interaction between multimedia component 1008 and processing component 1002.

[0134] Memory 1004 is configured to store various types of data to support the operation of electronic device 1000. Examples of such data include instructions for any application or method operating on electronic device 1000, contact data, phonebook data, messages, pictures, videos, etc. Memory 1004 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, optical disk, or graphene memory.

[0135] Power supply component 1006 provides power to various components of electronic device 1000. Power supply component 1006 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to electronic device 1000.

[0136] Multimedia component 1008 includes a screen that provides an output interface between the electronic device 1000 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touchscreen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touches, swipes, and gestures on the touch panel. The touch sensors may sense not only the boundaries of the touch or swipe action but also the duration and pressure associated with the touch or swipe operation. In some embodiments, multimedia component 1008 includes a front-facing camera and / or a rear-facing camera. When the electronic device 1000 is in an operating mode, such as a shooting mode or a video mode, the front-facing camera and / or the rear-facing camera may receive external multimedia data. Each front-facing camera and rear-facing camera may be a fixed optical lens system or have focal length and optical zoom capabilities.

[0137] Audio component 1010 is configured to output and / or input audio signals. For example, audio component 1010 includes a microphone (MIC) configured to receive external audio signals when electronic device 1000 is in an operating mode, such as call mode, recording mode, and voice recognition mode. The received audio signals may be further stored in memory 1004 or transmitted via communication component 1016. In some embodiments, audio component 1010 also includes a speaker for outputting audio signals.

[0138] I / O interface 1012 provides an interface between processing component 1002 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include, but are not limited to, home buttons, volume buttons, power buttons, and lock buttons.

[0139] Sensor assembly 1014 includes one or more sensors for providing state assessments of various aspects of electronic device 1000. For example, sensor assembly 1014 can detect the on / off state of electronic device 1000, the relative positioning of components such as the display and keypad of electronic device 1000, changes in position of electronic device 1000 or its components, the presence or absence of user contact with electronic device 1000, orientation or acceleration / deceleration of device 1000, and temperature changes of electronic device 1000. Sensor assembly 1014 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. Sensor assembly 1014 may also include a light sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, sensor assembly 1014 may also include an accelerometer, gyroscope, magnetometer, pressure sensor, or temperature sensor.

[0140] Communication component 1016 is configured to facilitate wired or wireless communication between electronic device 1000 and other devices. Electronic device 1000 can access wireless networks based on communication standards, such as WiFi, carrier networks (such as 2G, 3G, 4G, or 5G), or combinations thereof. In one exemplary embodiment, communication component 1016 receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In one exemplary embodiment, communication component 1016 also includes a near-field communication (NFC) module to facilitate short-range communication. For example, the NFC module may be implemented based on radio frequency identification (RFID) technology, Infrared Data Association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.

[0141] In an exemplary embodiment, the electronic device 1000 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the methods described above.

[0142] In one exemplary embodiment, a computer-readable storage medium including instructions is also provided, such as a memory 1004 including instructions, which can be executed by a processor 1020 of an electronic device 1000 to perform the above-described method. For example, the computer-readable storage medium may be a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage device, etc.

[0143] In one exemplary embodiment, a computer program product is also provided, which includes instructions that can be executed by a processor 1020 of an electronic device 1000 to perform the above-described method.

[0144] It should be noted that the above-mentioned apparatus, electronic equipment, computer-readable storage medium, computer program product, etc., may also include other implementation methods according to the description of the method embodiments. For specific implementation methods, please refer to the description of the relevant method embodiments, which will not be elaborated here.

[0145] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This disclosure is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the claims.

[0146] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

Claims

1. A data stream processing method, characterized in that, This is achieved through a data flow management platform, including: In response to a data stream processing instruction, the system obtains the business requirement information corresponding to the data stream processing instruction and creates a new task information table for the corresponding data stream to be processed. Based on the aforementioned business requirements information, configure the task information table for the data stream to be processed; Obtain the first verification result of the task information table in the test stream. If the first verification result is successful, obtain the second verification result of the task information table in the grayscale stream. If the second verification process passes, the task information table is published to the formal process. The data flow management platform classifies data flows according to business type and adds them to corresponding data flow groups for group maintenance. The data flows include base flows and incremental flows. The data flow group is a collection of base flows and incremental flows. The identifier of the data flow group exists in the base flow table and the incremental flow table as a foreign key. Each base flow or incremental flow is bound to a data flow group. The data flow management platform is also used to visualize the configuration information of data flows through a data flow relationship diagram.

2. The method according to claim 1, characterized in that, The step of responding to a data stream processing instruction by obtaining the business requirement information corresponding to the data stream processing instruction and creating a corresponding task information table for the data stream to be processed includes: When the data stream processing instruction is a data stream change instruction, the change field information is obtained as the business requirement information corresponding to the data stream change instruction, and a corresponding task information table for the data stream to be changed is created.

3. The method according to claim 1, characterized in that, The step of responding to a data stream processing instruction, obtaining the business requirement information corresponding to the data stream processing instruction, and creating a corresponding task information table for the data stream to be processed, further includes: When the data stream processing instruction is a data stream creation instruction, the data stream feature information is obtained as the business requirement information corresponding to the data stream creation instruction, and a corresponding task information table for the data stream to be created is created.

4. The method according to claim 3, characterized in that, The method further includes: When a new data stream is obtained, determine the business type corresponding to the new data stream; Based on the business type, the newly created data stream is added to the corresponding data stream group.

5. The method according to claim 3, characterized in that, The method further includes a data stream creation method, which includes: Generate the main task corresponding to the data stream to be created; the main task includes the baseline stream creation main task and the incremental stream creation main task, and both the baseline stream creation main task and the incremental stream creation main task include multiple sub-tasks; The subtasks included in the main task of creating the benchmark stream are executed sequentially. When all the subtasks included in the main task of creating the benchmark stream have been completed, the creation of the benchmark stream is determined to be complete. The incremental stream creation main task is executed sequentially, and the incremental stream creation is determined to be complete when all the subtasks included in the incremental stream creation main task have been executed.

6. The method according to claim 5, characterized in that, The sequential execution of the various subtasks included in the main task of creating the baseline stream includes: Obtain the first priority order of each subtask included in the main task of creating the baseline stream; The subtasks included in the main task of creating the baseline stream are executed sequentially according to the first priority order.

7. The method according to claim 5, characterized in that, The sequential execution of the subtasks included in the main task of incremental stream creation includes: Obtain the second priority order of each subtask included in the main task of creating the incremental stream; The subtasks included in the incremental stream creation main task are executed sequentially according to the second priority order.

8. The method according to any one of claims 1 to 7, characterized in that, The method further includes: In response to a display command for a data stream, the data stream relationship diagram of the data stream is invoked; the data stream relationship diagram is used to provide a graphical representation of the input, intermediate services, output, and objects used in the data stream. The data flow relationship diagram is shown.

9. A data stream processing device, characterized in that, The data flow management platform set up includes: The acquisition unit is configured to respond to a data stream processing instruction by acquiring the business requirement information corresponding to the data stream processing instruction and creating a task information table for the corresponding data stream to be processed. The configuration unit is configured to execute the task information table of the data stream to be processed based on the business requirement information; The verification unit is configured to perform a first verification process to obtain the task information table in the test stream, and if the first verification process result is passed, to obtain a second verification process result to obtain the task information table in the grayscale stream. The publishing unit is configured to publish the task information table to the formal flow if the second verification process passes. The data flow management platform classifies data flows according to business type and adds them to corresponding data flow groups for group maintenance. The data flows include base flows and incremental flows. The data flow group is a collection of base flows and incremental flows. The identifier of the data flow group exists in the base flow table and the incremental flow table as a foreign key. Each base flow or incremental flow is bound to a data flow group. The data flow management platform is also used to visualize the configuration information of data flows through a data flow relationship diagram.

10. An electronic device, characterized in that, include: processor; Memory used to store the processor's executable instructions; The processor is configured to execute the instructions to implement the data stream processing method as described in any one of claims 1 to 8.

11. A computer-readable storage medium, characterized in that, When the instructions in the computer-readable storage medium are executed by the processor of the electronic device, the electronic device is able to perform the data stream processing method as described in any one of claims 1 to 8.

Images