Operator-based data processing flow construction method and electronic device
By constructing a target operator directory and access scripts, the data processing mode is decoupled, the resource waiting problem in the batch processing method is solved, and the stability and efficiency of data processing are improved.
Patent Information
- Application Number
- CN202211342016.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-10-28
- Publication Date
- 2026-11-06
- Estimated Expiration
- 2042-10-28
AI Technical Summary
Existing batch processing-based data processing methods suffer from resource waiting issues, which affects the timeliness of data processing.
By acquiring the initial process and task operators in the first data processing mode of the target scenario, and constructing the target operator directory and access script in the second data processing mode, the initial process and the target process are decoupled and run independently, thereby optimizing data processing performance and timeliness.
It improved the stability and efficiency of data processing, optimized the performance and timeliness of data processing, and reduced labor costs.
Smart Images
Figure CN115712490B_ABST
Abstract
Description
Technical Field
[0001] This disclosure relates to the fields of data processing and high-precision map technology, and in particular to a method for constructing an operator-based data processing workflow and an electronic device. Background Technology
[0002] High-precision maps, also known as high-resolution maps, are used by autonomous vehicles. They possess precise vehicle location information and rich road element data, helping cars anticipate complex road conditions such as slope, curvature, and heading, thus better avoiding potential risks. Compared to ordinary maps, high-precision maps have a larger data scale, thus requiring higher data processing performance and efficiency. In practice, various data processing modes exist.
[0003] In related technologies, data can be processed in batch processing. However, batch processing involves resource waiting, which affects the timeliness of data processing to some extent. Summary of the Invention
[0004] This disclosure proposes an operator-based data processing workflow construction method and an electronic device.
[0005] According to a first aspect of this disclosure, a method for constructing a data processing flow based on operators is proposed. The method includes: obtaining an initial flow and task operators included in the initial flow under a first data processing mode in a target scenario; obtaining a second data processing mode and obtaining a target operator directory under the second data processing mode according to the task operators; constructing an operator access script for the target operator directory according to the task operators; and constructing a target flow under the second data processing mode in the target scenario based on the target operator directory and the operator access script for the target operator directory.
[0006] According to a second aspect of this disclosure, an operator-based data processing flow construction apparatus is proposed. The apparatus includes: an acquisition module, configured to acquire an initial flow and task operators included in the initial flow under a first data processing mode in a target scenario; an operator directory module, configured to acquire a second data processing mode and acquire a target operator directory under the second data processing mode according to the task operators; a first construction module, configured to construct an operator access script of the target operator directory according to the task operators; and a second construction module, configured to construct a target flow under the second data processing mode in the target scenario based on the target operator directory and the operator access script of the target operator directory.
[0007] According to a third aspect of this disclosure, an electronic device is proposed, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the operator-based data processing flow construction method described in the first aspect above.
[0008] According to a fourth aspect of this disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to cause the computer to execute the operator-based data processing flow construction method described in the first aspect above.
[0009] According to a fifth aspect of this disclosure, a computer program product is proposed, comprising a computer program that, when executed by a processor, implements the operator-based data processing flow construction method described in the first aspect above.
[0010] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description
[0011] The accompanying drawings are provided to better understand this solution and do not constitute a limitation of this disclosure. Wherein:
[0012] Figure 1 This is a flowchart illustrating an embodiment of the operator-based data processing flow construction method of this disclosure;
[0013] Figure 2 This is a flowchart illustrating another embodiment of the operator-based data processing flow construction method of this disclosure;
[0014] Figure 3 This is a flowchart illustrating another embodiment of the operator-based data processing flow construction method of this disclosure;
[0015] Figure 4 This is a flowchart illustrating another embodiment of the operator-based data processing flow construction method of this disclosure;
[0016] Figure 5 This is a schematic diagram of the structure of an operator-based data processing flow construction apparatus according to an embodiment of the present disclosure;
[0017] Figure 6 This is a schematic block diagram of an electronic device according to an embodiment of the present disclosure. Detailed Implementation
[0018] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.
[0019] Data processing is a fundamental aspect of systems engineering and automatic control. Data is a form of expression of facts, concepts, or instructions, which can be processed manually or by automated devices. After being interpreted and given meaning, data becomes information. Data processing involves the acquisition, storage, retrieval, processing, transformation, and transmission of data. The basic purpose of data processing is to extract and derive valuable and meaningful data from large amounts of potentially disorganized and difficult-to-understand data.
[0020] Figure 1 This is a flowchart illustrating an embodiment of the operator-based data processing flow construction method of this disclosure, as shown below. Figure 1 As shown, the method includes:
[0021] S101, Obtain the initial process and the task operators included in the initial process under the first data processing mode in the target scenario.
[0022] In this embodiment of the disclosure, a scenario with data processing requirements can be identified as a target scenario. In the target scenario, at least one data processing mode can exist, and any one of the data processing modes can be identified as the first data processing mode in the target scenario.
[0023] For example, if target scenario A includes two data processing modes, namely batch processing mode and stream processing mode, then batch processing mode can be determined as the first data processing mode in target scenario A.
[0024] Optionally, the data to be processed in the target scenario can be processed based on the first data processing mode. For example, based on the above example, it is known that the first data processing mode in target scenario A is the batch processing mode, so the data to be processed in target scenario A can be processed based on the batch processing mode.
[0025] During the data calculation and processing process, the first data processing mode has its corresponding data calculation and processing flow, which can be determined as the initial flow in the first data processing mode.
[0026] Furthermore, the operators used for data computation and processing included in the initial process can be identified as task operators in the initial process.
[0027] S102, obtain the second data processing mode, and obtain the target operator directory under the second data processing mode according to the task operator.
[0028] Optionally, other data processing modes that are not the first data processing mode among the data processing modes included in the target scenario can be determined as the second data processing mode in the target scenario.
[0029] For example, based on the above example, we know that target scenario A includes two data processing modes: batch processing mode and stream processing mode. Since batch processing mode is the first data processing mode in target scenario A, then stream processing mode can be identified as the second data processing mode in target scenario A.
[0030] In this embodiment of the disclosure, in scenarios where the first data processing mode and the second data processing mode use different methods to calculate and process the same data, the calculation results obtained by calculating and processing the same data based on the first data processing mode and the second data processing mode may be the same.
[0031] In this scenario, the task operators under the first data processing mode can be obtained, and based on the task operators under the first data processing mode, relevant information of the operators that perform data calculation and processing under the second data processing mode can be obtained.
[0032] In some implementations, operators in the second data processing mode can exist in the form of a directory. In this case, the operators in the second data processing mode that exist in the form of a directory can be defined as the target operator directory in the second data processing mode.
[0033] Optionally, the task operators in the first data processing mode can be divided based on a preset division method, and the target operator directory in the second data processing mode can be obtained based on the result of the division.
[0034] For example, based on the above example, we can obtain the task operators in the batch processing mode in the target scenario A, and divide the task operators in the batch processing mode according to the preset division method. Based on the result of the division, we can obtain the target operator directory in the stream processing mode.
[0035] S103, based on the target operator directory, construct the target process under the second data processing mode in the target scenario.
[0036] In this embodiment of the disclosure, the second data processing mode may have at least one target operator directory, wherein the operators in the target operator directory have a corresponding calculation and processing order when performing data calculation and processing.
[0037] For example, the data to be processed in the target scenario is set as data S, and data processing is performed on data S based on operator 1 in target operator directory 1 and operator 2 in target operator directory 2 included in the second data processing mode.
[0038] In this context, when operator 2 in target operator directory 2 processes data, its input data is the output data obtained by operator 1 in target operator directory 1 processing data S. Therefore, it can be determined that there is a processing order between target operator directory 1 and target operator directory 2 in the second data processing mode, and the processing corresponding to target operator directory 1 is earlier than the processing corresponding to target operator directory 2.
[0039] In this scenario, all target operator directories can be sorted based on the calculation and processing order among all target operator directories in the second data processing mode, and the data processing flow in the second data processing mode can be obtained based on the sorted results.
[0040] Specifically, the data processing flow obtained under the second data processing mode can be determined as the target flow for data processing based on the second data processing mode in the target scenario.
[0041] For example, based on the above example, we know that the second data processing mode in target scenario A is the stream processing mode. We can obtain all the target operator directories included in the stream processing mode, and sort all the target operator directories based on the calculation and processing order among all the target operator directories in the stream processing mode. Based on the sorted result, we can obtain the data processing flow in the stream processing mode, and determine this data processing flow as the target flow for data processing based on the stream processing mode constructed in target scenario A.
[0042] This disclosure proposes an operator-based data processing flow construction method. It obtains the initial flow and task operators included in the initial flow under a first data processing mode in a target scenario, and obtains a target operator directory under a second data processing mode based on the task operators. Based on the target operator directory, a target flow under the second data processing mode is constructed in the target scenario. In this disclosure, the target operator directory obtained from the task operators in the initial flow is used to construct the target flow under the second data processing mode in the target scenario. This allows the initial flow under the first data processing mode and the target flow under the second data processing mode to run independently in the target scenario, thereby decoupling them. This improves the stability of data processing between the initial flow under the first data processing mode and the target flow under the second data processing mode, optimizes the data processing performance of the target scenario, improves the data processing efficiency of the target scenario, and optimizes the timeliness of data processing in the target scenario.
[0043] In the above embodiments, the acquisition of the target operator directory and the target process can be combined with... Figure 2 To understand further, Figure 2 This is a flowchart illustrating another embodiment of the operator-based data processing flow construction method disclosed herein, as shown below. Figure 2 As shown, the method includes:
[0044] S201, Based on the task operators, construct the operator access script for the target operator directory.
[0045] In this embodiment of the disclosure, the metadata of the task operator can be obtained, and a target operator directory of the task operator in the second data processing mode can be generated based on the metadata.
[0046] Optionally, a method for generating the target operator directory under the second data processing mode can be obtained, wherein the initial directory structure of the target operator directory can be obtained, and the metadata of the obtained task operators can be integrated into the initial directory structure to obtain the target operator directory of the task operators under the second data processing mode.
[0047] In this embodiment of the disclosure, the target operator directory in the second data processing mode stores the metadata of the operators. In implementation, data processing in the second data processing mode needs to be implemented based on the operation of the operators.
[0048] In this process, it is necessary to configure the corresponding runtime environment for the operator to be run in order to enable the operator to run. In this scenario, the corresponding entry script can be configured for the target operator directory and the script can be identified as the operator access script of the target operator directory.
[0049] Optionally, the operator access script in the target operator directory can be run_gate.sh, or other scripts that can configure the corresponding runtime environment for the operator; no specific limitation is made here.
[0050] S202, based on the target operator directory and the operator access script of the target operator directory, construct the target process under the second data processing mode in the target scenario.
[0051] Optionally, the target operator directory and the corresponding operator access script can be integrated to obtain the target flow for data processing in the second data processing mode.
[0052] This includes the ability to obtain a preset target operator directory and an integration method for the operator access scripts corresponding to the target operator directory, and to integrate the target operator directory and the operator access scripts corresponding to the target operator directory based on the preset integration method.
[0053] This disclosure proposes a data processing flow construction method based on operators. It constructs operator access scripts for a target operator directory based on task operators, and then constructs a target flow under a second data processing mode within the target scenario based on the target operator directory and its access scripts. In this disclosure, the settings of the operator access scripts enable the target operator directory under the second data processing mode to configure the runtime environment for operators, achieving data processing based on the target flow. This saves labor costs, decouples the task operators in the initial flow from the target operator directory in the target flow, and allows the initial flow under the first data processing mode and the target flow under the second data processing mode to run independently within the target scenario. This improves the stability of data processing in both modes, optimizes the data processing performance of the target scenario, increases the data processing efficiency, and enhances the timeliness of data processing in the target scenario.
[0054] In the above embodiments, data processing can be performed in the target scenario based on the acquired target process, and can be combined with... Figure 3 To understand further, Figure 3 This is a flowchart illustrating another embodiment of the operator-based data processing flow construction method disclosed herein, as shown below. Figure 3 As shown, the method includes:
[0055] S301, obtain the candidate data to be processed in the target scenario, and obtain the execution operator directory for processing the candidate data from all target operator directories of the target process.
[0056] In some implementations, data processing can be performed based on the operators corresponding to some of the target operators included in the target process. In this case, the data that needs to be processed in the target scenario can be identified as the candidate data to be processed in the target scenario.
[0057] Optionally, the target operator directories corresponding to the operators required for the computational processing tasks of the candidate data can be obtained from all the target operator directories included in the target process, and these target operator directories can be determined as the execution operator directories for processing candidate data in the target process.
[0058] S302, obtain the execution operator metadata corresponding to the execution operator directory, and obtain the corresponding execution operator program package based on the execution operator metadata.
[0059] Optionally, the metadata of the operators stored in the execution operator directory can be read and identified as the execution operator metadata corresponding to the execution operator directory.
[0060] In implementation, the program package corresponding to the operator can be stored in a preset storage area. During data processing, the program package corresponding to the operator metadata can be obtained from the preset storage area by executing the operator metadata, and this part of the program package is identified as the execution operator program package corresponding to the operator metadata.
[0061] Specifically, the execution operator metadata, including information such as the operator's storage address, can be queried and searched in a preset storage area through the application program interface (API) to obtain the execution operator package corresponding to the execution operator metadata.
[0062] S303 obtains the dependency information of the execution operator package by executing the execution operator access script in the operator directory.
[0063] Optionally, the execution operator directory contains its corresponding operator access script, where the execution operator package can be configured with the corresponding runtime environment based on the operator access script corresponding to the execution operator directory.
[0064] In the implementation, operators have corresponding dependency information. The dependency information required to run the operator package can be obtained from the storage area of the operator's dependency information by executing the operator access script corresponding to the operator directory.
[0065] Optionally, the dependency information required for the execution of the operator package may include information related to the general logic required for the execution of the operator package and information related to the private logic required for the execution of the operator package. The dependency information required for the execution of the operator package can be obtained by executing the relevant scripts in the execution operator access scripts in the execution operator directory.
[0066] For example, if the execution operator access script in the execution operator directory includes the common logic script common_gate.sh, then the execution of this common logic script can be used to obtain and configure the relevant information of the common logic of the execution operator package corresponding to the execution operator directory.
[0067] S304, run the execution operator package based on dependency information to obtain the target data obtained by processing the candidate data based on the target process.
[0068] In this embodiment of the disclosure, the execution of the operator package requires a container, wherein the container for executing the operator package can be determined as a task container.
[0069] Optionally, candidate data may exist in multiple execution operator directories, wherein each execution operator directory may include at least one operator for performing computational processing on the candidate data.
[0070] In some implementations, the operators that perform the computation and processing of candidate data in the target process may be stored in different code libraries. In this scenario, the execution operator package can be embedded into the execution operator directory to generate the target image file.
[0071] This involves embedding multiple execution operator packages into their respective execution operator directories, packaging the embedded execution operator directories to generate the corresponding image file (docker), and then using the generated image file as the target image file.
[0072] Optionally, a task container for obtaining the target image file is acquired, and the target image file is run in the task container based on the dependency information to obtain the target data.
[0073] In this embodiment of the disclosure, a container that can provide a runtime environment for the execution operator package in the target image file can be selected from a pre-configured container that can provide a runtime environment for the operator, and used as the task container for the target image file.
[0074] After obtaining the dependency information of the execution operator package based on the operator access script corresponding to the execution operator directory, the execution operator package in the target image file can be run in the task container based on the dependency information, thereby realizing data processing of candidate data based on the second data processing mode, and obtaining the target data of candidate data based on the processing results obtained from the data processing.
[0075] It should be noted that while the target image file is running in the task container, there may be other image files running in the task container. In this scenario, the isolation mechanism of the task container can be obtained, and the execution operator package can be run in the task container based on the isolation mechanism.
[0076] Optionally, the task container has a preset isolation mechanism. Based on the isolation mechanism, the execution of the target image file running in the task container can be prevented from being interfered with by the execution of other image files. In this scenario, the preset isolation mechanism in the task container can be obtained, and the execution position of the target image file in the task container can be determined based on the preset isolation mechanism, thereby enabling the execution operator package in the target image file to run in the task container.
[0077] Specifically, before the execution operator package in the target image file enters the task container for execution, a blank running space can be set up in the task container. Based on the working path of the blank running space in the task container, the execution operator package in the target image file can be transferred to the blank running space, thereby enabling the execution operator package in the target image file to run in the task container based on the isolation mechanism.
[0078] It should be noted that the blank running space is set in the task container for the target image file. During the execution of the execution operator package in the target image file, the input data and process data of the execution operator package can be obtained.
[0079] The operator-based data processing flow construction method proposed in this disclosure obtains the execution operator directory from all target operator directories in the target flow based on the data to be processed in the target scenario. Then, based on the execution operator access script in the execution operator directory, it runs the corresponding execution operator program package to achieve data processing of candidate data and obtain the target data. In this disclosure, the setting of the operator access script allows the target operator directory in the second data processing mode to configure the runtime environment for operators, realizing data processing based on the target flow, saving labor costs, and decoupling the task operators in the initial flow from the target operator directory in the target flow. This allows the initial flow in the first data processing mode and the target flow in the second data processing mode to run independently in the target scenario, improving the stability of data processing in the initial flow in the first data processing mode and the target flow in the second data processing mode, optimizing the data processing performance of the target scenario, improving the data processing efficiency of the target scenario, and optimizing the timeliness of data processing in the target scenario.
[0080] During implementation, the target process may require additional target subdirectories, which can be addressed by combining... Figure 4 understand, Figure 4 This is a flowchart illustrating another embodiment of the operator-based data processing flow construction method disclosed herein, as shown below. Figure 4 As shown, the method includes:
[0081] S401, in response to the requirement to add a target operator directory in the target process, obtain the update operator directory corresponding to the addition requirement, and obtain the first operator type of the update operator package corresponding to the update operator directory.
[0082] In this embodiment of the disclosure, there may be a target operator directory that needs to be added in the target process. The target operator directory that needs to be added to the target process can be identified as the update operator directory.
[0083] Optionally, based on the data processing requirements of the target scenario, it can be determined whether there is a need to add a target operator directory to the target process.
[0084] For example, in the target scenario of map updates, the second data processing mode is set to stream processing mode. When it is found that there is data in the target scenario of map updates that cannot be processed by the target operator directory included in the current target process, it can be determined that there is a need to add a target operator directory in the target process of the current target scenario of map updates.
[0085] Specifically, based on the data processing requirements corresponding to the data that cannot be processed in the current target scenario, operators that can perform calculations on the data can be determined, and then the update operator directory corresponding to the target process can be obtained based on the operators.
[0086] To reduce the workload of adding update operator directories to the target process, the target operator directory that matches the update operator directory can be obtained from all the target operator directories included in the target process, and this target operator directory can be used as a reference to add the update operator directory to the target process.
[0087] Optionally, based on the relevant attribute information of the operator, a target operator directory matching the updated operator directory can be obtained from the entire target operator directory as a reference.
[0088] In this scenario, the operator type of the update operator package associated with the update operator directory can be obtained and determined as the first operator type.
[0089] S402, obtain the third operator type that is the same as the first operator type from the second operator type of the operator package corresponding to each of the target operator directories.
[0090] Optionally, operator type information of the operator packages of all target operator directories included in the target process can be obtained. In this case, the metadata of the operators included in the target operator directories can be read and analyzed to obtain relevant information about the operator types.
[0091] Furthermore, based on the obtained information about the selection type, the second operator type of the operator package corresponding to each target operator directory in all target operator directories is determined.
[0092] In this embodiment of the disclosure, the first operator type can be compared with the second operator type, wherein an operator type that is the same as the first operator type can be obtained from the second operator type and determined as the third operator type.
[0093] S403, retrieve the reference operator directory corresponding to the third operator type from all target operator directories.
[0094] In this embodiment of the disclosure, the target operator directory of the operator package with the operator type of the third operator type can be obtained from all the target operator directories, and it can be determined as the reference operator directory corresponding to the third operator type among all the target operator directories.
[0095] S404, based on the reference operator directory, add the updated operator directory to the target flow to obtain the new target flow.
[0096] In this embodiment of the disclosure, the operator access script corresponding to the reference operator directory can be used as a template for updating the operator directory. Specifically, the reference operator access script of the reference operator directory can be obtained and used as the update operator access script of the update operator directory.
[0097] Optionally, the operator access script corresponding to the reference operator directory can be determined as the reference operator access script of the reference operator directory.
[0098] In this scenario, all script information included in the reference operator access script can be obtained, and the script information that matches the update operator directory can be selected as the operator access script corresponding to the update operator directory, and it can be marked as the update operator access script of the update operator directory.
[0099] In this case, all the script information included in the reference operator access script can be used as the script information in the update operator access script, or a portion of the script information in the reference operator access script can be selected and the update operator script of the update operator directory can be constructed based on that portion of the script information. No specific limitation is made here.
[0100] Optionally, the update location of the update operator directory in the target process is determined, and the update operator access script and the update operator directory are added to the target process according to the update location to obtain a new target process.
[0101] In this embodiment of the disclosure, each of the target operator directories included in the target process has its corresponding location. During the process of adding an update operator directory, a blank location can be obtained from the storage location provided for the target operator directory in the current target process as the update location after the update operator directory is added to the target process.
[0102] It should be noted that any blank space can be selected as the update location for the update operator directory in the storage location provided by the target process for the target operator directory, or a blank space that conforms to the preset sorting rules in the remaining blank space can be determined as the update location for the update operator directory based on the preset sorting rules in the target process. No specific limitation is made here.
[0103] Optionally, based on a determined update location, the update operator directory and the update operator access script can be added to that update location to update the target process and obtain a new target process.
[0104] The operator-based data processing workflow construction method proposed in this disclosure responds to the need to add target operator directories to the target workflow. It obtains the corresponding update operator directory, determines a reference operator directory for the update operator directory from the target operator directories included in the target workflow, and obtains the update operator access script for the update operator directory based on the reference operator access script of the reference operator directory. Then, it adds the update operator directory and update operator script to the target workflow to obtain a new target workflow. In this disclosure, the reference operator directory included in the target workflow is used as a reference template to realize the addition of update operator directories to the target workflow. This reduces the workload of adding update operator directories to the target workflow, simplifies the operation steps of adding update operator directories to the target workflow, improves the efficiency of adding update operator directories to the target workflow, enhances the data processing capabilities of the target scenario, and optimizes the data processing performance of the target scenario.
[0105] Corresponding to the operator-based data processing flow construction methods proposed in the above embodiments, an embodiment of this disclosure also proposes an operator-based data processing flow construction device. Since the operator-based data processing flow construction device proposed in this disclosure corresponds to the operator-based data processing flow construction methods proposed in the above embodiments, the implementation methods of the above operator-based data processing flow construction methods are also applicable to the operator-based data processing flow construction device proposed in this disclosure, and will not be described in detail in the following embodiments.
[0106] Figure 5 This is a schematic diagram of the structure of an operator-based data processing flow construction apparatus according to an embodiment of the present disclosure, as shown below. Figure 5 As shown, the operator-based data processing flow construction device 500 includes an acquisition module 51, an operator directory module 52, a first construction module 53, and a second construction module 54, wherein:
[0107] The acquisition module 51 is used to acquire the initial process and the task operators included in the initial process in the first data processing mode of the target scenario.
[0108] The operator directory module 52 is used to obtain the second data processing mode and obtain the target operator directory under the second data processing mode according to the task operator;
[0109] The first construction module 53 is used to construct the operator access script of the target operator directory based on the task operator;
[0110] The second construction module 54 is used to construct the target process under the second data processing mode in the target scenario based on the target operator directory and the operator access script of the target operator directory.
[0111] In this embodiment of the disclosure, the operator directory module 52 is further configured to: obtain the metadata of the task operator, and generate a target operator directory of the task operator in the second data processing mode based on the metadata.
[0112] In this embodiment of the present disclosure, the apparatus further includes an operation module, configured to: acquire candidate data to be processed in the target scenario, and acquire the execution operator directory for processing the candidate data from all target operator directories of the target process; acquire the execution operator metadata corresponding to the execution operator directory, and acquire the corresponding execution operator package based on the execution operator metadata; acquire the dependency information of the execution operator package through the execution operator access script of the execution operator directory; and run the execution operator package based on the dependency information to acquire the target data obtained by processing the candidate data based on the target process.
[0113] In this embodiment of the disclosure, the running module is further configured to: embed the execution operator package into the execution operator directory to generate a target image file; obtain the task container of the target image file, and run the target image file in the task container based on the dependency information to obtain target data.
[0114] In this embodiment of the disclosure, the running module is further configured to: obtain the isolation mechanism of the task container, and run the execution operator package in the task container based on the isolation mechanism.
[0115] In this embodiment of the disclosure, the operator directory module 52 is further configured to: in response to a request to add a target operator directory in the target process, obtain the update operator directory corresponding to the request, and obtain the first operator type of the update operator package corresponding to the update operator directory; obtain a third operator type that is the same as the first operator type from the second operator types of the operator packages corresponding to all target operator directories; obtain a reference operator directory corresponding to the third operator type from all target operator directories; and add the update operator directory to the target process according to the reference operator directory to obtain a new target process.
[0116] In this embodiment of the disclosure, the operator directory module 52 is further configured to: obtain the reference operator access script of the reference operator directory, and use the reference operator access script as the update operator access script of the update operator directory; determine the update position of the update operator directory in the target process, and add the update operator access script and the update operator directory to the target process according to the update position to obtain a new target process.
[0117] The operator-based data processing flow construction apparatus disclosed herein acquires an initial flow and the task operators included in the initial flow under a first data processing mode in a target scenario, and acquires a target operator directory under a second data processing mode based on the task operators. Based on the target operator directory, a target flow under the second data processing mode is constructed in the target scenario. In this disclosure, the target operator directory obtained based on the task operators in the initial flow is used to construct the target flow under the second data processing mode in the target scenario, allowing the initial flow under the first data processing mode and the target flow under the second data processing mode to run independently in the target scenario. This decouples the initial flow under the first data processing mode and the target flow under the second data processing mode, improving the stability of data processing between them, optimizing the data processing performance of the target scenario, increasing the data processing efficiency of the target scenario, and improving the timeliness of data processing in the target scenario.
[0118] According to embodiments of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium, and a computer program product.
[0119] Figure 6 A schematic block diagram of an example electronic device 600 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.
[0120] like Figure 6 As shown, device 600 includes a computing unit 601, which can perform various appropriate actions and processes based on a computer program stored in read-only memory (ROM) 602 or a computer program loaded from storage unit 608 into random access memory (RAM) 603. RAM 603 may also store various programs and data required for the operation of device 600. The computing unit 601, ROM 602, and RAM 603 are interconnected via bus 604. Input / output (I / O) interface 605 is also connected to bus 604.
[0121] Multiple components in device 600 are connected to I / O interface 605, including: input unit 606, such as keyboard, mouse, etc.; output unit 606, such as various types of monitors, speakers, etc.; storage unit 608, such as disk, optical disk, etc.; and communication unit 609, such as network card, modem, wireless transceiver, etc. Communication unit 609 allows device 600 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.
[0122] The computing unit 601 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 601 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 601 performs the various methods and processes described above, such as the operator-based data processing flow construction method. For example, in some embodiments, the operator-based data processing flow construction method can be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 608. In some embodiments, part or all of the computer program can be loaded and / or installed on device 600 via ROM 602 and / or communication unit 609. When the computer program is loaded into RAM 603 and executed by the computing unit 601, one or more steps of the operator-based data processing flow construction method described above can be performed. Alternatively, in other embodiments, the computing unit 601 may be configured to perform an operator-based data processing flow construction method by any other suitable means (e.g., by means of firmware).
[0123] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.
[0124] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.
[0125] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.
[0126] To facilitate interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to facilitate interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).
[0127] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.
[0128] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, servers in distributed systems, or servers incorporating blockchain technology.
[0129] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.
[0130] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.
Claims
1. A method for constructing a data processing flow based on operators, wherein, The method includes: Obtain the initial process and the task operators included in the first data processing mode in the target scenario, wherein the first data processing mode is one of batch processing mode and stream processing mode. Obtain a second data processing mode, and obtain a target operator directory under the second data processing mode according to the task operator. The second data processing mode is another one of the batch processing mode and the stream processing mode. Based on the task operator, construct the operator access script for the target operator directory; Based on the target operator directory and the operator access script of the target operator directory, a target process under the second data processing mode is constructed in the target scenario; The method further includes: Obtain candidate data to be processed in the target scenario, and obtain the execution operator directory for processing the candidate data from all target operator directories of the target process; Obtain the execution operator metadata corresponding to the execution operator directory, and obtain the corresponding execution operator program package based on the execution operator metadata; The dependency information of the execution operator package is obtained through the execution operator access script in the execution operator directory; Based on the dependency information, the execution operator package is run to obtain the target data obtained by processing the candidate data based on the target process. The step of running the execution operator package based on the dependency information to obtain the target data obtained by processing the candidate data based on the target process includes: The execution operator package is embedded into the execution operator directory to generate a target image file; A task container is obtained from the target image file, and the target image file is run in the task container based on the dependency information to obtain the target data.
2. The method according to claim 1, wherein, The step of obtaining the second data processing mode and obtaining the target operator directory under the second data processing mode according to the task operator includes: Obtain the metadata of the task operator, and generate the target operator directory of the task operator in the second data processing mode based on the metadata.
3. The method according to claim 1, wherein, The method further includes: Obtain the isolation mechanism of the task container, and run the execution operator package in the task container based on the isolation mechanism.
4. The method according to any one of claims 1-2, wherein, The method further includes: In response to the requirement to add a target operator directory in the target process, the update operator directory corresponding to the added requirement is obtained, and the first operator type of the update operator package corresponding to the update operator directory is obtained; Obtain the third operator type that is the same as the first operator type from the second operator types of the operator packages corresponding to each of the target operator directories; Obtain the reference operator directory corresponding to the third operator type from all target operator directories; Based on the reference operator directory, the updated operator directory is added to the target process to obtain a new target process.
5. The method according to claim 4, wherein, The step of adding the updated operator directory to the target process according to the reference operator directory to obtain a new target process includes: Obtain the reference operator access script of the reference operator directory, and use the reference operator access script as the update operator access script of the update operator directory; Determine the update position of the update operator directory in the target process, and add the update operator access script and the update operator directory to the target process according to the update position to obtain a new target process.
6. An apparatus for constructing an operator-based data processing flow, wherein, The device includes: The acquisition module is used to acquire the initial process and the task operators included in the first data processing mode in the target scenario, wherein the first data processing mode is one of batch processing mode and stream processing mode. The operator directory module is used to obtain a second data processing mode and obtain a target operator directory under the second data processing mode according to the task operator. The second data processing mode is another one of the batch processing mode and the stream processing mode. The first construction module is used to construct the operator access script of the target operator directory based on the task operator; The second construction module is used to construct a target process under the second data processing mode in the target scenario based on the target operator directory and the operator access script of the target operator directory; The device further includes an operating module, used for: Obtain candidate data to be processed in the target scenario, and obtain the execution operator directory for processing the candidate data from all target operator directories of the target process; Obtain the execution operator metadata corresponding to the execution operator directory, and obtain the corresponding execution operator program package based on the execution operator metadata; The dependency information of the execution operator package is obtained through the execution operator access script in the execution operator directory; Based on the dependency information, the execution operator package is run to obtain the target data obtained by processing the candidate data based on the target process. The operating module is also used for: The execution operator package is embedded into the execution operator directory to generate a target image file; A task container is obtained from the target image file, and the target image file is run in the task container based on the dependency information to obtain the target data.
7. The apparatus according to claim 6, wherein, The operator directory module is also used for: Obtain the metadata of the task operator, and generate the target operator directory of the task operator in the second data processing mode based on the metadata.
8. The apparatus according to claim 6, wherein, The operating module is also used for: Obtain the isolation mechanism of the task container, and run the execution operator package in the task container based on the isolation mechanism.
9. The apparatus according to any one of claims 6-7, wherein, The operator directory module is also used for: In response to the requirement to add a target operator directory in the target process, the update operator directory corresponding to the added requirement is obtained, and the first operator type of the update operator package corresponding to the update operator directory is obtained; Obtain the third operator type that is the same as the first operator type from the second operator types of the operator packages corresponding to each of the target operator directories; Obtain the reference operator directory corresponding to the third operator type from all target operator directories; Based on the reference operator directory, the updated operator directory is added to the target process to obtain a new target process.
10. The apparatus according to claim 9, wherein, The operator directory module is also used for: Obtain the reference operator access script of the reference operator directory, and use the reference operator access script as the update operator access script of the update operator directory; Determine the update position of the update operator directory in the target process, and add the update operator access script and the update operator directory to the target process according to the update position to obtain a new target process.
11. An electronic device, comprising: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-5.
12. A non-transitory computer-readable storage medium storing computer instructions, wherein, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-5.
13. A computer program product comprising a computer program that, when executed by a processor, implements the method according to any one of claims 1-5.