Data processing method, apparatus and device, and computer medium

Abstract

The invention discloses a data processing method and device, equipment and a computer medium, and the method comprises the steps: obtaining a data processing type which is data collection or data development; determining at least one target component corresponding to the data processing type from a plurality of preset components based on the data processing type; and processing the to-be-processed data of which the type is the data processing type based on the at least one target component, so that the effect of improving the efficiency of data acquisition and data development can be achieved.

Description

Technical Field

[0001] The present application belongs to the field of data processing technology, and in particular, relates to a data processing method, device, equipment, and computer medium. Background Art

[0002] With the advent of the big data era, data collection and data development have become key steps for enterprises to gain insights and build competitive advantages. However, traditional data collection and data development methods face multiple challenges: on the one hand, they are often highly dependent on complex programming and cumbersome configuration processes, which not only greatly increases development costs, but also significantly prolongs project cycles. On the other hand, existing data collection and data development tools generally lack sufficient flexibility and scalability, making it difficult to quickly adapt to the frequent changes in business needs and the ever-changing technical environment.

[0003] In terms of data collection, companies usually use customized scripts or dedicated software to extract data from a variety of data sources. However, a significant drawback of this approach is that different codes often need to be written for different data sources, which not only makes code maintenance extremely difficult, but also greatly reduces the reusability of the code. In the field of data development, data engineers usually need to invest a lot of time to manually write data conversion logic, which is not only time-consuming and labor-intensive, but also prone to errors.

[0004] In the related technologies, the methods of data collection and data development are relatively complicated, and the efficiency of data collection and data development is relatively low. Summary of the invention

[0005] The embodiment of the present application provides an implementation scheme different from the prior art to solve the technical problems in the related art that the data collection and data development methods are relatively complicated and the efficiency of data collection and data development is relatively low.

[0006] In a first aspect, the present application provides a data processing method, comprising:

[0007] Acquire a data processing type, where the data processing type is data collection or data development;

[0008] Based on the data processing type, determining at least one target component corresponding to the data processing type from a plurality of preset components;

[0009] Based on the at least one target component processing type, data to be processed is of the data processing type.

[0010] In a second aspect, the present application provides a data processing device, including:

[0011] An acquisition unit, used for acquiring a data processing type, wherein the data processing type is data acquisition or data development;

[0012] a determining unit, configured to determine, based on the data processing type, at least one target component corresponding to the data processing type from a plurality of preset components;

[0013] A processing unit is used for processing data to be processed of the data processing type based on the at least one target component.

[0014] In a third aspect, the present application provides an electronic device, including:

[0015] A processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to execute any method in the first aspect or any possible implementation manner of the first aspect by executing the executable instructions.

[0016] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the computer program implements any method in the first aspect or any possible implementation manner of the first aspect.

[0017] In a fifth aspect, an embodiment of the present application provides a computer program product, including a computer program, which, when executed by a processor, implements the method described in the first aspect or any possible implementation manner of the first aspect.

[0018] The present application provides for obtaining a data processing type, wherein the data processing type is data acquisition or data development; based on the data processing type, at least one target component corresponding to the data processing type is determined from a plurality of preset components; based on a scheme in which the at least one target component processes data to be processed of the data processing type, a corresponding preset component can be selected based on the data processing type, without the need to write corresponding code for a specific data processing type, thereby improving the efficiency of data acquisition and data development. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following is a brief introduction to the drawings required for use in the embodiments or the prior art descriptions. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work. In the drawings:

[0020] Figure 1a A flowchart of a data processing method provided in one embodiment of the present application;

[0021] Figure 1b A schematic diagram of the flow of each operator in the operator provided in an embodiment of the present application when performing data processing;

[0022] Figure 1c A schematic diagram of the structure of a data processing framework provided in one embodiment of the present application;

[0023] Figure 2 A schematic diagram of the structure of a data processing device provided in one embodiment of the present application;

[0024] Figure 3 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0025] The embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present application, but cannot be understood as limiting the present application.

[0026] The terms "first" and "second" etc. in the specification, claims and drawings of the embodiments of the present application are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein, for example. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, the process, method, system, product or equipment comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or equipment.

[0027] First, some terms in the embodiments of the present application are explained below to facilitate understanding by those skilled in the art.

[0028] YAML (YAML Ain't Markup Language) is a human-readable, data serialization format that uses indentation and concise markup syntax to represent data structures, making documents easy to understand and edit.

[0029] API, short for Application Programming Interface, is a set of definitions and protocols that provide access to web tools or databases. These tools and databases interact with various software applications through APIs without having to understand all the details of what's going on "behind the scenes". In simple terms, API is like a bridge between software, enabling different programs to communicate with each other and share data.

[0030] CDC, short for Change Data Capture, is a technology used to capture and record data changes in a database. It focuses on tracking and recording data changes within a specific time period, which can include operations such as inserting, updating, and deleting data. CDC technology is widely used in data integration, data warehouse, real-time analysis, and auditing.

[0031] SPI, the full name of which is Service Provider Interface, is an interface used to provide an extension mechanism for a service framework. Unlike an API (application programming interface), SPI allows a service framework to dynamically discover and load service providers at runtime, thereby achieving decoupling and high scalability. The core idea of ​​SPI is the loose coupling between service providers and service frameworks. Through configuration files or other mechanisms, the service framework can dynamically determine which service provider to use at runtime.

[0032] Data development refers to a series of processes to analyze, design, implement, deploy and maintain data solutions, aiming to maximize the value of enterprise data resources. Data development is a subset of project activities in the System Development Lifecycle (SDLC), focusing on the definition of data requirements, the design and implementation of data solution components. These components include databases, data structures, information products (such as screen displays and reports), and data access interfaces.

[0033] With the advent of the big data era, data collection and data development have become key steps for enterprises to gain insights and build competitive advantages. However, traditional data collection and data development methods face multiple challenges: on the one hand, they are often highly dependent on complex programming and cumbersome configuration processes, which not only greatly increases development costs, but also significantly prolongs project cycles. On the other hand, existing data collection and data development tools generally lack sufficient flexibility and scalability, making it difficult to quickly adapt to the frequent changes in business needs and the ever-changing technical environment.

[0034] In terms of data collection, companies usually use customized scripts or dedicated software to extract data from diverse data sources. However, a significant drawback of this approach is that different codes often need to be written for different data sources, which not only makes code maintenance extremely difficult, but also greatly reduces the reusability of the code. In the field of data development, data engineers usually need to invest a lot of time to manually write data conversion logic, which is not only time-consuming and labor-intensive, but also prone to errors.

[0035] In the related technologies, the methods of data collection and data development are relatively complicated, and the efficiency of data collection and data development is relatively low.

[0036] In order to meet the above challenges, the industry has explored and launched some big data processing frameworks and tools, which provide abstraction layers and automation support to a certain extent. However, these existing frameworks still have obvious limitations: most of them are optimized for a single data processing scenario (such as batch processing or real-time stream processing) and fail to provide a unified processing paradigm that can support multiple scenarios at the same time. In addition, they lack the ability to adapt to multi-engine data processing environments. More importantly, existing solutions have obvious deficiencies in ease of use and visual operation, which seriously hinders non-technical users from participating in the process of data collection and data development.

[0037] The technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems are described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.

[0038] Figure 1a A flowchart of a data processing method provided for an exemplary embodiment of the present application, wherein the execution subject of the method may be any computer device, wherein the computer device may be a terminal or a server and other devices. The terminal may be a smart phone, a tablet computer, a laptop computer, an intelligent voice interaction device, a smart home appliance, a wearable smart device, an aircraft, an intelligent vehicle terminal and other devices, and the terminal may also include a client, which may be a video client, a browser client or an instant messaging client, etc. The server may be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms. The method may include the following steps S101-S103:

[0039] S101, obtaining a data processing type, where the data processing type is data collection or data development;

[0040] In some optional embodiments of the present application, the present application may be implemented by relying on a visual operation page. Accordingly, the method further comprises: in response to a user selecting a data processing type on the visual operation page, determining the data processing type.

[0041] Optionally, the user may select a data processing type by selecting a button corresponding to the data processing type.

[0042] Optionally, the user may also select a data processing type by dragging a draggable control corresponding to the data processing type into the area used to determine the data processing type.

[0043] S102: Based on the data processing type, determine at least one target component corresponding to the data processing type from a plurality of preset components;

[0044] Specifically, the aforementioned multiple preset components include components used in data collection scenarios and components used in data open scenarios. Among the multiple preset components, there are components used in both data collection scenarios and data development scenarios.

[0045] The at least one target component corresponding to the data processing type refers to: among the multiple preset components, at least one preset component used when processing data to be processed of the data processing type.

[0046] When the data processing type is data acquisition, the at least one target component refers to: among the multiple preset components, at least one preset component used when processing the to-be-processed data whose processing type is data acquisition.

[0047] When the data processing type is data development, the at least one target component refers to: at least one preset component among multiple preset components used when processing data to be processed whose processing type is data development.

[0048] S103: Processing data to be processed of the data processing type based on the at least one target component.

[0049] When the data processing type is data collection, the data to be processed is the data to be collected, and processing the data to be processed refers to collecting the data to be processed.

[0050] When the data processing type is data development, the data to be processed is data to be developed, and processing the data to be processed refers to developing the data to be processed.

[0051] In some optional embodiments of the present application, in the aforementioned step S103, based on the at least one target component processing type of the to-be-processed data being the data processing type, the following steps S1031-S1033 are included:

[0052] S1031, determining a target engine for processing the to-be-processed data of the data processing type based on the pipeline component in the at least one target component;

[0053] Optionally, the engine in the pipeline component may be any one of a Flink engine, a Spark engine, and a Flume engine.

[0054] The pipeline component specifically includes parameters for processing data, including but not limited to: the number of threads running simultaneously, the checkpoint interval, and the engine type.

[0055] Determining a target engine for processing the data to be processed of the data processing type based on the pipeline component in the at least one target component includes: using the engine category in the pipeline component in the at least one target component as the target engine for processing the data to be processed of the data processing type.

[0056] S1032, constructing an operation operator corresponding to the data processing type;

[0057] When the data processing type is data collection, the operation operators corresponding to the data processing type include the operation operators at the collection source end and the operation operators at the target end. The operation operators at the collection source end and the operation operators at the target end play a vital role. These operators are responsible for reading, converting and outputting data.

[0058] On the source side, the operator is mainly responsible for reading data from various data sources. These data sources can be relational databases, HDFS / OBS (Hadoop Distributed File System / Object Storage Service), Kafka, etc. On the target side, the operator is mainly responsible for outputting the processed data to the specified target location. These targets can be databases, file systems, message queues, etc.

[0059] When the data processing type is data development, the operation operators corresponding to the data processing type include the operation operators at the acquisition source end, the operation operators at the target end, and the operator operator. Among them, the operator operator may include the replace operator and the Left-join operator. Optionally, in the data development scenario, the data processing flow of each operator in the operator operator when performing data processing can be found in Figure 1b shown. Figure 1b In the operator, the replace plug-in operator, projection plug-in operator, filter plug-in operator, Left-join plug-in operator, and sink plug-in operator all belong to operator operators. Figure 1b The ssd_catalog.dk.ods_table and ssd_catalog.dk.dim_table in the table can be the tables to be processed in the data source; table1-table4 are the intermediate processing results; ssd_catalog.doris_sink.sink_table is the table to be stored in the database after the data processing is completed.

[0060] The replace operator is mainly used for string replacement operations. In the process of data processing, it is often necessary to replace specific strings in certain fields to meet the needs of data cleaning, data conversion or data formatting. The replace operator can efficiently implement this function. It accepts a source string and one or more substrings to be replaced and the corresponding replacement string, and then returns a new string in which all specified substrings are replaced with the corresponding string.

[0061] The left-join operator is mainly used for data association operations. In data development, it is often necessary to associate data from different data sources or different tables for further analysis or processing. The left-join operator can implement a left join operation, that is, return all records in the left table and the records in the right table that meet the join conditions. If there are no records in the right table that meet the conditions, the right table part of the result will contain null values ​​(NULL).

[0062] S1033. Process the data to be processed based on the operation operator, the target engine, and the at least one target component.

[0063] In some optional embodiments of the present application, before processing the data to be processed based on the operation operator, the target engine, and the at least one target component, the method further includes: verifying whether the data source of the data to be processed is legal, and if so, determining to execute the step of processing the data to be processed based on the operation operator, the target engine, and the at least one target component.

[0064] Optionally, a source component may be obtained from at least one component, and a data source of the data to be processed may be obtained from the source component, wherein the data source may be an address of a current storage device of the data to be processed.

[0065] Verifying the legitimacy of the data source is a crucial step. The legitimacy verification of the data source includes: verifying the legitimacy of the data provider, verifying whether the data provider has legal business qualifications and data collection and processing rights, and ensuring that the data provider does not obtain data through illegal means, such as hacker attacks, data theft, etc.

[0066] Optionally, the validity verification of the data source further includes: verifying whether the data source complies with a preset format, and verifying whether the data in the data source is readable.

[0067] Optionally, when verifying whether the data in the data source is readable, it may be specifically verified whether the CDC is enabled. If enabled, it is determined that the data in the data source is readable.

[0068] In some optional embodiments of the present application, when the data processing type is data acquisition, the at least one target component includes: source, sink, route, pipeline; when the data processing type is data development, the at least one target component includes: catalog, source, sink, transform, pipeline.

[0069] Catalog: A catalog defines a data hierarchy for accessing and managing metadata about data sources. In data development scenarios, catalogs play a vital role because they provide descriptive information about data sources so that data developers can understand and use them.

[0070] Source: Source represents the source of data, that is, the starting point of data collection. In the data collection scenario, source is responsible for reading data from various data sources (such as databases, file systems, etc.) to provide input for subsequent processing steps. In the data development scenario, the role of the source component is equally important. It is not only the starting point of the data flow, but also a key link in data quality control. Before caching the data, the source component can perform preliminary preprocessing on the data, such as data cleaning and data filtering. This helps to reduce the complexity and workload of subsequent data processing and analysis. The source component can ensure the quality and accuracy of the data by configuring parameters and data verification mechanisms. For example, you can set constraints such as the format, type, and range of the data, and perform verification and filtering when receiving the data to avoid the influx of invalid or erroneous data. The source component can be integrated with a variety of data sources, such as databases, file systems, message queues, etc.

[0071] Sink: sink represents the target end of the data, that is, the destination for storage or transmission of data after it is collected.

[0072] In the data collection scenario, the sink is responsible for writing the collected data to a specified target location (such as another database, file system, etc.) or transferring it to other systems for further use. The main role of the sink is to obtain data from the Channel buffer and store it in the target storage system for subsequent data processing and analysis. Specifically, as the final receiver of the data, the sink is responsible for obtaining data from the Channel buffer and storing it in the local disk, memory, or remote storage system. Before storing the data, the sink can format the data to meet the needs of the target storage system. For example, convert the data into formats such as JSON, CSV, etc. for better data analysis and processing. The sink supports a reliable event transmission mechanism to ensure that the data is not lost or damaged during transmission. This is achieved through mechanisms such as transaction processing and data verification, thereby ensuring the integrity and accuracy of the data. In the data development scenario, the role of the sink is equally important. It is not only the end point of data storage, but also a key link in data flow and collaborative processing. Specifically, in some cases, the sink may need to forward the processed data to other components or systems to achieve data sharing and collaborative processing. This helps to build a more flexible and scalable data processing architecture and improve the efficiency and accuracy of data processing. The sink can also further process and transform the received data to meet the needs of subsequent business logic. For example, it can clean, filter, aggregate and other operations on the data to generate more valuable data sets. The sink can also integrate and merge with multiple data sources and storage systems to achieve unified management and access of data. This helps to build a more comprehensive and accurate data view, providing strong support for data analysis and business decision-making.

[0073] route: route represents the direct synchronization relationship in the data collection process.

[0074] In the data collection scenario, route defines the flow path and mode of data from source to sink, ensuring that data can be collected and transmitted in the expected manner. In the data collection process, route mainly plays a key role in data forwarding path selection and management. Route can select the best transmission path for data packets based on the routing table and network topology. This helps to ensure that data packets can be efficiently and accurately transmitted from the source to the target. In a complex network environment, there may be multiple paths to the target network. Route can evaluate the pros and cons of these paths and select the best path for data transmission based on actual conditions. The route command allows network administrators to view and modify routing tables to achieve precise control of network traffic and data transmission paths. This helps to optimize network performance and improve data transmission efficiency. In some cases, network administrators may need to manually configure static routes. With the route command, this requirement can be easily achieved to ensure that data for a specific network is transmitted through a predetermined gateway or interface. When a network failure occurs, the route command can help network administrators quickly locate the problem and take appropriate measures to recover. For example, by viewing the routing table, you can determine whether a data packet is lost or delayed due to a routing error. Route ensures that data packets are transmitted along a predetermined path, thus avoiding data loss or corruption. This helps ensure the integrity and accuracy of data collection. By optimizing routing selection and data transmission paths, route can improve the efficiency of data collection. This helps reduce delays and bottlenecks in the data collection process and improve overall system performance. In multi-source data collection scenarios, route ensures that data packets from different data sources can be correctly forwarded to the target storage system or processing component.

[0075] Transform: In data development scenarios, transform plays a vital role. It is mainly used to transform and process data sets to meet the needs of subsequent analysis, modeling or storage. Transform can identify and process missing values ​​in data sets, such as filling, deleting or interpolating to make up for missing data. Transform can identify, mark or replace outliers in data to ensure data accuracy and consistency. Through transform, data can be standardized or normalized so that data with different characteristics have the same scale or distribution, which is convenient for subsequent analysis and modeling. Transform supports converting one data type to another, such as converting a string to a numeric type, or converting a date and time type to a specific format. It can also convert data from one format to another, such as converting a CSV file to JSON format, or converting data in a database to a Pandas DataFrame. Transform can map data according to specific rules or functions, such as mapping the value of a field to another value, or performing complex mathematical operations. By setting conditions or rules, transform can filter data sets to filter out data subsets that meet specific requirements. Transform can extract meaningful features from raw data, such as extracting multi-scale features of signals through wavelet transform, or converting text into numerical features through methods such as TF-IDF. It can also perform feature selection based on the importance or relevance of features to optimize model performance and reduce computational complexity. Transform supports aggregation operations on data, such as calculating statistics such as average, sum, maximum, minimum, etc. It can also group data according to specific fields or rules for more detailed analysis and comparison In the field of machine learning or deep learning, transform can expand the scale of data sets and improve the generalization ability of models through data enhancement techniques. It can also transform data appropriately to make it more suitable for specific algorithms or models. For example, time series data can be processed through methods such as difference and logarithmic transformation to make it more suitable for predictive analysis.

[0076] Pipeline: Pipeline represents the environment parameters of the configured execution engine. It not only defines the overall process and logic of data processing, but also includes the environment parameters required to execute these processes, such as data processing type, execution engine type, engine version, and job execution parameters.

[0077] In some optional embodiments of the present application, in the aforementioned step S1033, the data to be processed is processed based on the operation operator, the target engine, and the at least one target component, including: based on the operation operator and the target engine, the data to be processed is processed using the configuration information of each target component in the at least one target component.

[0078] In some optional embodiments of the present application, the method further includes: preprocessing the configuration information of each target component in the at least one target component into a format recognizable by the target engine.

[0079] Specifically, the configuration information of each target component in the at least one target component is determined based on the data processing type selection operation performed by the user on the visualization operation page. The configuration information of each target component refers to the setting content in each target component.

[0080] Preprocessing the configuration information of each target component in the at least one target component into a format recognizable by the target engine refers to processing the configuration information of each target component in the at least one target component in the YAML format into a format recognizable by the target engine.

[0081] In some optional embodiments of the present application, see further Figure 1c As shown, Figure 1c A structural diagram of a data processing framework provided for an embodiment of the present application. The solution of the present application can be understood as providing a unified, efficient, easy-to-use, and extensible data processing framework. The framework defines a data pipeline (Data Pipeline, referred to as Pipeline) based on a user API in YAML format. Pipeline is a framework composed of multiple key components that work together to achieve data collection, conversion, and processing.

[0082] Specifically, the data processing framework may include a UI layer, an API access layer, a control layer, a logic layer, and an engine layer. Among them, the UI layer is used to provide a visual operation interface for displaying the aforementioned visual operation page, and the visual operation interface supports users to manually define or configure data processing types, as well as set the start and stop of the data processing process. The API access layer is used to determine at least one target component and target engine in the logic layer based on the data processing type selected by the user in the UI layer; the control layer is used to perform operations and management on the page for acquisition tasks or development tasks (for example, start, stop, etc.), as well as detection operations (for example, check the running status of the job, check job statistics, etc.); the logic layer is used to provide the aforementioned multiple preset components; the engine layer is used to provide multiple optional engines.

[0083] This application significantly improves the efficiency and flexibility of data collection and data development through innovative data pipeline definition and multi-engine support technology. The YAML-based API simplifies the configuration and management process of data collection and data development tasks, reduces dependence on professional technicians, and thus reduces overall operation and maintenance costs. Since this application supports multiple execution engines, including various batch and streaming processing engines, the system can be easily expanded to adapt to different data processing needs, improving the scalability of the system.

[0084] In addition, the decoupling design of the data pipeline definition and execution engine of this application improves the fault tolerance of the system and ensures the continuity and stability of data processing. The end-to-end data integration framework and easy-to-use API provided by this application allow users to implement complex data collection and data development tasks through simple configuration rather than programming, which facilitates docking to achieve low-code development and lowers the technical threshold. The data pipeline paradigm of this application can be flexibly applied to a variety of data processing scenarios, including but not limited to ETL jobs, data warehouse construction, real-time data analysis, etc., which is especially important for application scenarios that need to process batch data and real-time data streams.

[0085] The application provides a method for obtaining a data processing type, wherein the data processing type is data acquisition or data development; based on the data processing type, at least one target component corresponding to the data processing type is determined from multiple preset components; based on a scheme in which the at least one target component processes data to be processed of the data processing type, the corresponding preset component can be selected based on the data processing type, without the need to write the corresponding code for a specific data processing type, which can improve the efficiency of data acquisition and data development. The application provides users with an efficient, flexible, easy-to-use and cost-effective solution for data acquisition and data development paradigms, which is particularly suitable for modern enterprises to maintain competitiveness in a rapidly changing data environment.

[0086] Figure 2 A schematic diagram of the structure of a data processing device provided for an exemplary embodiment of the present application; wherein the device comprises:

[0087] An acquisition unit 21 is used to acquire a data processing type, where the data processing type is data acquisition or data development;

[0088] A determination unit 22, configured to determine, based on the data processing type, at least one target component corresponding to the data processing type from a plurality of preset components;

[0089] The processing unit 23 is configured to process the to-be-processed data of the data processing type based on the at least one target component.

[0090] In some optional embodiments of the present application, when the aforementioned apparatus is used to process the to-be-processed data of the data processing type based on the at least one target component, it is specifically used to:

[0091] Determine, based on the pipeline component in the at least one target component, a target engine for processing the to-be-processed data of the data processing type;

[0092] Constructing an operation operator corresponding to the data processing type;

[0093] The data to be processed is processed based on the operator, the target engine, and the at least one target component.

[0094] In some optional embodiments of the present application, the device is also used to: determine the data processing type in response to a user's data processing type selection operation on a visualization operation page.

[0095] In some optional embodiments of the present application, before processing the data to be processed based on the operation operator, the target engine, and the at least one target component, the apparatus is further configured to:

[0096] Verify whether the data source of the data to be processed is legal. If so, determine to execute the step of processing the data to be processed based on the operation operator, the target engine, and the at least one target component.

[0097] In some optional embodiments of the present application,

[0098] When the data processing type is data acquisition, the at least one target component includes: source, sink, route, pipeline;

[0099] When the data processing type is data development, the at least one target component includes: catalog, source, sink, transform, pipeline.

[0100] In some optional embodiments of the present application, when the aforementioned apparatus is used to process the data to be processed based on the operation operator, the target engine, and the at least one target component, it is specifically used to:

[0101] Based on the operation operator and the target engine, the data to be processed is processed using configuration information of each target component in the at least one target component.

[0102] In some optional embodiments of the present application, the device is also used for:

[0103] The configuration information of each target component in the at least one target component is preprocessed into a format recognizable by the target engine.

[0104] It should be understood that the device embodiment and the method embodiment may correspond to each other, and similar descriptions may refer to the method embodiment. To avoid repetition, no further description is given here. Specifically, the device may perform the above method embodiment, and the above and other operations and / or functions of each module in the device are the corresponding processes in each method in the above method embodiment, respectively, and no further description is given here for the sake of brevity.

[0105] The above describes the device of the embodiment of the present application from the perspective of the functional module in conjunction with the accompanying drawings. It should be understood that the functional module can be implemented in hardware form, can be implemented by instructions in software form, and can also be implemented by a combination of hardware and software modules. Specifically, the steps of the method embodiment in the embodiment of the present application can be completed by the hardware integrated logic circuit and / or software form instructions in the processor, and the steps of the method disclosed in the embodiment of the present application can be directly embodied as a hardware decoding processor to perform, or a combination of hardware and software modules in the decoding processor to perform. Optionally, the software module can be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, an electrically erasable programmable memory, a register, etc. The storage medium is located in a memory, and the processor reads the information in the memory, and completes the steps in the above method embodiment in conjunction with its hardware.

[0106] Figure 3 is a schematic block diagram of an electronic device provided in an embodiment of the present application, and the electronic device may include:

[0107] The memory 301 and the processor 302, the memory 301 is used to store the computer program and transmit the program code to the processor 302. In other words, the processor 302 can call and run the computer program from the memory 301 to implement the method in the embodiment of the present application.

[0108] For example, the processor 302 may be configured to execute the above method embodiments according to instructions in the computer program.

[0109] In some embodiments of the present application, the processor 302 may include but is not limited to:

[0110] General-purpose processor, digital signal processor (DSP), application-specific integrated circuit (ASIC), field programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic device, discrete hardware components, etc.

[0111] In some embodiments of the present application, the memory 301 includes but is not limited to:

[0112] Volatile memory and / or non-volatile memory. Among them, the non-volatile memory can be read-only memory (ROM), programmable ROM (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM) or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link DRAM (SLDRAM), and direct RAM bus random access memory (DR RAM).

[0113] In some embodiments of the present application, the computer program may be divided into one or more modules, which are stored in the memory 301 and executed by the processor 302 to complete the method provided by the present application. The one or more modules may be a series of computer program instruction segments capable of completing specific functions, and the instruction segments are used to describe the execution process of the computer program in the electronic device.

[0114] like Figure 3 As shown, the electronic device may also include:

[0115] The transceiver 303 may be connected to the processor 302 or the memory 301 .

[0116] The processor 302 may control the transceiver 303 to communicate with other devices, specifically, to send information or data to other devices, or to receive information or data sent by other devices. The transceiver 303 may include a transmitter and a receiver. The transceiver 303 may further include an antenna, and the number of antennas may be one or more.

[0117] It should be understood that the various components in the electronic device are connected via a bus system, wherein the bus system includes not only a data bus but also a power bus, a control bus and a status signal bus.

[0118] The present application also provides a computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a computer, the computer can perform the method of the above method embodiment. In other words, the present application embodiment also provides a computer program product containing instructions, and when the instructions are executed by a computer, the computer can perform the method of the above method embodiment.

[0119] When software is used for implementation, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function according to the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (digital subscriber line, DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integration. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a digital video disc (digital video disc, DVD)), or a semiconductor medium (e.g., a solid state drive (solid state disk, SSD)), etc.

[0120] Those of ordinary skill in the art will appreciate that the modules and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0121] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the module is only a logical function division. There may be other division methods in actual implementation, such as multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or modules, which can be electrical, mechanical or other forms.

[0122] The modules described as separate components may or may not be physically separated, and the components displayed as modules may or may not be physical modules, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. For example, each functional module in each embodiment of the present application may be integrated into a processing module, or each module may exist physically separately, or two or more modules may be integrated into one module.

[0123] The above are only specific implementations of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A data processing method, characterized in that: include: Acquire a data processing type, where the data processing type is data collection or data development; Based on the data processing type, determining at least one target component corresponding to the data processing type from a plurality of preset components; Based on the at least one target component processing type, data to be processed is of the data processing type.

2. The method according to claim 1, characterized in that The processing type of the at least one target component for the data to be processed is the data processing type, comprising: Determine, based on the pipeline component in the at least one target component, a target engine for processing the to-be-processed data of the data processing type; Constructing an operation operator corresponding to the data processing type; The data to be processed is processed based on the operator, the target engine, and the at least one target component.

3. The method according to claim 1, characterized in that The method further includes: determining the data processing type in response to a data processing type selection operation by a user on a visualization operation page.

4. The method according to claim 2, characterized in that: Before processing the data to be processed based on the operator, the target engine, and the at least one target component, the method further includes: Verify whether the data source of the data to be processed is legal, and if so, determine to execute the step of processing the data to be processed based on the operation operator, the target engine, and the at least one target component.

5. The method according to claim 1, characterized in that When the data processing type is data acquisition, the at least one target component includes: source, sink, route, pipeline; When the data processing type is data development, the at least one target component includes: catalog, source, sink, transform, pipeline.

6. The method according to claim 2, characterized in that Processing the data to be processed based on the operator, the target engine, and the at least one target component includes: Based on the operation operator and the target engine, the data to be processed is processed using configuration information of each target component in the at least one target component.

7. The method according to claim 6, characterized in that The method further comprises: The configuration information of each target component in the at least one target component is preprocessed into a format recognizable by the target engine.

8. A data processing device, characterized in that: include: An acquisition unit, used for acquiring a data processing type, wherein the data processing type is data acquisition or data development; a determining unit, configured to determine, based on the data processing type, at least one target component corresponding to the data processing type from a plurality of preset components; A processing unit is used for processing data to be processed of the data processing type based on the at least one target component.

9. An electronic device, characterized in that: include: processor; as well as A memory, configured to store executable instructions of the processor; The processor is configured to perform the method of any one of claims 1 to 7 by executing the executable instructions.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images