Data processing method and computer equipment

Abstract

The invention discloses a data processing method and computer equipment, and relates to the technical field of data processing. The method comprises the following steps: based on a workflow arranged by a user, performing data processing on data in a data source to obtain to-be-processed data; based on the attribute parameters of the to-be-processed data, a data processing task is executed, the data processing task is used for indicating to process the to-be-processed data according to a stream processing mode or a batch processing mode corresponding to the attribute parameters, and the attribute parameters are used for describing the data category to which the to-be-processed data belong. According to the data processing method, the data from the data source is uniformly processed, and the data processing logic can be dynamically adjusted according to the attribute parameters of the to-be-processed data in the data processing process, so that the to-be-processed data can obtain the most suitable data processing mode, and the data processing mode is more efficient and flexible.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of data processing technology, and in particular to a data processing method and a computer device. Background Art

[0002] With the widespread application of big data technology, there is an increasing demand for rapid processing and intelligent analysis of massive data.

[0003] Related technologies accumulate data over a certain period of time and then centrally process it, a method also known as "batch processing." While this method leverages the advantages of batch processing, reducing processing times and costs, it cannot meet user needs for streaming data analysis.

[0004] Therefore, how to meet users' needs for data processing efficiency and real-time performance during the data processing process is an important issue that needs to be solved urgently. Summary of the Invention

[0005] The embodiments of the present application provide a data processing method and a computer device, which improve data processing efficiency.

[0006] To achieve the above objectives, the present invention adopts the following technical solutions:

[0007] In a first aspect, a data processing method is provided. The method includes: processing data in a data source based on a user-orchestrated workflow to obtain data to be processed. Based on attribute parameters of the data to be processed, a data processing task is executed, where the data processing task indicates whether to process the data to be processed in a stream processing mode or a batch processing mode corresponding to the attribute parameters. The attribute parameters describe the data category to which the data to be processed belongs.

[0008] The solution provided in this application uniformly processes the data in the data source, and can dynamically adjust the data processing logic according to the attribute parameters of the data to be processed during the data processing process, so that the data to be processed can obtain the most suitable data processing method, making the data processing method more efficient and flexible.

[0009] One possible implementation method is to process the data in the data source based on the workflow orchestrated by the user to obtain the data to be processed. It can be specifically implemented as follows: data is collected from the data source through the data acquisition processor in the workflow. The collected data is preprocessed by the data preprocessor in the workflow, and the data preprocessing includes data cleaning and / or format conversion. The preprocessed data is sent to the message queue or storage through the data push processor in the workflow to obtain the data to be processed. Users can customize workflows according to their specific needs instead of relying on preset or general workflows. The flexibility of workflow orchestration allows users to precisely define each step to adapt to specific business logic or data analysis needs.

[0010] Another possible implementation involves processing the data based on its attribute parameters, using the corresponding stream processing mode. This can be achieved by reading the data from a message queue or storage device and extracting its attribute parameters. If the attribute parameters meet the stream processing conditions, the data is stream processed according to the stream processing mode. Stream processing enables immediate response to data, which is crucial for applications requiring rapid decision support, such as financial market transactions and emergency response. Furthermore, stream processing of data allows for faster completion of data processing tasks, thereby improving overall data processing efficiency.

[0011] Another possible implementation involves processing the data based on its attribute parameters, using a batch processing mode corresponding to the attribute parameters. This can be achieved by reading the data from a message queue or memory and extracting its attribute parameters. If the attribute parameters do not meet the stream processing conditions or meet the batch processing conditions, the data is batch processed using the batch processing mode. Batch processing can be performed when the load is low, which allows for more efficient use of system resources and avoids excessive resource usage during peak hours. Furthermore, batch processing is more suitable for processing large amounts of data, as it can combine multiple operations into a single processing flow, simplifying data processing logic and reducing system complexity.

[0012] In another possible implementation, the attribute parameters include: at least one of: response time requirement, data volume, processing complexity score, or amount of system resources required for data processing; the stream processing conditions include at least one of: response time requirement is less than or equal to a response time threshold; data volume is less than or equal to a data volume threshold; processing complexity is less than or equal to a preset complexity; or, the amount of system resources required for data processing is less than or equal to a resource volume threshold.

[0013] In another possible implementation, the batch processing conditions include at least one of the following: the response time requirement is greater than the response time threshold; the data volume of the data to be processed is greater than the data volume threshold; the processing complexity of the data to be processed is greater than the preset complexity; or the amount of system resources required for data processing is greater than the resource volume threshold.

[0014] In another possible implementation, the data processing method provided by the present application further includes storing the results of the data processing on the data to be processed in a database corresponding to the data processing mode. By storing the results of the data processing in the respective corresponding databases, data consistency can be ensured and data conflicts can be avoided.

[0015] Another possible implementation involves storing the results of data processing on the data to be processed in a memory device corresponding to the data processing mode. This can be achieved by storing the stream processing data analysis results obtained by the stream processing mode in a relational database. A relational database is a database that stores data based on relational attributes. Relational databases provide structured data storage, ensuring that the results of real-time processing are neatly stored in a tabular format for easy management and query.

[0016] Another possible implementation method is to store the results of data processing on the data to be processed in a memory corresponding to the data processing mode. This can be specifically implemented as follows: the batch data analysis results obtained by processing in batch mode are stored in a distributed database or a non-relational database (not only structured query language, NoSQL). A distributed database refers to a database that stores data in a distributed form. Distributed databases and NoSQL databases are generally designed to be horizontally scalable, which means that when the amount of data increases, storage and processing capabilities can be increased by adding more server nodes. In addition, NoSQL databases generally have more relaxed requirements on data schemas and allow the storage of data with different structures, which is very useful for batch processing of diverse data sources.

[0017] Another possible implementation method is that the data processing method provided in the present application also includes: obtaining a data processing task; decomposing the data processing task to determine the subtasks in the data processing task, where the subtasks include: data collection, data preprocessing or data push; according to the processing order of each subtask in the data processing task, the processors corresponding to each subtask in the data processing task are arranged to obtain a workflow.

[0018] Another possible implementation method is to obtain the data processing task, which can be specifically implemented as follows: parsing the user's data processing requirements to obtain source data and data processing targets; obtaining the data processing task based on the source data and data processing targets, and the data processing task is to process the source data to obtain the data processing target.

[0019] In another possible implementation, the data processing method provided herein further includes: sending the results of the data processing to a user interface for display. The user interface can provide a visual display of the data, making the results easier to understand and analyze, and improving the accessibility of the information.

[0020] In another possible implementation, the results of the data processing can be queried, edited, or downloaded through a user interface or application programming interface (API). Results displayed in the user interface can be queried, edited, or downloaded through the user interface or API. Users can directly interact with the data, such as filtering, sorting, and searching, to find the information they need, improving the user experience.

[0021] In a second aspect, a data processing device is provided, comprising: an acquisition module and a data processing module.

[0022] The acquisition module is used to process the data in the data source based on the workflow arranged by the user to obtain the data to be processed.

[0023] The above-mentioned data processing module is used to execute data processing tasks based on the attribute parameters of the data to be processed. The data processing tasks are used to indicate whether to perform stream processing or batch processing on the data to be processed according to the stream processing mode or batch processing mode corresponding to the attribute parameters. The attribute parameters are used to describe the data category to which the data to be processed belongs.

[0024] The data processing device provided in the second aspect is used to execute the data processing method provided by the first aspect or any possible implementation of the first aspect. The technical effects corresponding to any one of the implementations in the second aspect can be referred to the technical effects corresponding to any one of the implementations in the first aspect, and will not be repeated here.

[0025] In a third aspect, a computer device is provided, comprising: a processor and a memory, wherein at least one computer program is stored in the memory, and the at least one computer program is loaded and executed by the processor to implement the data processing method described in the first aspect or any possible implementation of the first aspect.

[0026] In a fourth aspect, a computer-readable storage medium is provided, in which at least one computer program is stored. The at least one computer program is loaded and executed by a processor to implement the data processing method described in the first aspect or any one of the implementation methods of the first aspect.

[0027] In a fifth aspect, a computer program product is provided, which includes a computer program or instructions. When the computer program or instructions are executed by a processor, the data processing method described in the first aspect or any one of the implementation methods of the first aspect is implemented.

[0028] In the sixth aspect, an embodiment of the present application provides a chip system comprising at least one processor and at least one interface circuit, wherein the at least one interface circuit is used to perform transceiver functions and send instructions to the at least one processor. When the at least one processor executes the instructions, the at least one processor executes to implement the data processing method described in the first aspect or any one of the implementation methods in the first aspect.

[0029] The solutions provided in aspects 3 to 6 above are used to implement the methods provided in aspect 1 above or any one of the implementations of aspect 1 above, and their specific implementations are not described in detail here. The technical effects corresponding to any one of the implementations of aspects 3 to 6 above can be found in the technical effects corresponding to aspect 1 above or any one of the implementations of aspect 1 above, and are not described in detail here.

[0030] It should be noted that various possible implementations of any of the above aspects can be combined under the premise that the solutions are not contradictory. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 A schematic diagram of the architecture of a computer system provided in an embodiment of the present application;

[0032] Figure 2 A flowchart of a data processing method provided in an embodiment of the present application;

[0033] Figure 3 A schematic diagram of an orchestration workflow provided in an embodiment of the present application;

[0034] Figure 4 A flowchart of another data processing method provided in an embodiment of the present application;

[0035] Figure 5 A schematic diagram showing the results of processing in a stream processing mode according to an embodiment of the present application;

[0036] Figure 6 A schematic diagram showing the results of batch processing provided in an embodiment of the present application;

[0037] Figure 7 A schematic diagram of a data processing method provided in an embodiment of the present application;

[0038] Figure 8 A flowchart of a data processing method provided in an embodiment of the present application;

[0039] Figure 9 A schematic diagram of the structure of a data processing device provided in an embodiment of the present application;

[0040] Figure 10 A schematic diagram of the structure of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0041] In the embodiments of the present application, in order to clearly describe the technical solutions of the embodiments of the present application, words such as "first" and "second" are used to distinguish between identical or similar items with substantially the same functions and effects. Those skilled in the art will understand that words such as "first" and "second" do not limit the quantity or execution order, and words such as "first" and "second" do not necessarily mean different. There is no order of precedence or priority between the technical features described by "first" and "second".

[0042] In the embodiments of this application, words such as "exemplary" or "for example" are used to indicate examples, illustrations, or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of this application should not be construed as being preferred or advantageous over other embodiments or designs. Rather, the use of words such as "exemplary" or "for example" is intended to present the relevant concepts in a concrete manner to facilitate understanding.

[0043] In the embodiments of the present application, at least one can also be described as one or more, and multiple can be two, three, four or more, which is not limited in this application.

[0044] In addition, the network architecture and scenarios described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application, and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Ordinary technicians in this field can know that with the evolution of network architecture and the emergence of new business scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0045] Since the embodiments of the present application involve data processing technology, for ease of understanding, the following introduces the terminology-related concepts in the data processing technology involved in the embodiments of the present application.

[0046] Workflow: A sequence of processors arranged in sequence according to data processing tasks. For example, the workflow processing system (NiFi) includes a data acquisition processor, a data preprocessor, a data push processor, and a monitoring processor. The data acquisition processor is used to collect data, the data preprocessor is used to preprocess the collected data, the data push processor is used to distribute the preprocessed data, and the monitoring processor is used to monitor the processing status of each processor.

[0047] Data collection: refers to the process of collecting data from data sources. Data collection is the first step in data processing, and its accuracy and completeness are crucial to the subsequent processing results.

[0048] Visualization: This refers to presenting data processing results in the form of visual elements such as graphics, images, and animations, allowing users to more intuitively understand and analyze data. Visualization is intuitive, interactive, and dynamic, improving users' ability to understand and analyze data.

[0049] NiFi, a workflow processing system, is an open-source, easy-to-use, and powerful data stream processing system. It provides a visual interface that enables users to easily design, manage, and monitor data flow pipelines. Features: NiFi supports a variety of data sources and sinks and offers a rich set of processors and data flow operations.

[0050] Flink, a distributed processing framework, is an open-source stream and batch processing framework capable of handling both bounded and unbounded data streams. It provides a unified data processing model and supports high-throughput and low-latency data processing. Features: Flink supports event-time processing, state management, and window operations, enabling it to handle complex data processing tasks.

[0051] It should be noted that the information (including but not limited to device information, subject personal information, etc.), data (including but not limited to data used for analysis, stored data, displayed data, etc.), and signals involved in this application are all authorized by the subject or fully authorized by all parties, and the collection, use, and processing of relevant data must comply with the relevant laws, regulations, and standards of the relevant countries and regions. For example, the data to be processed and attribute parameters involved in this application are all obtained with full authorization.

[0052] An embodiment of the present application provides a data processing method, which is executed by a computer device. Specifically, the data processing method is executed by a workflow processing system NiFi and a distributed processing framework Flink in the computer device.

[0053] like Figure 1The computer system architecture shown in FIG. 1 includes a workflow processing system NiFi 102 and a distributed processing framework Flink 104. NiFi 102 and Flink 104 can be deployed in the same computer device or in different computer devices. The structural diagram of the computer device can be found in FIG. Figure 10 Introduced in [1]. NiFi 102, a workflow processing system in a computer system architecture, obtains data from a data source 101 and processes the data according to a workflow orchestrated by the user in NiFi 102 to obtain unprocessed data 103. Subsequently, Flink 104, a distributed processing framework, executes data processing tasks based on the attribute parameters of unprocessed data 103 to obtain data processing results 105.

[0054] The above-mentioned computer device can be a terminal device or a network device.

[0055] Among them, the terminal device can be an electronic device such as a mobile phone, a tablet computer, a vehicle-mounted terminal (vehicle computer), a wearable device, a personal computer (PC), a vehicle-mounted terminal, an aircraft, an unmanned vending terminal, etc.

[0056] The network device can be a server. The server can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers. It can also be a cloud server that provides cloud computing services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and other basic cloud computing services.

[0057] It should be understood that a server in the usual sense does not have a display screen. Therefore, when the computer device is specifically a server, the server needs to use an external display screen (such as a terminal device with a display screen) to display information, such as the construction process of a display workflow.

[0058] It should be noted that the solution provided in this application can be applied to at least one of the following: the financial industry, the Internet of Things, the e-commerce retail industry, smart manufacturing and industrial monitoring, or smart cities and public security, but is not limited thereto. The embodiments of this application do not specifically limit this. The following are respectively described:

[0059] 1) In the financial industry, the solution provided by this application can monitor transaction data in real time, provide risk warnings, detect abnormal transactions, and analyze historical transactions. This allows for rapid identification of abnormal behavior and prevention of risks.

[0060] For example, in the financial industry, data to be processed is collected from data sources. This data includes transaction system data (such as payment gateway data and ATM transaction data). If abnormal transactions (such as single transactions exceeding 100,000 yuan or high-frequency transactions within the same account) need to be detected, stream processing is performed on the transaction system data, and the results are displayed on the user interface. For example, current abnormal transactions can be displayed in real time on the user interface. If a daily customer transaction report is to be generated, batch processing is performed on the transaction system data. After a full day of transaction system data is processed uniformly, the results are displayed on the user interface, such as displaying a daily transaction report.

[0061] 2) In the field of Internet of Things, the solution provided by this application can be used for real-time processing of sensor data, equipment monitoring and fault warning. In scenarios such as smart homes and industrial monitoring, the working status of devices can be collected and analyzed in real time to detect potential problems in a timely manner.

[0062] For example, in the field of the Internet of Things, data to be processed is collected from data sources. The data to be processed includes: smart home detection data (such as temperature and humidity sensors, smart door locks, etc.) and industrial equipment detection data (such as controllers, vibration sensors). If it is necessary to perform abnormal detection or early warning on smart home or industrial equipment, the smart home detection data or industrial equipment detection data is stream processed, and the stream processing results are displayed in real time on the user interface. For example, the offline / online status diagram of the industrial equipment is displayed in real time on the user interface, or the temperature fluctuations of the smart home are displayed in real time on the user interface. If a weekly health report for the equipment is to be generated, the smart home detection data or industrial equipment detection data is batch processed. After a week of smart home detection data or industrial equipment detection data is uniformly processed, the results are displayed on the user interface. For example, the weekly failure rate of the industrial equipment is displayed on the user interface.

[0063] 3) In the e-commerce and retail sectors, the solution provided by this application can be used for real-time order processing, behavior analysis, inventory management, product popularity analysis, etc. It can also analyze real-time sales data, optimize inventory, and improve supply chain efficiency.

[0064] For example, in the e-commerce and retail sectors, data to be processed, such as order data, is collected and acquired. If product inventory needs to be monitored, stream processing is performed on the order data, and the results of the stream processing are displayed in real time on the user interface. For example, the order data or inventory balance are displayed in real time on the user interface. To generate a daily best-selling product list, batch processing is performed on the order data. After a day's order data is processed uniformly, the results are displayed on the user interface, for example, displaying the day's top-selling products.

[0065] 4) In smart manufacturing and industrial monitoring, the solution provided by this application can be used for real-time production line data collection and analysis, production scheduling optimization, and equipment health status monitoring. By monitoring equipment status through streaming data, downtime can be reduced and production efficiency can be improved.

[0066] For example, in intelligent manufacturing and industrial monitoring, data to be processed is collected from the data source. The data to be processed includes: industrial equipment detection data (such as production line operation data, equipment usage time detection data). If it is necessary to perform abnormal detection or early warning on the production line, the production line operation data is stream processed, and the stream processing results are displayed in real time on the user interface. For example, the offline / online status diagram of the industrial equipment is displayed in real time on the user interface. If the equipment health report is to be generated, the equipment usage time detection data is batch processed. After unified processing of the equipment usage time detection data for one month, the results are displayed on the user interface. For example, the equipment health report of the industrial equipment is displayed on the user interface.

[0067] 5) In smart cities and public safety, the solution provided by this application can be used for real-time traffic flow monitoring, environmental monitoring, safety warnings, etc. In urban management, real-time processing of data from various sensors ensures the safety and efficiency of urban operations.

[0068] For example, in smart cities and public safety, data to be processed is collected from data sources, including intersection camera video streams and vehicle positioning data. If the current crowd density needs to be identified, the intersection camera video stream is stream processed and the results are displayed in real time on the user interface, such as a crowd density heat map. For a single-day peak traffic map, the vehicle positioning data is batch processed. After uniformly processing the vehicle positioning data for a day, the results are displayed on the user interface, such as a single-day peak traffic map. Figure 2 A flowchart of a data processing method is provided for an embodiment of the present application, and the data processing method is executed by a computer device.

[0069] like Figure 2 As shown, the data processing method may include:

[0070] Step 200: The computer device obtains a data processing task.

[0071] Exemplarily, the process of constructing a data processing task includes: a computer device analyzes a data processing requirement input by a user to obtain source data and a data processing target, and the computer device obtains a data processing task based on the source data and the data processing target.

[0072] Source data refers to the data that the user wants to process, that is, part or all of the data in the data source. The data processing target refers to the data processing result that the user wants to obtain. The data processing task processes the source data to obtain the data processing target.

[0073] For example, in an e-commerce data analysis scenario, a user enters a data processing requirement: they want to predict the best-selling products for the next month by analyzing the e-commerce platform's historical sales data. The computer interprets the data processing requirement and obtains the source data: the e-commerce platform's historical sales data, including information such as product name, sales quantity, sales date, and price. The data processing goal is to predict the best-selling products for the next month. Therefore, the data processing task is to predict the best-selling products for the next month based on the historical sales data: product name, sales quantity, sales date, price, and other information.

[0074] Step 201: The computer device arranges a workflow based on the data processing task.

[0075] Exemplarily, the workflow orchestration process includes decomposing a data processing task and identifying subtasks within the data processing task. For example, the identified subtasks may include multiple components of data acquisition, data preprocessing, or data push. Based on the processing order of the subtasks within the data processing task, the processors corresponding to the subtasks are orchestrated to produce a workflow.

[0076] For example, a computer device receives a data processing task: predicting the hot-selling products for the next month based on historical sales data, including product name, sales quantity, sales date, and price. The data processing task is broken down into a data collection subtask, a data preprocessing subtask, and a data push subtask. The data collection processor corresponding to the data collection subtask, the data preprocessor corresponding to the data preprocessing subtask, and the data push processor corresponding to the data push subtask are then connected to form a workflow.

[0077] Optionally, data preprocessing includes at least one of: data format conversion, data cleaning (e.g., cleaning invalid or erroneous data), and data filtering (e.g., retaining only data under specific conditions), but is not limited to these. The embodiments of the present application do not specifically limit this.

[0078] Data format conversion refers to the process of converting data from one structured representation to another, for example, converting from JavaScript object notation (JSON) to comma-separated values ​​(CSV).

[0079] Step 202: The computer device processes the data in the data source based on the workflow arranged by the user to obtain data to be processed.

[0080] The data to be processed refers to data to be processed.

[0081] The data in the data source may be data from the same data source, data from different types of data sources, or data in different locations of the same data source, but is not limited thereto.

[0082] Optionally, different types of data sources include multiple items of databases, message queues, sensors, or logs, but are not limited thereto, and the embodiments of the present application do not make specific limitations on this.

[0083] In some embodiments, after obtaining data from a data source, the computer device may directly use the data obtained from the data source as data to be processed.

[0084] For example, a computer device acquires data from a data source through a workflow processing system (NiFi) within the computer device and processes the data according to a user-programmed workflow to obtain data to be processed. NiFi also provides a user interface that allows users to interactively program workflows, reducing the complexity of configuring and adjusting workflows. Furthermore, users can configure multiple workflows through NiFi's user interface based on business needs.

[0085] Optionally, NiFi includes a data acquisition processor, a data preprocessor, and a data push processor, but is not limited to this, and the embodiments of the present application do not specifically limit this. NiFi collects data from the data source through the data acquisition processor in the workflow; the data preprocessor in the workflow performs data preprocessing on the collected data, and the data preprocessing includes data cleaning and / or format conversion. Afterwards, the data push processor in the workflow sends the preprocessed data to a message queue (for example, a distributed stream processing system Kafka) or storage to obtain the data to be processed. NiFi is an easy-to-use workflow processing system. It provides a visual interface that enables users to easily design, manage, and monitor workflows for automatically collecting, routing, converting, and processing data between different data sources or systems.

[0086] For example, Figure 3The schematic diagram of the orchestration workflow shown in NiFi provides a visual interface, which displays a toolbar 301 and a workflow design area 302, but is not limited to this. The toolbar 301 displays components such as a data acquisition processor, a data preprocessor, a connection line, and a data push processor. By dragging components from the toolbar 301 to the workflow design area 302, each component represents a data processing operation, and the workflow is constructed through the components. For example, dragging the data acquisition processor to the workflow design area 302 allows the data acquisition processor to collect data from different types of data sources (such as local files and external devices). Drag the data preprocessor and the connection line to the workflow design area 302, connect the data acquisition processor and the data preprocessor through the connection line, and allow the data preprocessor to perform data cleaning and / or format conversion on the data collected by the data acquisition processor. Afterwards, drag the data push processor and the connection line to the workflow design area 302, connect the data push processor and the data preprocessor through the connection line, and allow the data push processor to send the data output by the data preprocessor to a message queue or storage to obtain data to be processed.

[0087] Step 203: The computer device executes the data processing task based on the attribute parameters of the data to be processed.

[0088] The attribute parameters are used to describe the data category to which the data to be processed belongs.

[0089] The data processing models in data processing tasks include: stream processing mode and batch processing mode.

[0090] Stream processing refers to a model that processes data immediately. Specifically, it processes data at a near-instantaneous rate, requiring a continuous inflow and outflow of data to maintain real-time insights. When data is received, it is processed immediately, enabling near-instantaneous decision-making. Stream processing offers high processing speeds and the ability to quickly respond to data changes, making it suitable for scenarios requiring immediate feedback.

[0091] Batch processing refers to a mode in which data is accumulated and then processed uniformly. In contrast to stream processing, batch processing collects and stores a certain amount of data, then processes it all at once. Batch processing is relatively slow, but it can handle large amounts of data and is suitable for scenarios where processing time is not critical.

[0092] Optionally, the attribute parameters include at least one of the following: the response time requirement for the data to be processed, the data volume to be processed, the processing complexity of the data to be processed, the amount of system resources required for data processing, or data processing model requirement instructions, but are not limited thereto and are not specifically limited in this embodiment of the present application. In principle, any attribute parameter that can characterize the data category of the data to be processed as stream data or batch data is acceptable.

[0093] In one possible implementation, the data to be processed is read from a message queue or a memory, and attribute parameters of the data to be processed are extracted. If the attribute parameters of the data to be processed meet the stream processing conditions, the data to be processed is stream processed according to the stream processing mode.

[0094] Stream processing conditions are used to measure whether the data to be processed is suitable for processing in stream processing mode.

[0095] Optionally, the stream processing condition includes at least one of the following, but is not limited to:

[0096] (1) The response time requirement must be less than or equal to the response time threshold. The response time requirement refers to the response time for data processing. For example, transaction data and user behavior data in games require immediate response. Stream processing mode can be used for data that requires real-time response. Batch processing mode can be used for data that does not require immediate data processing or can be accumulated for a certain period of time before processing.

[0097] (2) The amount of data to be processed is less than or equal to the data volume threshold. When the data volume is small, stream processing mode can be used for stream processing; when the data volume is large, the data to be processed needs to be accumulated for a certain period of time before processing, such as a day or a month of data, and batch processing mode can be used for batch processing.

[0098] (3) The processing complexity of the data to be processed is less than or equal to the preset complexity. The processing complexity of the data to be processed. If the data processing complexity is low, for example, it is just a simple data processing operation, the stream processing mode can be used for stream processing to quickly complete the data processing. If the data processing complexity is high, for example, large-scale data mining, machine learning model training, etc., the batch processing mode can be used for batch processing.

[0099] Alternatively, the processing complexity of the data to be processed can be assessed by the number and type of processing steps involved. For example, the number of processing steps required for the data to be processed can be determined. The greater the number of steps, the more complex the process, and the greater the potential for error, indicating a higher level of processing complexity. Another example is whether the processing steps involved involve complex algorithms, machine learning models, real-time analytics, etc. These steps are typically more complex than simple data transformations.

[0100] (4) The amount of system resources required for data processing is less than or equal to the resource threshold. Since the stream processing mode requires long-term use of system resources, if there are continuously available system resources, the stream processing mode can be used for stream processing. If there are continuously available system resources, the stream processing mode can be used for stream processing when system resources are idle.

[0101] Exemplarily, when the attribute parameters meet the stream processing conditions, the stream processing mode is selected to perform data processing on the data to be processed, thereby obtaining a stream data processing result.

[0102] For example, a computer device reads data to be processed from a message queue or memory using the distributed processing framework Flink. For example, to monitor e-commerce orders for anomalies, the data read by Flink is the real-time e-commerce order transaction stream, which includes, but is not limited to, order ID, user ID, amount, and timestamp. Flink's processing logic is as follows: Flink obtains the real-time e-commerce order transaction stream in real time and filters it, i.e., removing test orders with an amount of 0. Once the acquired real-time e-commerce order transaction stream is monitored, upon detecting an abnormal order (e.g., a single order amount exceeding 10,000 yuan), the risk control rules are triggered and an early warning is issued. Finally, the abnormal order results are displayed in real time on the user interface.

[0103] In another possible implementation, the data to be processed is read from a message queue or memory, and attribute parameters of the data to be processed are extracted. If the attribute parameters of the data to be processed do not meet the stream processing conditions or meet the batch processing conditions, the data to be processed is batch processed according to the batch processing mode.

[0104] The batch processing condition is used to measure whether the data to be processed is suitable for batch processing mode.

[0105] Optionally, the batch processing conditions include at least one of the following, but are not limited thereto:

[0106] (a) The response time requirement is greater than the response time threshold.

[0107] (b) The amount of data to be processed is greater than the data amount threshold.

[0108] (c) The processing complexity of the data to be processed is greater than the preset complexity.

[0109] (d) The amount of system resources required for data processing is greater than the resource threshold.

[0110] Exemplarily, the batch processing of the data to be processed in the batch processing mode includes: selecting the batch processing mode to periodically accumulate the data to be processed and then performing parallel processing to obtain batch data processing results.

[0111] Optionally, the periodic accumulation includes: time period accumulation or data quantity period accumulation, but is not limited thereto, and the embodiments of the present application do not make specific limitations on this.

[0112] Time period accumulation refers to the unified processing of the data to be processed for one day or one week.

[0113] The periodic accumulation of data quantity refers to the unified processing after accumulating n data, where n is a positive integer.

[0114] For example, computers use Flink to read data to be processed from a message queue or storage. For example, to generate a daily e-commerce sales report, Flink imports the previous day's order data from the message queue or storage at dawn each day. Flink's processing logic is as follows: Every dawn, Flink retrieves the previous day's order data and calculates the total sales and order volume for each product category. It then sorts the data to identify the top 10 selling products and records them in a report.

[0115] In some embodiments, the method of extracting attribute parameters of the data to be processed includes at least one of the following methods: extracting from the log file corresponding to the data to be processed, extracting by obtaining the processing requirements of the data to be processed input by the user, obtaining by evaluating the data volume of the data to be processed, or obtaining by evaluating the amount of system resources required for data processing, but is not limited to this, and the embodiments of the present application do not make specific limitations on this.

[0116] To sum up, the solution provided by this application, through unified processing of data from the data source, can dynamically adjust the data processing logic according to the attribute parameters of the data to be processed during the data processing process, so that the data to be processed can obtain the most suitable data processing method, making the data processing method more efficient and flexible.

[0117] Furthermore, after the data to be processed is processed according to the data processing mode corresponding to the attribute parameter, the result of the data processing on the data to be processed may be stored in a database corresponding to the data processing mode.

[0118] Figure 4 A flowchart of a data processing method is provided for an embodiment of the present application, and the model deployment method is executed by a computer device.

[0119] like Figure 4 As shown, the data processing method also includes:

[0120] Step 204: The computer device stores the stream processing data analysis results obtained by processing in accordance with the stream processing mode in a relational database.

[0121] A relational database is a database that stores data based on relational attributes. It provides the Structured Query Language (SQL), which allows for complex queries and supports transaction processing, ensuring data consistency and integrity.

[0122] Optionally, the relational database includes at least one of MySQL and PostgreSQL, but is not limited thereto, and the embodiments of the present application do not make specific limitations on this.

[0123] For example, relational databases are suitable for scenarios requiring high-concurrency queries, meaning many users or systems execute queries simultaneously, and these queries typically require fast responses. For example, consider a trading platform that processes trade orders in real time and stores the results in MySQL. This way, when a user queries their transaction history, the system can quickly retrieve the data from MySQL and display it to the user.

[0124] Step 205: The computer device stores the batch data analysis results obtained by processing in the batch processing mode in a distributed database or a non-relational database NoSQL.

[0125] A distributed database refers to a database that stores data in a distributed manner. By distributing data across multiple physical servers or storage nodes, it provides high availability, scalability, and fault tolerance.

[0126] NoSQL refers to a database that stores data based on non-relational attributes.

[0127] Optionally, the distributed database includes: Hadoop distributed file system (Hadoop distributed file system, HDFS), but is not limited thereto, and the embodiments of the present application do not make specific limitations on this.

[0128] NoSQL includes: at least one of a document-based NoSQL database and a distributed NoSQL database, but is not limited thereto, and the embodiments of the present application do not make specific limitations on this.

[0129] For example, NoSQL databases typically offer flexible data models, can easily handle unstructured or semi-structured data, and have good horizontal scalability. Distributed file systems can store batch data analysis results across multiple physical servers, providing high reliability and scalability. For example, a video streaming service runs a batch job every night to analyze user viewing habits and stores the results in Cassandra. These results can be used to optimize recommendation algorithms. Given the massive amount of data, Cassandra can efficiently process and store this data.

[0130] To sum up, by storing the results obtained from different data processing modes in different databases, we can maximize the advantages of different data and meet business needs.

[0131] Furthermore, after the data to be processed is processed, the result of the data processing may be displayed on the user interface.

[0132] like Figure 4 As shown, the data processing method also includes:

[0133] Step 206: The computer device sends the result of processing the data to be processed to the user interface for display.

[0134] For example, the data to be processed, the processing results corresponding to the data to be processed, or the analysis trends may be displayed in the user interface, and the user may view the data through the user interface.

[0135] like Figure 5 The following diagram shows the results of stream processing. Viewable data includes, but is not limited to, current online users, transactions per second, number of offline devices, and vehicle distribution. Stream processing results can be displayed in real time in the user interface. For example, when a user selects "View Current Online Users," the user interface displays changes to the current user. Furthermore, users can adjust the window refresh rate, such as switching from 5 seconds to 1 second.

[0136] like Figure 6 The result display diagram of the batch processing mode is shown in FIG. Figure 6 As shown in Figure (a), the data that can be viewed in the figure include: daily sales, monthly user retention rate and quarterly growth curve, but are not limited to these. For example, when the user chooses to view the monthly user retention rate, three static reports are displayed in the user interface, namely: user retention rate from January to April, user retention rate from May and user retention rate from June to September. Figure 6 As shown in Figure (b), when the user chooses to view the static report corresponding to the user retention rate from January to April, the changes in the user retention rate from January to April are displayed in the user interface.

[0137] In some embodiments, the results of data processing on the data to be processed can be queried, edited, or downloaded through the user interface or a programming interface (API). That is, users can query, edit, or download the results displayed in the user interface by interacting with the user interface. The results displayed in the user interface can also be queried, edited, or downloaded through the API. These features allow users to interact with data more effectively and make more informed decisions.

[0138] For example, users can enter the name of a city on the user interface to query real-time weather conditions. At the same time, API interfaces can also be used to allow developers to integrate weather query functions into their applications, so that weather information can be obtained through these applications.

[0139] In summary, the method provided in the embodiment of the present application uses NiFi and Flink as core components to provide an efficient, flexible, and scalable workflow processing method. Through Flink's stream processing capabilities, workflows can be quickly analyzed and fed back, and abnormal situations such as production line failures and fraudulent behavior in trading systems can be discovered in a timely manner. Real-time performance is greatly improved, ensuring immediate response to business. The combination of NiFi and Flink makes the management, processing, and storage of workflows more efficient, especially in a big data environment, where the system can process massive amounts of data and provide real-time analysis results.

[0140] The above embodiments provide a detailed introduction to the solutions provided by the present application. The following will provide a detailed introduction to the solutions provided by the present application with reference to specific examples.

[0141] like Figure 7 A schematic diagram of a data processing method is provided for an embodiment of the present application.

[0142] Computer devices collect data from data sources 701 (such as databases, message queues, sensors, logs, etc.) through a workflow processing system 702 (e.g., NiFi), and perform format conversion, data cleaning, data filtering, and so on according to the workflow. After NiFi processes the data to be processed and obtains the data to be processed, a distributed processing framework 703 (e.g., Flink) performs stream processing or batch processing on the data to be processed. For streaming data, Flink performs stream processing operations to quickly respond to real-time events. For batch data, Flink provides powerful batch processing capabilities, periodically analyzing and calculating historical data to generate regular reports and trend analysis.

[0143] Optionally, the data source 701 may be located inside or outside the computer device, which is not specifically limited in the embodiment of the present application.

[0144] After processing the data to be processed, the results of the data processing are stored in a database 704 corresponding to the data processing mode. For example, the analysis results of stream data obtained by processing in a stream processing mode are stored in a relational database. The analysis results of batch data obtained by processing in a batch processing mode are stored in a distributed database or a non-relational NoSQL database.

[0145] After processing the data to be processed, the results of the data processing may be sent to the user interface 705 for display. The user may query, edit, or download the results displayed in the user interface 705. The results may also be viewed through the backend, for example, through an API interface, to query, edit, or download the results displayed in the user interface.

[0146] like Figure 8 A flowchart of a data processing method is provided for an embodiment of the present application, and the data processing method is executed by a computer device.

[0147] Step 801: Start.

[0148] Step 802: The computer device builds a data processing environment.

[0149] The data processing environment includes: obtaining the source database corresponding to the data source, deploying the workflow processing system NiFi to perform data collection and / or data preprocessing operations on the data in the data source, deploying the distributed processing framework Flink to process the data to be processed, and determining the database for storing the data processing results.

[0150] For example, install the Java runtime environment, download and configure Flink and NiFi, install MySQL as the source database corresponding to the data source, and set up Hadoop HDFS and Hive as the database for storing data processing results.

[0151] Step 803: The computer device choreographs the workflow in the workflow processing system NiFi based on the data processing task.

[0152] For example, based on the data processing task, a workflow is created in NiFi, which includes acquisition, data preprocessing operations, and sending of data to be processed.

[0153] For example, NiFi's visual interface displays components such as a data acquisition processor, a data preprocessor, connection lines, and a data push processor. By dragging and dropping components, each component represents a data processing operation, and dragging and dropping components enables the construction of workflows. For example, dragging a data acquisition processor causes the data acquisition processor to collect data from a data source. Drag a data preprocessor and a connection line to connect the data acquisition processor and the data preprocessor through the connection line, so that the data preprocessor performs data cleaning and / or format conversion on the data collected by the data acquisition processor. Then, drag a data push processor and a connection line to connect the data push processor and the data preprocessor through the connection line, so that the data push processor sends the data output by the data preprocessor to a message queue or storage device to obtain data to be processed.

[0154] Step 804: The computer device configures data processing parameters in the distributed processing framework Flink.

[0155] Exemplarily, data processing parameters are configured in Flink, such as defining the source of the data to be processed and the storage location of the data results.

[0156] For example, by configuring Flink's data parameters to connect to the database where NiFi pushes data to be processed.

[0157] Step 805: The computer device continuously monitors the data to be processed and performs data processing.

[0158] For example, start Flink to process data.

[0159] Run Flink to execute the monitoring task, which continuously monitors Kafka from NiFi. Flink processes the pending data in Kafka. The specific processing process can be found in step 203 above and will not be repeated here.

[0160] Step 806: The computer device stores the data processing results in a database.

[0161] Exemplarily, the database includes a relational database, a distributed database, or a non-relational NoSQL database. After processing the data to be processed, the stream processing data analysis results obtained by processing in a stream processing mode are stored in the relational database. The batch data analysis results obtained by processing in a batch processing mode are stored in a distributed database or a non-relational NoSQL database.

[0162] Step 807: The computer device sends the data processing result to the user interface for display.

[0163] Exemplarily, a user interface is used to display and analyze the processed data.

[0164] For example, create dashboards and charts in the user interface to display data trends, statistics, and analysis results.

[0165] Step 808: End.

[0166] The above mainly introduces the solution provided by this application. Correspondingly, this application also provides a data processing device, which is used to implement the above method embodiment.

[0167] In some embodiments, the data processing device includes hardware structures and / or software modules corresponding to the execution of each function in order to realize the above functions. Those skilled in the art should easily realize that, in combination with the units and algorithm steps of each example described in the embodiments disclosed herein, the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0168] The embodiment of the present application can divide the data processing device into functional modules according to the above method embodiment. For example, each functional module can be divided according to each function, or two or more functions can be integrated into one processing module. The above integrated modules can be implemented in the form of hardware or in the form of software functional modules. It should be noted that the division of modules in the embodiment of the present application is schematic and is only a logical functional division. In actual implementation, there may be other division methods.

[0169] Figure 9 This is a structural diagram of a data processing device provided in an embodiment of the present application. Figure 9 The data processing device includes an acquisition module 901 and a data processing module 902.

[0170] The acquisition module 901 is used to process the data in the data source based on the workflow arranged by the user to obtain the data to be processed.

[0171] The data processing module 902 is used to execute data processing tasks based on the attribute parameters of the data to be processed. The data processing tasks are used to indicate that the data to be processed is processed according to the stream processing mode or batch processing mode corresponding to the attribute parameters. The attribute parameters are used to describe the data category to which the data to be processed belongs.

[0172] In one possible implementation, acquisition module 901 is further configured to collect data from a data source via a data collection processor in the workflow. The collected data is preprocessed by a data preprocessor in the workflow, including data cleaning and / or format conversion. The preprocessed data is then sent to a message queue or storage via a data push processor in the workflow to obtain data to be processed.

[0173] In another possible implementation, the data processing module 902 is further configured to read the data to be processed from the message queue or the memory, extract the attribute parameters of the data to be processed, and perform stream processing on the data to be processed according to the stream processing mode if the attribute parameters meet the stream processing conditions.

[0174] In another possible implementation, the data processing module 902 is further configured to read the data to be processed from the message queue or the memory and extract the attribute parameters of the data to be processed. If the attribute parameters do not meet the stream processing conditions or meet the batch processing conditions, the data to be processed is batch processed according to the batch processing mode.

[0175] In another possible implementation, the data processing module 902 is further configured to perform at least one of data aggregation, data filtering, and data statistics and data analysis on the data to be processed.

[0176] In another possible implementation, the data processing module 902 is further configured to periodically accumulate the data to be processed and then perform parallel processing.

[0177] In another possible implementation, the attribute parameters include: at least one of: response time requirement, data volume, processing complexity score, or amount of system resources required for data processing; the stream processing conditions include at least one of: response time requirement is less than or equal to a response time threshold; data volume is less than or equal to a data volume threshold; processing complexity is less than or equal to a preset complexity; or, the amount of system resources required for data processing is less than or equal to a resource volume threshold.

[0178] In another possible implementation, the batch processing conditions include at least one of the following: the response time requirement is greater than the response time threshold; the data volume of the data to be processed is greater than the data volume threshold; the processing complexity of the data to be processed is greater than the preset processing complexity; or the amount of system resources required for data processing is greater than the resource volume threshold.

[0179] In another possible implementation, the data processing module 902 is further configured to store the result of processing the data to be processed in a database corresponding to the data processing mode.

[0180] In another possible implementation, the data processing module 902 is further configured to store the stream processing data analysis results obtained by processing in the stream processing mode in a relational database, where a relational database refers to a database that stores data based on relational attributes.

[0181] In another possible implementation, the data processing module 902 is further configured to store the batch data analysis results obtained by batch processing in a distributed database or a non-relational database NoSQL. A distributed database refers to a database that stores data in a distributed manner.

[0182] In another possible implementation, the data processing module 902 is further configured to send the result of processing the data to be processed to a user interface for display.

[0183] In another possible implementation, the result of the data processing on the data to be processed can be queried, edited, or downloaded through a user interface or an API.

[0184] In another possible implementation, the acquisition module 901 is further configured to acquire the data processing task.

[0185] The data processing module 902 is also used to decompose the data processing task and determine the subtasks in the data processing task, where the subtasks include: multiple items of data acquisition, data preprocessing or data push; according to the processing order of the subtasks in the data processing task, the data acquisition processor corresponding to the data acquisition, the data preprocessor corresponding to the data preprocessing or the data push processor corresponding to the data push in the data processing task are arranged to obtain the workflow.

[0186] In another possible implementation, the acquisition module 901 is also used to parse the data processing requirements input by the user to obtain source data and data processing targets. The source data refers to the data that the user wants to process, and the data processing target refers to the data processing result that the user wants to obtain; the data processing task is obtained based on the source data and the data processing target.

[0187] like Figure 10 As shown, the computer device provided in the embodiment of the present application may include a processor 1001, a bus 1002, a communication interface 1003, and a memory 1004. The processor 1001, the memory 1004, and the communication interface 1003 communicate with each other via the bus 1002. It should be understood that the present application does not limit the number of processors and memories in the computer device.

[0188] The bus 1002 may be a PCI bus or an extended industry standard architecture (EISA) bus, or a USB bus. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 10 The bus 1002 may include a path for transmitting information between various components of a computer device (eg, memory 1004, processor 1001, communication interface 1003).

[0189] The processor 1001 may include any one or more processors such as a CPU, a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP).

[0190] The memory 1004 may include a volatile memory, such as a random access memory (RAM). The processor 1001 may also include a non-volatile memory, such as a read-only memory (ROM), a flash memory, a hard disk drive (HDD), or a solid state drive (SSD).

[0191] The communication interface 1003 uses a transceiver module such as, but not limited to, a network interface card or a transceiver to implement communication between the computer device and other devices or a communication network.

[0192] The memory 1004 stores executable program codes, and the processor 1001 executes the executable program codes to respectively implement the functions of the data processing device or the CPU core in the aforementioned method embodiment. That is, the memory 1004 stores the program codes for executing the aforementioned data processing method.

[0193] On the other hand, a computer-readable storage medium is provided, wherein at least one computer program is stored in the computer-readable storage medium, and the at least one computer program is loaded and executed by a processor to implement the data processing method provided in the above-mentioned method embodiments.

[0194] On the other hand, a computer program product is provided. The computer program product includes a computer program or instructions. When the computer program or instructions are executed by a processor, the data processing method described above is implemented.

[0195] On the other hand, a chip system is provided, comprising at least one processor and at least one interface circuit, wherein the at least one interface circuit is used to perform transceiver functions and send instructions to the at least one processor. When the at least one processor executes the instructions, the at least one processor executes to implement the data processing method described above.

[0196] The computer device, computer-readable storage medium, or computer program product provided in the embodiments of the present application are all used to execute the corresponding methods provided above. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects of the corresponding methods provided above, and will not be repeated here.

[0197] Through the description of the above embodiments, those skilled in the art can clearly understand that for the convenience and brevity of description, only the division of the above functional modules is used as an example. In actual application, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device (such as a computer device) is divided into different functional modules to complete all or part of the functions described above. The specific working process of the system, device (such as a computer device) and unit described above can refer to the corresponding process in the aforementioned method embodiment and will not be repeated here.

[0198] In the several embodiments provided in this application, it should be understood that the disclosed systems, devices (such as computer devices) and methods can be implemented in other ways. For example, the device (such as computer device) embodiments described above are merely schematic. For example, the division of the modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0199] The units described above as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0200] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0201] If the above-mentioned integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) or a processor to execute all or part of the steps of the method of each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as flash memory, mobile hard disk, read-only memory, random access memory, magnetic disk or optical disk.

[0202] The above description is merely a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of the present application. Therefore, the scope of protection of the present application should be based on the scope of protection of the claims.

Claims

1. A data processing method, characterized in that: The method comprises: Based on the workflow orchestrated by the user, the data in the data source is processed to obtain the data to be processed; Based on the attribute parameters of the data to be processed, a data processing task is executed, where the data processing task is used to indicate that the data to be processed is processed in accordance with the stream processing mode or batch processing mode corresponding to the attribute parameters, and the attribute parameters are used to describe the data category to which the data to be processed belongs.

2. The method according to claim 1, characterized in that The workflow based on user arrangement processes the data in the data source to obtain the data to be processed, including: Collecting data from the data source through a data collection processor in the workflow; Performing data preprocessing on the collected data by a data preprocessor in the workflow, wherein the data preprocessing includes data cleaning and / or format conversion; The data push processor in the workflow sends the pre-processed data to a message queue or storage to obtain the data to be processed.

3. The method according to claim 2, characterized in that The processing of the data to be processed based on the attribute parameters of the data to be processed and in accordance with the stream processing mode or the batch processing mode corresponding to the attribute parameters includes: Reading the data to be processed from the message queue or the memory, and extracting the attribute parameters of the data to be processed; If the attribute parameters of the data to be processed meet the stream processing conditions, stream processing is performed on the data to be processed according to the stream processing mode; or, If the attribute parameters of the data to be processed do not meet the stream processing condition or meet the batch processing condition, the data to be processed are batch processed according to the batch processing mode.

4. The method according to claim 3, characterized in that The attribute parameters include: at least one of response time requirement, data volume, processing complexity score, or system resource amount required for data processing; the stream processing conditions include at least one of the following: The response time requirement is less than or equal to the response time threshold; The data volume is less than or equal to the data volume threshold; The processing complexity is less than or equal to the preset complexity; Alternatively, the amount of system resources required for data processing is less than or equal to a resource threshold.

5. The method according to claim 3, characterized in that The attribute parameters include: at least one of response time requirement, data volume, processing complexity score, or system resource amount required for data processing; the batch processing conditions include at least one of the following: The response time requirement must be greater than the response time threshold; The amount of the data to be processed is greater than the data amount threshold; The processing complexity of the data to be processed is greater than the preset complexity; Alternatively, the amount of system resources required for data processing is greater than a resource threshold.

6. The method according to any one of claims 1 to 5, characterized in that The method further comprises: The result of data processing on the data to be processed is stored in the database corresponding to the data processing mode.

7. The method according to claim 6, characterized in that The step of storing the result of processing the data to be processed in a memory corresponding to the data processing mode includes: Storing the stream processing data analysis results obtained by processing in the stream processing mode in a relational database, wherein the relational database refers to a database that stores data based on relational attributes; or, The batch data analysis results obtained by processing in batch mode are stored in a distributed database or a non-relational database NoSQL. The distributed database refers to a database that stores data in a distributed form.

8. The method according to any one of claims 1 to 5, characterized in that The method further comprises: Obtaining the data processing task; Decomposing the data processing task to determine subtasks within the data processing task, wherein the subtasks include multiple items of data acquisition, data preprocessing, or data push; According to the processing order of each subtask in the data processing task, the processors corresponding to each subtask in the data processing task are arranged to obtain the workflow.

9. The method according to claim 8, characterized in that The obtaining of the data processing task includes: Parsing the data processing requirements input by the user to obtain source data and data processing targets; The data processing task is obtained according to the source data and the data processing target, and the data processing task is used to instruct to process the source data to obtain the data processing target.

10. A computer device, characterized in that: The computer device includes a processor and a memory; the processor and the memory are coupled; The memory is used to store computer program instructions; The processor is configured to call the computer program instructions in the memory to execute the method according to any one of claims 1 to 9.

Images