Data processing methods, devices, equipment and media

Abstract

This invention discloses a data processing method, apparatus, device, and medium, comprising: acquiring multi-source heterogeneous raw data to be processed, and standardizing the raw data to obtain standardized data to be processed; acquiring data processing logic, generating a first executable script corresponding to a batch data processing engine, and a second executable script corresponding to a streaming data processing engine; based on the first executable script, processing batch data in the standardized data to be processed through the batch data processing engine to obtain batch processing results; based on the second executable script, processing streaming data in the standardized data to be processed in real time through the streaming data processing engine to obtain streaming processing results; and determining a target data processing result based on the batch processing results and the streaming processing results. This method decomposes the data processing logic into executable scripts corresponding to the batch data processing engine and the streaming data processing engine, achieving unified scheduling and collaborative processing of batch data and streaming data.

Description

Technical Field

[0001] This invention relates to the field of multi-source data processing technology, and in particular to a data processing method, apparatus, device, and medium. Background Technology

[0002] In big data and enterprise digital transformation scenarios, data sources are diverse and heterogeneous, encompassing relational databases, object storage, IoT streaming data, and file systems. Simultaneously, business scenarios continuously increase demands for real-time performance, consistency, and traceability in data processing. Traditional architectures often separate batch processing and stream processing, failing to achieve data processing logic reuse and unified scheduling, and struggling to support responsive business data needs. Therefore, a data processing method that enables collaborative processing of batch and streaming data is urgently needed. Summary of the Invention

[0003] This invention provides a data processing method, apparatus, device, and medium to solve the problem of fragmented batch data processing logic and stream data processing logic in related technologies, which makes the logic difficult to reuse.

[0004] In a first aspect, the present invention provides a data processing method, comprising: Obtain the raw, heterogeneous data from multiple sources and standardize it to obtain standardized data. The data processing logic is obtained, and a first executable script corresponding to the batch data processing engine and a second executable script corresponding to the stream data processing engine are generated. Based on the first executable script, the batch data in the standardized data to be processed is processed by the batch data engine to obtain the batch processing results; Based on the second executable script, the streaming data in the standardized data to be processed is processed in real time through the streaming data processing engine to obtain the streaming processing result; Based on the batch processing results and the stream processing results, the target data processing result is determined.

[0005] In some embodiments, the data processing logic includes at least one of the following: In response to a logic selection instruction, the corresponding data processing template is retrieved from the preset logic library, and the data processing logic is determined based on the data processing template. In response to a logical editing command, the logical editing content is obtained, and based on the logical editing content, the data processing logic is determined.

[0006] In some embodiments, based on a first executable script, a batch data processing engine processes batch data in standardized data to be processed to obtain batch processing results, including: Based on the first executable script, the data partitioning strategy and / or data filtering strategy are determined through the batch data processing engine; Based on data partitioning and / or data filtering strategies, the first target data in the batch data is determined through the batch data processing engine; Based on the first target data, the first executable script is executed through the batch data processing engine to obtain the batch processing results.

[0007] In some embodiments, based on a second executable script, a streaming data processing engine processes the streaming data in the standardized data to be processed in real time to obtain streaming processing results, including: Based on the second executable script, at least one of the following is determined by the streaming data processing engine: data aggregation window, data processing state, and data processing sequence; Based on at least one of the data aggregation window, data processing status, and data processing sequence, the second target data in the streaming data is determined through the streaming data processing engine. Based on the second target data, the second executable script is executed through the streaming data processing engine to obtain the streaming processing result.

[0008] In some embodiments, determining the target data processing result based on batch processing results and stream processing results includes: Perform permission verification on batch processing results and stream processing results to obtain permission verification results; Based on the permission verification results, the batch processing results and stream processing results are anonymized to obtain the target data processing results.

[0009] In some embodiments, after determining the target data processing result based on batch processing results and stream processing results, the method further includes: Store the data processing logic and generate the corresponding logic version identifier; Associate the target data processing results with the logical version identifier to generate a data version snapshot.

[0010] In some embodiments, before processing batch data in standardized data to be processed using a batch data processing engine based on a first executable script to obtain batch processing results, the method further includes: Obtain the load parameters of the batch data processing engine and the streaming data processing engine; When the load parameters of the streaming data processing engine exceed the first preset load threshold, increase the computing resources of the streaming data processing engine; When the load parameters of the batch data processing engine are less than the first preset load threshold, the computing resources of the batch data processing engine are released.

[0011] Secondly, this disclosure provides a data processing apparatus, comprising: The data acquisition module is used to acquire multi-source heterogeneous raw data to be processed, and to standardize the raw data to be processed to obtain standardized data to be processed. The script generation module is used to obtain data processing logic and generate a first executable script corresponding to the batch data processing engine and a second executable script corresponding to the stream data processing engine. The batch processing module is used to process batch data in the standardized data to be processed based on the first executable script and through the batch data processing engine to obtain batch processing results. The stream processing module is used to process the stream data in the standardized data to be processed in real time based on the second executable script and through the stream data processing engine to obtain the stream processing result; The data generation module is used to determine the target data processing result based on batch processing results and stream processing results.

[0012] Thirdly, the present invention provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the above-described data processing method.

[0013] Fourthly, the present invention provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described data processing method.

[0014] The above-mentioned data processing method, apparatus, equipment, and medium implement a scheme in which the method includes: acquiring multi-source heterogeneous raw data to be processed, and standardizing the raw data to obtain standardized data to be processed; acquiring data processing logic, generating a first executable script corresponding to a batch data processing engine, and a second executable script corresponding to a streaming data processing engine; based on the first executable script, processing batch data in the standardized data to be processed through the batch data processing engine to obtain batch processing results; based on the second executable script, processing streaming data in the standardized data to be processed in real time through the streaming data processing engine to obtain streaming processing results; and determining the target data processing result based on the batch processing results and streaming processing results. This method, by splitting the data processing logic into executable scripts corresponding to the batch data processing engine and the streaming data processing engine respectively, achieves logical reuse between the batch data processing logic and the streaming data processing logic, as well as unified scheduling and collaborative processing of batch data and streaming data, reducing the resource overhead and operational complexity of data processing. Attached Figure Description

[0015] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0016] Figure 1 This is a flowchart of a data processing method according to an embodiment of the present invention; Figure 2 This is a flowchart of a data processing method according to an embodiment of the present invention; Figure 3 This is a flowchart of a data processing method according to an embodiment of the present invention; Figure 4 This is a flowchart of a data processing method according to an embodiment of the present invention; Figure 5 This is a flowchart of a data processing method according to an embodiment of the present invention; Figure 6 This is a flowchart of a data processing method according to an embodiment of the present invention; Figure 7 This is a schematic block diagram of a data processing device according to an embodiment of the present invention; Figure 8 This is a schematic block diagram of a computer device according to an embodiment of the present invention. Detailed Implementation

[0017] As an example, such as Figure 1 As shown, a data processing method is provided, including the following steps: S101: Obtain the raw data to be processed from multiple heterogeneous sources, and perform standardization processing on the raw data to be processed to obtain standardized data to be processed. S102, Obtain data processing logic, generate a first executable script corresponding to the batch data processing engine, and a second executable script corresponding to the stream data processing engine; S103, based on the first executable script, uses a batch data processing engine to process batch data in the standardized data to be processed, and obtains batch processing results; S104, based on the second executable script, uses a streaming data processing engine to process the streaming data in the standardized data to be processed in real time to obtain the streaming processing result; S105, based on batch processing results and stream processing results, determine the target data processing result.

[0018] As an example, in step S101, a lightweight acquisition agent deployed on the data source side can be used to obtain multi-source heterogeneous raw data to be processed, and the obtained raw data to be processed can be converted into a unified format to complete the standardization of the raw data to be processed, resulting in standardized data to be processed with a consistent format.

[0019] Among them, multi-source heterogeneous raw data to be processed refers to data that comes from different types of data storage carriers and has different data formats and transmission methods.

[0020] In an optional embodiment, metadata appending processing can also be performed on the original data to be processed, that is, during the standardization process, structured meta-information such as source identifier, collection timestamp, and data quality label is dynamically injected into each piece of data, thereby laying the foundation for data lineage tracing and anomaly location in subsequent processing stages.

[0021] In one optional embodiment, the original data to be processed can be acquired by full data collection, which can be used for the initialization scenario of first access to the data source; in another optional embodiment, the original data to be processed can be acquired by incremental data collection, which only collects the data content that has changed in the data source, and is suitable for the continuous data synchronization scenario after the data source has completed its first access.

[0022] For example, when collecting data from a database, the database log files can be parsed to identify operations such as insertion, update, and deletion, extracting the corresponding data change records. These changes are then used as raw data to be processed. Finally, a standardized format conversion is performed on this raw data to obtain standardized data to be processed. When collecting data through an Application Programming Interface (API), the corresponding data file can be obtained by calling the API. Based on the file format (e.g., CSV, JSON, Parquet), the metadata of the data file can be extracted and associated with the data body contained within the file, thus obtaining the raw data. When collecting data from IoT devices via the MQTT / Kafka protocol, the contextual features and event sequences of the streaming data can be extracted based on the timestamps and device identification information in the message header. This allows for time-series alignment and semantic annotation, thereby achieving standardized processing of the collected data.

[0023] As an example, in step S102, in response to the user's setting of the data processing logic, the data processing logic can be obtained, and based on the preset engine adaptation rules, the obtained data processing logic can be converted into a first executable script that can be executed by the batch data processing engine and a second executable script that can be executed by the streaming data processing engine.

[0024] Among them, data processing logic refers to a set of rules used to perform processing operations such as filtering, transformation, aggregation, and association on data, which can be adapted to the processing needs of batch data and / or streaming data.

[0025] In one optional embodiment, data processing logic can be defined through visual configuration, and the system automatically generates corresponding executable scripts based on the configured processing components and parameters; alternatively, data processing logic can be defined through code writing, i.e., generating corresponding executable scripts based on the code content written by the user.

[0026] In one optional embodiment, the data processing logic can be parsed to identify its logical structure and semantic dependencies, thereby splitting it into sub-logical units supported by the batch processing engine and the stream processing engine respectively, and then generating corresponding first executable scripts and second executable scripts.

[0027] As an example, in step S103, the generated first executable script can be sent to a preset batch data processing engine. The batch data processing engine loads and runs the script, performs corresponding processing operations on the batch data contained in the standardized data to be processed, and obtains the batch processing result.

[0028] Batch data can be a static dataset that has been accumulated over a certain period of time and has fixed data boundaries. It is suitable for scenarios such as offline statistics and historical feature calculation.

[0029] In one optional embodiment, the batch data processing engine can optimize the data processing flow using a preset calculation optimization strategy before performing processing operations, thereby reducing invalid data scanning and calculation.

[0030] In one optional embodiment, the batch data processing engine can record the entire process operation information of the processing operation for subsequent task status monitoring and troubleshooting.

[0031] As an example, in step S104, the generated second executable script can be sent to a preset streaming data processing engine. The streaming data processing engine loads and runs the script, performs corresponding real-time processing operations on the continuously arriving streaming data in the standardized data to be processed, and obtains the streaming processing result.

[0032] Among them, streaming data can be a dynamic data sequence that is continuously generated and arrives in chronological order without fixed data boundaries, and is suitable for scenarios such as real-time monitoring, real-time alarms, and real-time feature generation.

[0033] In one optional embodiment, the streaming data processing engine can report the real-time running status of the processing operation to monitor the running status and data processing progress of the streaming computing task in real time.

[0034] As an example, in step S105, batch processing results and stream processing results can be associated and integrated according to preset integration rules to obtain a unified target data processing result. For example, the historical data results obtained by the batch data processing engine from batch offline calculation of product data can be merged, matched, or fused with the dynamic data results obtained by the stream data processing engine from real-time stream calculation of stream data to form a comprehensive data output that simultaneously covers historical and real-time dimensions.

[0035] In one optional embodiment, the batch processing results and the stream processing results can be merged after field alignment and structural unification to obtain a unified dataset that includes both historical statistical data and real-time dynamic data; alternatively, the batch processing results and the stream processing results can be correlated and calculated according to preset association fields and association conditions to obtain a comprehensive data result that integrates historical data features and real-time data features.

[0036] For example, when generating a dynamic user profile, the user's historical preference tags obtained from batch processing can be correlated and fused with the user's real-time behavioral features obtained from stream processing to obtain a dynamic user profile result that includes the user's long-term preferences and real-time interests; when generating real-time business metrics, the historical business data obtained from batch processing and the real-time business data obtained from stream processing can be merged and compared to obtain business metric results that include historical benchmarks and real-time progress.

[0037] In summary, this disclosure proposes a data processing method, comprising: acquiring multi-source heterogeneous raw data to be processed, and standardizing the raw data to obtain standardized data to be processed; acquiring data processing logic, generating a first executable script corresponding to a batch data processing engine, and a second executable script corresponding to a streaming data processing engine; based on the first executable script, processing batch data in the standardized data to be processed through the batch data processing engine to obtain batch processing results; based on the second executable script, processing streaming data in the standardized data to be processed in real time through the streaming data processing engine to obtain streaming processing results; and determining the target data processing result based on the batch processing results and the streaming processing results. This method achieves logical reuse between batch data processing logic and streaming data processing logic, as well as unified scheduling and collaborative processing of batch data and streaming data, by splitting the data processing logic into executable scripts corresponding to the batch data processing engine and the streaming data processing engine, thereby reducing the resource overhead and operational complexity of data processing.

[0038] As an example, such as Figure 2 As shown, step S102, i.e., the data processing logic, includes at least one of the following: S201, in response to the logic selection instruction, retrieve the corresponding data processing template from the preset logic library, and determine the data processing logic based on the data processing template; S202, in response to a logic editing instruction, obtains the logic editing content and determines the data processing logic based on the logic editing content.

[0039] As an example, in step S201, in response to the logical selection instruction input by the user, a data processing template matching the business requirements can be retrieved from a preset logic library. Alternatively, the data processing template can be adjusted based on the parameter configuration information input by the user to determine the final data processing logic.

[0040] Among them, the preset logic library refers to a database that pre-stores multiple standardized and reusable data processing templates; the data processing template refers to a logical unit that encapsulates a general data processing flow and basic parameter configuration, which can adapt to the data processing needs of different business scenarios.

[0041] In one optional embodiment, data processing templates in the preset logic library can be classified and stored according to the type of business scenario, such as data cleaning templates, field mapping templates, window aggregation templates, data association templates, and outlier handling templates, to facilitate quick retrieval and retrieval by users; user-defined and verified data processing logic can also be automatically stored in the preset logic library as new data processing templates for subsequent reuse, further improving the content of the logic library.

[0042] For example, users can drag and drop components (i.e., the logic selection instructions mentioned above), such as dragging and dropping components corresponding to templates for data filtering, field mapping, window aggregation, real-time association, and outlier handling, to build batch and stream integrated data processing logic, thereby achieving visual orchestration and retrieval of data processing logic.

[0043] As an example, in step S202, in response to the logical editing instruction input by the user, the logical editing content submitted by the user can be obtained, and the logical editing content can also be subjected to syntax verification, dependency verification and legality detection. After the detection is passed, the final data processing logic is determined.

[0044] Among them, logical editing instructions refer to the instructions used by users to trigger editing operations such as creating, modifying, and deleting data processing logic; logical editing content refers to the content entered by users through the editing entry provided by the system, which is used to define the data processing flow and rules, and can adapt to the processing needs of both batch data and streaming data.

[0045] In one optional embodiment, the logical editing content can be obtained through a code editor, that is, in response to the programming language code content entered by the user in the code editing area, the corresponding code text is obtained as the logical editing content. Supported programming languages ​​include, but are not limited to, Python, Java and other programming languages.

[0046] In one optional embodiment, Git branch management mode can be used to record the content and time of each change in data processing logic, supporting version rollback and difference comparison; unit testing and canary release can also be performed on the determined data processing logic, first applying the new data processing logic to a portion of the data traffic to verify the running stability and the accuracy of the results before full release, so as to avoid the logic change affecting online data.

[0047] For example, users can write Python or Java code in the code editor to define data processing logic. The system obtains the code content as the logic editing content. After passing syntax verification and functional testing, the final custom data processing logic is determined. When new data processing logic needs to be launched, it can be applied to a portion of the production data traffic first, and the running status and accuracy of the results can be continuously monitored. After verification, it can be gradually switched to the full traffic.

[0048] As an example, such as Figure 3 As shown, step S103, which involves processing batch data in the standardized data to be processed using a batch data processing engine based on the first executable script to obtain batch processing results, includes: S301, based on the first executable script, determines the data partitioning strategy and / or data filtering strategy through the batch data processing engine; S302, based on data partitioning strategy and / or data filtering strategy, through batch data processing engine, determine the first target data in batch data; S303: Based on the first target data, the first executable script is executed through the batch data processing engine to obtain the batch processing results.

[0049] As an example, in step S301, based on the first executable script sent to the batch data processing engine, the batch data processing engine can parse the processing logic and data range requirements in the script to determine the corresponding data partitioning strategy and / or data filtering strategy.

[0050] Among them, the data partitioning strategy refers to the rules used to logically divide the batch-stored dataset, which is used to narrow the data scanning range; the data filtering strategy refers to the rules used to filter data records that do not meet the processing requirements, which is used to reduce the amount of invalid data calculation.

[0051] In one optional embodiment, the data partitioning strategy can be set based on dimensions such as time, region, and business type, for example, partitioning batch data by day or month; the data filtering strategy can be set based on conditions such as field values, data status, and business rules. The system executing this method can push the filtering conditions to the storage layer for execution, reducing the amount of data transmission.

[0052] For example, when the first executable script contains the processing logic of "statistically analyzing the historical behavior data of users within a specified time period", the batch data processing engine can parse and determine the data partitioning strategy as "partitioning by the date of the behavior" and the data filtering strategy as "the behavior date falls within the specified time interval", providing a basis for subsequent data filtering.

[0053] In an optional embodiment, other strategies, such as indexing strategies and predicate pushdown strategies, can also be determined based on the first executable script. This disclosure does not limit the scope of these strategies. For example, taking an indexing strategy as an example, the system can automatically identify frequently queried fields in the script and construct an index list, thereby improving the efficiency of partitioned data retrieval.

[0054] As an example, in step S302, based on the determined data partitioning strategy and / or data filtering strategy, the batch data processing engine can filter and locate the batch data to determine the first target data in the batch data that meets the processing requirements.

[0055] The first target data refers to the dataset in the batch data that meets the partition range requirements and filtering conditions and requires processing operations.

[0056] In one alternative embodiment, the first target data can be determined solely based on the data partitioning strategy, skipping all datasets that do not conform to the partitioning range; alternatively, the first target data can be determined by combining the data partitioning strategy and the data filtering strategy, while using columnar indexing technology to quickly locate the target field data, further improving data retrieval efficiency.

[0057] As an example, in step S303, the determined first target data can be loaded into the computing node of the batch data processing engine, and the processing operation defined in the first executable script can be executed by the batch data processing engine to obtain the corresponding batch processing result.

[0058] Among them, batch processing results refer to the structured data results generated after performing processing operations such as filtering, transformation, aggregation, and association on the first target data.

[0059] In one optional embodiment, after the batch data processing engine completes the processing operation, it can update the batch processing results to the target dataset and generate a corresponding data version snapshot. The data version snapshot is then associated with the processing logic version used in this operation and stored accordingly. In another optional embodiment, the batch data processing engine can record the entire process operation information of the processing operation, including task start time, execution duration, amount of data processed, and abnormal information, for subsequent task status monitoring and troubleshooting.

[0060] In one embodiment, the batch data processing engine can process the first target data offline to obtain batch processing results, thereby improving resource utilization efficiency.

[0061] As an example, such as Figure 4 As shown, step S104, which involves processing the stream data in the standardized data to be processed in real time using a stream data processing engine based on the second executable script, to obtain the stream processing result, includes: S401, based on the second executable script, determines at least one of the following through the streaming data processing engine: data aggregation window, data processing state, and data processing sequence; S402, based on at least one of the data aggregation window, data processing status, and data processing sequence, determine the second target data in the streaming data through the streaming data processing engine; S403, based on the second target data, executes the second executable script through the stream data processing engine to obtain the stream processing result.

[0062] As an example, in step S401, based on the second executable script sent to the streaming data processing engine, the streaming data processing engine can parse the real-time processing logic and calculation rules in the script to determine at least one of the corresponding data aggregation window, data processing status, and data processing sequence.

[0063] Among them, the data aggregation window refers to the time or quantity rules for dividing continuous streaming data without fixed boundaries into finite data segments, which is used to realize segmented aggregation calculation of streaming data; the data processing status refers to the historical intermediate results that need to be persistently saved during the streaming calculation process, which is used to support the cumulative calculation and correlation analysis across windows; the data processing sequence refers to the execution order followed when processing streaming data to ensure the time consistency of each data.

[0064] In one optional embodiment, the data aggregation window can be configured as a scrolling window, a session window, or other types, supporting time granularity settings from seconds to minutes; the data processing sequence can be implemented through a watermarking mechanism, generating watermarks based on event time, setting the maximum allowed data delay time to ensure the correct processing of out-of-order data, but is not limited to this, other sorting methods can also be used to determine the data processing sequence (for example, sorting streaming data according to the dependencies between data to determine the data processing sequence).

[0065] As an example, in step S402, the second target data in the streaming data can be determined by filtering and matching the continuously arriving streaming data through the streaming data processing engine based on at least one of the determined data aggregation window, data processing status, and data processing sequence.

[0066] The second target data can be real-time data to be processed that falls within the specified aggregation window range, is associated with the corresponding data processing status, and meets the data processing timing requirements. However, it is not limited to this and can be dynamically adjusted according to actual business needs. It can also be real-time data to be processed that only falls within the specified aggregation window range or real-time event stream fragments that are only associated with a specific data processing status.

[0067] In one alternative embodiment, the second target data may be determined solely based on the data aggregation window, and only the streaming data falling within the current window may be processed; alternatively, the second target data may be determined by combining the data aggregation window with the data processing sequence, but this is not the only option.

[0068] As an example, in step S403, the determined second target data can be loaded into the computing node of the streaming data processing engine, and the real-time processing operation defined in the second executable script can be executed by the streaming data processing engine to obtain the corresponding streaming processing result.

[0069] Among them, the stream processing result refers to the dynamic data result generated after performing real-time aggregation, association and other processing operations on the second target data.

[0070] In one optional embodiment, after the streaming data processing engine completes the processing operation, it can write the streaming processing result to the downstream storage system in real time or trigger a preset business alarm rule; the streaming data processing engine can also update the corresponding data processing status synchronously and persist the status to ensure that the program can continue to calculate after restarting or recovering from a fault.

[0071] In one embodiment, the streaming data processing engine can perform data processing on the second target data in a low-latency computing manner to obtain streaming processing results, thereby improving data processing efficiency.

[0072] As an example, such as Figure 5As shown, step S105, which determines the target data processing result based on the batch processing result and the stream processing result, includes: S501 performs permission verification on batch processing results and stream processing results to obtain permission verification results; S502, based on the permission verification result, performs desensitization processing on the batch processing result and the stream processing result to obtain the target data processing result.

[0073] As an example, in step S501, the preset access permission rules corresponding to the batch processing results and the stream processing results can be obtained, as well as the identity information and permission attributes of the data access requester. The requester's permissions are matched and verified with the data access permission rules to obtain the permission verification result.

[0074] In this context, permission verification refers to the process of verifying whether the data access subject has the legitimate permission to access the corresponding data content, which is used to ensure the security and compliance of data access throughout the entire chain.

[0075] In one optional embodiment, a complete permission verification log can be recorded, including the unique identifier of the requester, the request time, the range of requested data, the verification result, and other information, for subsequent security auditing and problem tracing.

[0076] For example, when a regular business user requests access to sensitive user transaction data, the system verifies their role permissions. If they only have basic user behavior data viewing permissions, the system returns a permission verification failure result. When the system internally analyzes service request data, it verifies the permission scope corresponding to their service key, allowing them to access only the specified fields required for the business and prohibiting access to irrelevant sensitive fields.

[0077] As an example, in step S502, based on the obtained permission verification result, the corresponding data desensitization rule can be matched, and the sensitive data in the batch processing result and the stream processing result can be dynamically processed according to the desensitization rule to obtain the target data processing result.

[0078] Data anonymization refers to the process of transforming sensitive data to hide the true data content while retaining data usability, in order to prevent the leakage of sensitive information.

[0079] In one optional embodiment, multiple de-identification processing methods can be supported, including partial hiding, replacement, and irreversible encryption. Corresponding de-identification rules can be configured according to the sensitive data type. Differentiated de-identification strengths can also be configured based on the permission verification results. Users with high permissions can view the complete original data, while users with low permissions can only view the de-identified data.

[0080] For example, for the mobile phone number field, the middle 4 digits are hidden; for the bank card number field, only the last 4 digits are displayed; for users with administrator privileges, the system can skip the de-identification process and directly return the content that has not been de-identified.

[0081] In one optional embodiment, after obtaining the target data processing result, the data value can be delivered in multiple scenarios through various standardized output channels, specifically including at least one of the following: automatically synchronizing the target data processing result to the enterprise data asset catalog, and automatically generating data asset description information based on the metadata of the target data processing result. The data asset description information may include at least one of the following: data source, update frequency, quality indicators, processing logic version, and access permissions, supporting business personnel to discover data value through keyword search and correlation analysis. It can also respond to data export commands and export the target data processing results to a preset database for storage. It supports both full export and incremental export modes and can configure scheduled export tasks through the scheduling platform.

[0082] It can also provide standardized data services for the target data processing results through preset APIs, and authenticate the requester based on the API key, and perform traffic control and access log recording, supporting third-party systems to call real-time data or batch data.

[0083] Furthermore, when outputting the target data processing results, TLS 1.3 can be used for encrypted transmission to ensure the confidentiality and integrity of the data during transmission. At the same time, when storing the target data processing results, batch processing results, stream processing results, etc., AES-256 encryption is used to ensure the security of static data.

[0084] As an example, such as Figure 6 As shown, after determining the target data processing result based on the batch processing result and the stream processing result in step S105, the following steps are also included: S601 stores the data processing logic and generates the corresponding logic version identifier; S602, associate the target data processing result with the logical version identifier, and generate a data version snapshot.

[0085] As an example, in step S601, the completed data processing logic can be stored in a preset logical version repository, and a globally unique logical version identifier can be generated for the stored data processing logic.

[0086] Among them, the logical version identifier refers to the identification information used to distinguish the data processing logic of different iteration versions, and can be associated with and record data such as the change content, change time, and operator of the corresponding logic.

[0087] In one optional embodiment, Git branch management mode can be used to version the data processing logic, record the specific content of each logic change, including the addition, modification, and deletion of components and the adjustment of code, and support the comparison of differences between different versions of logic and version rollback.

[0088] As an example, in step S602, the generated target data processing result can be associated with the logical version identifier, and a corresponding data version snapshot can be generated and stored based on the associated complete information.

[0089] Among them, a data version snapshot refers to a mirror file that records the complete state of the target data processing results and its associated processing logic version at a specific moment, which is used to realize the synchronous backtracking of data and data processing logic.

[0090] In one optional embodiment, the data version snapshot may simultaneously include metadata information such as the storage path of the target data processing result, the generation time, data quality indicators, and the associated logical version identifier; furthermore, it may also support the automatic generation of data version snapshots at preset time intervals, or support users to manually trigger snapshot generation, to meet the data backtracking needs in different scenarios.

[0091] In one optional embodiment, before processing the batch data in the standardized data to be processed using the batch data processing engine based on the first executable script to obtain the batch processing result, load balancing scheduling can also be performed to dynamically allocate computing tasks according to resource usage. Specifically, this can include: obtaining the load parameters of the batch data processing engine and the streaming data processing engine; increasing the computing resources of the streaming data processing engine when the load parameter of the streaming data processing engine is greater than a first preset load threshold; and releasing the computing resources of the batch data processing engine when the load parameter of the batch data processing engine is less than the first preset load threshold.

[0092] In other words, before batch data processing is performed by the batch data processing engine based on the first executable script, the load parameters of the batch data processing engine and the streaming data processing engine can be obtained in real time, and the obtained load parameters are compared with the first preset load threshold respectively. When the load parameter of the streaming data processing engine is greater than the first preset load threshold, computing resources are automatically added to the streaming data processing engine. When the load parameter of the batch data processing engine is less than the first preset load threshold, the idle computing resources of the batch data processing engine are automatically released.

[0093] In one optional embodiment, elastic scheduling of computing resources can be achieved based on the Kubernetes container orchestration framework, and the computing nodes of the batch data processing engine and the streaming data processing engine can be deployed in a containerized manner to support the scaling up and down of computing resources; the load parameters may include at least one of memory utilization, task queue waiting length, data processing latency, etc.

[0094] In one optional embodiment, a cooldown time can be set for computing resource adjustments. After a resource adjustment operation is completed, the next load detection and resource adjustment will be performed after a preset interval, so as to avoid frequent scaling up and down caused by instantaneous load fluctuations and ensure the stability of system operation.

[0095] In one optional embodiment, an end-to-end lineage tracing link can be constructed based on the data flow path from the original data to be processed to the target data processing result. The input data, output data, execution engine, logical version identifier and snapshot generation event of each processing link are structurally associated to support full-link lineage visualization in mixed batch processing and stream processing scenarios. This not only locates the source field and processing node of abnormal data, but also accurately restores the historical processing status by combining version snapshots, providing a reliable basis for fault review and model iteration.

[0096] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0097] In one embodiment, a data processing apparatus is provided, which corresponds one-to-one with the data processing methods described in the above embodiments. For example... Figure 7 As shown, the data processing device includes a data acquisition module 701, a script generation module 702, a batch processing module 703, a stream processing module 704, and a data generation module 705. Detailed descriptions of each functional module are as follows: The data acquisition module 701 is used to acquire multi-source heterogeneous raw data to be processed and to standardize the raw data to be processed to obtain standardized data to be processed. The script generation module 702 is used to obtain data processing logic and generate a first executable script corresponding to the batch data processing engine and a second executable script corresponding to the stream data processing engine. The batch processing module 703 is used to process batch data in the standardized data to be processed based on the first executable script and through the batch data processing engine to obtain batch processing results. The stream processing module 704 is used to process the stream data in the standardized data to be processed in real time based on the second executable script and through the stream data processing engine to obtain the stream processing result; The data generation module 705 is used to determine the target data processing result based on the batch processing result and the stream processing result.

[0098] In one embodiment, the script generation module 702 is further configured to, in response to a logic selection instruction, retrieve a corresponding data processing template from a preset logic library, and determine data processing logic based on the data processing template; In response to a logical editing command, the logical editing content is obtained, and based on the logical editing content, the data processing logic is determined.

[0099] In one embodiment, the batch processing module 703 is further configured to determine a data partitioning strategy and / or a data filtering strategy based on a first executable script and through a batch data processing engine. Based on data partitioning and / or data filtering strategies, the first target data in the batch data is determined through the batch data processing engine; Based on the first target data, the first executable script is executed through the batch data processing engine to obtain the batch processing results.

[0100] In one embodiment, the stream processing module 704 is further configured to, based on a second executable script, determine at least one of a data aggregation window, a data processing state, and a data processing sequence through a stream data processing engine; Based on at least one of the data aggregation window, data processing status, and data processing sequence, the second target data in the streaming data is determined through the streaming data processing engine. Based on the second target data, the second executable script is executed through the streaming data processing engine to obtain the streaming processing result.

[0101] In one embodiment, the data generation module 705 is further configured to perform permission verification on the batch processing results and the stream processing results to obtain permission verification results; Based on the permission verification results, the batch processing results and stream processing results are anonymized to obtain the target data processing results.

[0102] In one embodiment, the data generation module 705 is further configured to store data processing logic and generate a corresponding logic version identifier; Associate the target data processing results with the logical version identifier to generate a data version snapshot.

[0103] In one embodiment, the script generation module 702 is further configured to obtain the load parameters of the batch data processing engine and the streaming data processing engine; When the load parameters of the streaming data processing engine exceed the first preset load threshold, increase the computing resources of the streaming data processing engine; When the load parameters of the batch data processing engine are less than the first preset load threshold, the computing resources of the batch data processing engine are released.

[0104] This invention provides a data processing apparatus, comprising: a data acquisition module for acquiring multi-source heterogeneous raw data to be processed and standardizing the raw data to obtain standardized data to be processed; a script generation module for acquiring data processing logic and generating a first executable script corresponding to a batch data processing engine and a second executable script corresponding to a stream data processing engine; a batch processing module for processing batch data in the standardized data to be processed based on the first executable script and through the batch data processing engine to obtain batch processing results; a stream processing module for processing stream data in the standardized data to be processed in real time based on the second executable script and through the stream data processing engine to obtain stream processing results; and a data generation module for determining a target data processing result based on the batch processing results and the stream processing results. This apparatus achieves logical reuse between batch data processing logic and stream data processing logic, as well as unified scheduling and collaborative processing of batch data and stream data, by splitting the data processing logic into executable scripts corresponding to the batch data processing engine and the stream data processing engine, thereby reducing the resource overhead and operational complexity of data processing.

[0105] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 8 As shown, the computer device includes a processor, memory, network interface, and database connected via a system bus. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and database. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The database stores data used in data processing methods. The network interface is used for communication with external terminals via a network connection. When executed by the processor, the computer program can implement a data processing method.

[0106] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement a data processing method.

[0107] In one embodiment, a computer-readable storage medium is provided that stores a computer program, which, when executed by a processor, implements a data processing method.

[0108] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. This computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), IAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0109] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is used as an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0110] The above-described embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.

Claims

1. A data processing method, characterized in that, include: Obtain multi-source heterogeneous raw data to be processed, and perform standardization processing on the raw data to be processed to obtain standardized data to be processed. The data processing logic is obtained, and a first executable script corresponding to the batch data processing engine and a second executable script corresponding to the stream data processing engine are generated. Based on the first executable script, the batch data in the standardized data to be processed is processed by the batch data processing engine to obtain the batch processing result; Based on the second executable script, the streaming data in the standardized data to be processed is processed in real time through the streaming data processing engine to obtain the streaming processing result; Based on the batch processing results and the stream processing results, the target data processing result is determined.

2. The method according to claim 1, characterized in that, The data acquisition and processing logic includes at least one of the following: In response to a logic selection instruction, the corresponding data processing template is retrieved from a preset logic library, and the data processing logic is determined based on the data processing template. In response to a logical editing instruction, the logical editing content is obtained, and the data processing logic is determined based on the logical editing content.

3. The method according to claim 1, characterized in that, The step of processing batch data in the standardized data to be processed based on the first executable script, through the batch data processing engine, to obtain batch processing results includes: Based on the first executable script, the batch data processing engine determines the data partitioning strategy and / or data filtering strategy. Based on the data partitioning strategy and / or the data filtering strategy, the first target data in the batch data is determined by the batch data processing engine. Based on the first target data, the first executable script is executed through the batch data processing engine to obtain the batch processing result.

4. The method according to claim 1, characterized in that, The process, based on the second executable script and through the streaming data processing engine, performs real-time processing on the streaming data in the standardized data to be processed to obtain streaming processing results, including: Based on the second executable script, at least one of the following is determined by the streaming data processing engine: data aggregation window, data processing state, and data processing sequence. Based on at least one of the data aggregation window, the data processing status, and the data processing sequence, the second target data in the streaming data is determined by the streaming data processing engine. Based on the second target data, the second executable script is executed through the streaming data processing engine to obtain the streaming processing result.

5. The method according to claim 1, characterized in that, The determination of the target data processing result based on the batch processing result and the stream processing result includes: Perform permission verification on the batch processing results and the stream processing results to obtain permission verification results; Based on the permission verification result, the batch processing result and the stream processing result are anonymized to obtain the target data processing result.

6. The method according to claim 1, characterized in that, After determining the target data processing result based on the batch processing result and the stream processing result, the method further includes: Store the data processing logic and generate a corresponding logic version identifier; By associating the target data processing result with the logical version identifier, a data version snapshot is generated.

7. The method according to claim 1, characterized in that, Before the step of processing the batch data in the standardized data to be processed based on the first executable script and through the batch data processing engine to obtain the batch processing result, the method further includes: Obtain the load parameters of the batch data processing engine and the streaming data processing engine; When the load parameter of the streaming data processing engine is greater than the first preset load threshold, the computing resources of the streaming data processing engine are increased. When the load parameter of the batch data processing engine is less than the first preset load threshold, the computing resources of the batch data processing engine are released.

8. A data processing apparatus, characterized in that, include: The data acquisition module is used to acquire multi-source heterogeneous raw data to be processed, and to perform standardization processing on the raw data to be processed to obtain standardized data to be processed. The script generation module is used to obtain data processing logic and generate a first executable script corresponding to the batch data processing engine and a second executable script corresponding to the stream data processing engine. The batch processing module is used to process batch data in the standardized data to be processed based on the first executable script and through the batch data processing engine to obtain batch processing results. The stream processing module is used to process the stream data in the standardized data to be processed in real time based on the second executable script and through the stream data processing engine to obtain the stream processing result; The data generation module is used to determine the target data processing result based on the batch processing result and the stream processing result.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the data processing method as described in any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the data processing method as described in any one of claims 1 to 7.

Images