A data processing method in a stream computing scenario

Abstract

The application discloses a data processing method in a stream computing scene and relates to the technical field of data processing, which is executed by a stream computing platform and comprises the following steps: performing deserialization processing on metadata of an original CDC event to obtain structured metadata; determining whether the original CDC event is a deletion semantic event according to a value of an operation type field and a value of a business primary key field of the structured metadata; if the original CDC event is the deletion semantic event, performing semantic conversion processing on the structured metadata according to an event type of the structured metadata to obtain semantic conversion metadata; performing deserialization processing on business data of the original CDC event to obtain structured business data; and generating a RowData event according to the semantic conversion metadata and the structured business data and sending the RowData event to a downstream application for consumption. The application realizes semantic conversion of the original CDC event in the stream computing platform between an upstream database and a downstream application, avoids repeated conversion of the downstream application, and improves the processing efficiency of the event.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a data processing method in a streaming computing scenario. Background Technology

[0002] In scenarios such as real-time data warehouses, real-time lake warehouse integration, master data synchronization, and business system data subscription, CDC (Change Data Capture) technology is typically used to collect change events such as additions, modifications, and deletions in upstream databases in real time and send them to message queues or streaming computing platforms in encoding formats such as JSON.

[0003] In existing technologies, streaming computing platforms typically parse received CDC events and generate RowData events, which are then sent to downstream applications for further processing. During this downstream processing, the same semantic transformation logic may be repeatedly executed, leading to numerous redundant operations and low data processing efficiency. Summary of the Invention

[0004] This invention provides a data processing method in a streaming computing scenario to achieve semantic transformation of raw CDC events in a streaming computing platform between an upstream database and a downstream application, avoiding repeated transformations by the downstream application and improving event processing efficiency.

[0005] In a first aspect, the present invention provides a data processing method in a streaming computing scenario, comprising:

[0006] The metadata of the original CDC event sent from the upstream database is deserialized to obtain the structured metadata of the original CDC event;

[0007] Based on the operation type field value and business primary key field value of the structured metadata, determine whether the original CDC event is a deletion semantic event;

[0008] If the original CDC event is a deletion semantic event, then the structured metadata is semantically transformed according to the event type of the structured metadata to obtain semantically transformed metadata;

[0009] The original CDC event's business data is deserialized to obtain structured business data;

[0010] Based on the semantic transformation metadata and the structured business data, a RowData event is generated and sent to downstream applications for consumption.

[0011] In a second aspect, the present invention also provides a data processing apparatus, comprising:

[0012] The first deserialization module is used to deserialize the metadata of the original CDC event sent by the upstream database to obtain the structured metadata of the original CDC event.

[0013] The determination module is used to determine whether the original CDC event is a deletion semantic event based on the operation type field value and business primary key field value of the structured metadata;

[0014] The semantic conversion module is used to perform semantic conversion processing on the structured metadata according to the event type of the original CDC event if the original CDC event is a deletion semantic event, so as to obtain semantically converted metadata.

[0015] The second deserialization module is used to deserialize the business data of the original CDC event to obtain structured business data.

[0016] The event generation module is used to generate RowData events based on the semantic transformation metadata and the structured business data, and send them to downstream applications for consumption.

[0017] Thirdly, the present invention also provides an electronic device, comprising:

[0018] At least one processor; and

[0019] A memory that is communicatively connected to at least one processor; wherein

[0020] The memory stores instructions that can be executed by at least one processor, which enables the at least one processor to perform the data processing method in the streaming computing scenario provided in any embodiment of the present invention.

[0021] Fourthly, the present invention also provides a computer-readable storage medium storing computer instructions for causing a processor to execute and implement the data processing method in the streaming computing scenario of any embodiment of the present invention.

[0022] Fifthly, the present invention also provides a computer program product, characterized in that the computer program product includes a computer program, which, when executed by a processor, implements the data processing method in the streaming computing scenario of any embodiment of the present invention.

[0023] This invention deserializes the metadata of the original CDC event during the streaming computing platform processing; performs semantic recognition and transformation based on the metadata of the original CDC, and deserializes the business data of the original CDC data to obtain structured business data. Through the structured business data and semantic transformation metadata, a unified standardized RowData event is output, thereby unifying the semantic transformation processing scattered across various downstream applications into the streaming computing platform. This avoids multiple downstream applications repeatedly performing the same semantic transformation, reduces redundant code development and maintenance costs, and reduces the amount of data processed by downstream applications during operation, improving processing efficiency and achieving unified governance of the streaming computing platform.

[0024] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description

[0025] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0026] Figure 1 This is a flowchart of a data processing method in a streaming computing scenario provided by Embodiment 1 of the present invention;

[0027] Figure 2 This is a flowchart of a data processing method in a streaming computing scenario provided by Embodiment 2 of the present invention;

[0028] Figure 3 This is a schematic diagram of the structure of a data processing device according to Embodiment 3 of the present invention;

[0029] Figure 4 This is a structural diagram of an electronic device that implements a data processing method in a streaming computing scenario according to an embodiment of the present invention. Detailed Implementation

[0030] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0031] It should be noted that the terms "first" and "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0032] The acquisition, storage, and application of original CDC events, etc., in the technical solutions of the embodiments of the present invention all comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0033] Example 1

[0034] Figure 1 This is a flowchart of a data processing method in a streaming computing scenario provided in Embodiment 1 of the present invention. This embodiment is applicable to the situation of semantic transformation of CDC events in a streaming computing platform. The method can be executed by a data processing device, which can be implemented in hardware and / or software and specifically configured in an electronic device.

[0035] See Figure 1 The data processing methods in the streaming computing scenario shown are executed by the streaming computing platform and include:

[0036] S101. Deserialize the metadata of the original CDC event sent by the upstream database to obtain the structured metadata of the original CDC event.

[0037] S102. Determine whether the original CDC event is a deletion semantic event based on the operation type field value and business primary key field value of the structured metadata.

[0038] S103. If the original CDC event is a deletion semantic event, then according to the event type of the structured metadata, the structured metadata is semantically transformed to obtain semantically transformed metadata.

[0039] S104. Convert the semantic transformation metadata into RowData events and send the RowData events to downstream applications for consumption.

[0040] In this embodiment, the upstream database can be a database that communicates with the streaming computing platform and sends CDC events to the streaming computing platform. The original CDC event is the CDC event received by the streaming computing platform. The structured metadata can be the deserialization result of the metadata of the original CDC event. The structured metadata includes at least one field and the value of each field; wherein, the fields of the structured metadata may include, but are not limited to, a database name field, a table name field, an operation type field, an event time field, and a business primary key field.

[0041] The operation type field value is the field value of the operation type field, which can be used to represent the operation type of the original CDC event; the business primary key field value is the field value of the business primary key field, which can be used to uniquely identify the original CDC event. A deletion semantic event can be a CDC event representing the deletion of corresponding business data. The event type of a deletion semantic event can include, but is not limited to, a first deletion type and a second deletion type. The first deletion type can refer to the type of business data entity being deleted; the second deletion type can refer to the type of business data entity being retained, marking the business data as invalid. Semantic transformation metadata can be the semantic transformation result of structured metadata. RowData events can be events in row structure format. Downstream applications can be applications that communicate with the streaming computing platform and consume the RowData events output by the streaming computing platform; specific downstream applications can be real-time data warehouses, data synchronization systems, and data subscription systems, etc.

[0042] Specifically, the metadata of the original CDC event sent from the upstream database is deserialized to obtain the structured metadata of the original CDC event; at least one field value is extracted from the structured metadata, which may include, but is not limited to, operation type field and business primary key field value; using a certain algorithm, based on the operation type field value and business primary key field value of the structured metadata, it is determined whether the original CDC event is a deletion semantic event; if the original CDC event is a deletion semantic event, semantic transformation processing is performed on the structured metadata according to the event type of the structured metadata to obtain semantic transformation metadata; the business data of the original CDC event is deserialized to obtain structured business data; the structured business data and the semantic transformation metadata are output as a RowData event with a unified row structure, and the RowData event is sent to the downstream application for consumption.

[0043] Optionally, while performing semantic transformation processing on the structured metadata according to the event type of the structured metadata to obtain semantically transformed metadata, the method further includes: during the semantic transformation processing of the structured metadata, performing anomaly detection on the transformed fields and determining the transformation anomaly type of the anomaly field; querying the processing type of the anomaly field according to the transformation anomaly type, and executing the processing operation corresponding to the processing type; the processing type includes alarm type, empty type, repair type, and discard type.

[0044] Among them, the conversion exception types may include, but are not limited to, illegal database name exception type, illegal table name exception type, unknown operation type exception type, null event time exception type, abnormal event time format exception type, null business primary key exception type, abnormal business primary key format exception type, abnormal business primary key value out of bounds exception type, abnormal business field null value exception type, and abnormal business field value exceeding limit exception type.

[0045] The following exceptions apply: Illegal Database Name: The converted database name field contains an invalid database name, such as a database name not included in the preset whitelist of valid database names. Illegal Table Name: The converted table name field contains an invalid table name, such as a table name not within the scope of the business synchronization data tables. Unknown Operation Type: The operation type field's value is not a value from the preset standard change field. Null Event Time: The event time field contains a null value. Event Time Format: The event time field's format is not the set date and time format. Null Business Primary Key: The business primary key field contains a null value. Business Primary Key Format: The business primary key field contains non-numeric characters such as letters or symbols. Business Primary Key Value Out of Bounds: The business primary key field's value (numerical value) exceeds the set value range. Null Business Field: The business field contains a null value. Business Field Value Exceeds Limit: The business field's value (numerical value) exceeds the set value range.

[0046] Alarm type can be the type that generates alarm logs and sends them to the bypass link for processing; set type can be the type that sets the field value of the abnormal field to empty; repair type can be the type that replaces the field value of the abnormal field with the default value; the discard type can be the type that discards the semantic transformation metadata containing the abnormal field.

[0047] Specifically, based on the event type of the structured metadata, semantic transformation is performed on the structured metadata to obtain semantically transformed metadata. Simultaneously, anomaly detection is performed on the transformed fields, and the transformation anomaly type of the abnormal fields is determined. If the transformation anomaly type is an unknown operation type anomaly, a null value anomaly for the business primary key, or a null value anomaly for the business field, the processing type is determined to be an alarm type. If the transformation anomaly type is an null value anomaly for the event time or a value out of bounds anomaly for the business primary key, the processing type is determined to be a null type. If the transformation anomaly type is an event time format anomaly or a value out of bounds anomaly for the business field, the processing type is determined to be a repair type. If the transformation anomaly type is an illegal database name anomaly, an illegal table name anomaly, or a business primary key format anomaly, the processing type is determined to be a discard type.

[0048] If the processing type is an alarm type, an alarm log of the conversion anomaly type is generated, and the semantic conversion process ends. The structured metadata is output to the bypass link for processing to avoid sending the semantic conversion metadata with anomalies to downstream applications for consumption. If the processing type is a null type, the field value of the anomaly field is set to null, and the semantic conversion process continues. If the processing type is a repair type, the field value of the anomaly field is restored to the default field value of the anomaly field, and the semantic conversion process continues. If the processing type is a discard type, the semantic conversion process ends, and the structured metadata is output to the bypass link for processing.

[0049] Understandably, by adopting the above technical solution, field-level anomaly detection is performed during semantic transformation processing, and the matching processing type is determined based on the anomaly type of the anomaly field. Structured alarm information is generated through alarm types to facilitate rapid tracing of the root cause of problems by technical personnel. Erroneous and invalid data is discarded through discarding types. After processing the anomaly field through empty and repair types, subsequent processes continue, preventing a single field anomaly from causing the entire data link to be interrupted and reducing the flow of invalid data. This improves the system's fault tolerance and enhances the stability and reliability of real-time data processing.

[0050] In an optional embodiment, semantic transformation metadata can also be filtered according to pre-configured filtering rules. Each of the pre-configured filtering rules can be combined with at least one of the following fields: library name field, table name field, event type field, and event time field. Semantic transformation metadata that does not meet the rules is discarded, while semantic transformation metadata that meets the rules is retained. This allows the streaming computing platform to determine whether to filter the original CDC event based on the set filtering rules and semantic transformation metadata after determining the semantic transformation metadata. The business data of the original CDC event that meets the rules is deserialized, which facilitates the subsequent generation of unified RowData data. Compared with the existing technology where downstream applications receive the original CDC event and perform full parsing and rule filtering, this reduces the consumption of computing resources.

[0051] In another optional embodiment, the metadata of the semantic transformation metadata can be supplemented before converting the semantic transformation metadata into RowData events. That is, a rule hit field can be created according to the rules satisfied by the semantic transformation metadata, and the field value of the rule hit field can be set to the rule satisfied by the semantic transformation metadata. If the processing type corresponding to the semantic transformation metadata is a null value type and a repair type, an exception field can also be created according to the corresponding transformation exception type, and the field value of the exception field can be set to the transformation exception type, thereby improving the interpretability and auditability of the data processing process.

[0052] This invention deserializes the metadata of the original CDC event during the streaming computing platform processing; performs semantic recognition and transformation based on the metadata of the original CDC, and deserializes the business data of the original CDC data to obtain structured business data. Through the structured business data and semantic transformation metadata, a unified standardized RowData event is output, thereby unifying the semantic transformation processing scattered across various downstream applications into the streaming computing platform. This avoids multiple downstream applications repeatedly performing the same semantic transformation, reduces redundant code development and maintenance costs, and reduces the amount of data processed by downstream applications during operation, improving processing efficiency and achieving unified governance of the streaming computing platform.

[0053] Example 2

[0054] Figure 2 This is a flowchart of a data processing method in a streaming computing scenario provided in Embodiment 2 of the present invention. The present invention is an optimization and improvement based on the technical solution of the above embodiments.

[0055] Furthermore, the statement "determine whether the original CDC event is a deletion semantic event based on the operation type field value and the business primary key field value of the structured metadata" is refined to "if the operation type field value of the structured metadata is a first string, or the business primary key field value of the structured metadata is a negative value, then the original CDC event is determined to be a deletion semantic event".

[0056] It should be noted that for any parts not described in detail in the embodiments of the present invention, please refer to the description in the foregoing embodiments.

[0057] See Figure 2 The data processing methods shown in the streaming computing scenario include:

[0058] S201. Deserialize the metadata of the original CDC event sent by the upstream database to obtain the structured metadata of the original CDC event.

[0059] S202. If the operation type field value of the structured metadata is a first string, or the business primary key field value of the structured metadata is a negative value, then the original CDC event is determined to be a deletion semantic event.

[0060] S203. If the original CDC event is a deletion semantic event, then according to the event type of the structured metadata, the structured metadata is semantically transformed to obtain semantically transformed metadata.

[0061] S204. Convert the semantic transformation metadata into RowData events and send the RowData events to downstream applications for consumption.

[0062] In this embodiment, the first string can be a string representing a physical deletion type; for example, the first string can be "DELETE". The negative value type can be a negative number.

[0063] Optionally, after determining that the original CDC event is a deletion semantic event, the method further includes: if the operation type field value of the structured metadata is a first string, and the business primary key field value of the structured metadata is not a negative value, then the event type of the original CDC event is determined to be a first deletion type; the first string is a string representing a physical deletion type; if the operation type field value of the structured metadata is not the first string, and the business primary key field value of the structured metadata is a negative value, then the event type of the original CDC event is determined to be a second deletion type; if the operation type field value of the structured metadata is the first string, and the business primary key field value of the structured metadata is a negative value, then the event type of the original CDC event is determined to be a third deletion type; wherein, the third deletion type can refer to deleting a business data entity, or marking business data as invalid.

[0064] Understandably, by adopting the above technical solution, the specific type of deletion semantic event can be determined in the first deletion type of physical deletion, the second deletion type of negative primary key implicit deletion, and the third deletion scenario of mixed deletion, thereby eliminating deletion semantic bias and matching different subsequent semantic conversion processes to achieve targeted semantic conversion and improve the accuracy of semantic conversion.

[0065] Optionally, the step of performing semantic transformation processing on the structured metadata according to the event type of the structured metadata to obtain semantically transformed metadata includes: if the event type of the structured metadata is a first deletion type, then writing a logical deletion identifier field into the structured metadata and setting the value of the logical deletion identifier field to a deletion status string; changing the value of the operation type field in the structured metadata to a second string; and performing type transformation on each field in the structured metadata to obtain semantically transformed metadata.

[0066] The deletion status string, serving as the field value of the logical deletion identifier field, can be used to characterize an event where the semantically converted metadata is a logical deletion. Logical deletion can be understood as not deleting the business data entity corresponding to the event, but rather marking the business data corresponding to the event as invalid through the field value. The second string can be a string used to characterize an operation type of update; for example, the second string could be "UPDATE".

[0067] Specifically, if the event type of the structured metadata is the first deletion type, a logical deletion identifier field is written into the structured metadata, for example, the field name can be "is_deleted", and the value of the logical deletion identifier field is set to the deletion status string, for example, the field value can be "1"; the value of the operation type field in the structured metadata is changed to the second string "UPDATE"; for each field in the structured metadata, spaces in the field name are removed and converted to the set field name style, for example, removing underscores in the field name; the field value of the event time field is converted to a long integer, and the field value of the business primary key field is converted to a long integer, to obtain semantic transformation metadata.

[0068] Understandably, by adopting the above technical solution, for the structured metadata of the original CDC event of the first deletion type of physical deletion, a logical deletion identifier field is created to convert physical deletion into logical deletion, thereby preserving the business data in the original CDC event; by modifying the field value of the operation type to change the operation type to update, it adapts to the update processing logic of downstream applications, thereby avoiding anomalies in the downstream applications during processing; by converting each field to a fixed standard type, subsequent downstream application processing anomalies are avoided, thereby improving data compatibility and flow stability in the subsequent downstream application processing.

[0069] Optionally, the step of performing semantic transformation processing on the structured metadata according to the event type of the structured metadata to obtain semantically transformed metadata includes: if the event type of the structured metadata is a second deletion type, then writing a logical deletion identifier field into the structured metadata and setting the value of the logical deletion identifier field to a deletion status string; changing the value of the business primary key field in the structured metadata to a positive number; and performing type transformation on each field in the structured metadata to obtain semantically transformed metadata.

[0070] In this context, positive number types can be positive values. Specifically, if the event type of the structured metadata is the second deletion type, a logical deletion identifier field is written into the structured metadata, and the value of the logical deletion identifier field is set to a deletion status string; by removing the negative sign character from the value of the business primary key field, the value of the business primary key field in the structured metadata is changed to a positive number type; for each field in the structured metadata, the field is type-converted to obtain semantically converted metadata. It should be noted that the process of type conversion for the fields is similar to the process of type conversion for each field when the event type of the structured metadata is the first deletion type, and will not be repeated here.

[0071] Understandably, by adopting the above technical solution, a logical deletion identifier field is created to represent the deletion semantics, and the negative numeric field values ​​of the business primary key field are corrected to positive numeric field values. Based on the deletion semantics represented by the logical deletion identifier field, the validity of the business primary key field value is restored by correcting the negative numeric field values ​​of the business primary key field to positive numeric field values. By converting each field to a fixed standard type, subsequent downstream application processing anomalies are avoided, thereby improving data compatibility and flow stability in subsequent downstream application processing.

[0072] This invention determines whether the original CDC event is a deletion semantic event by judging whether the operation type is the first string and whether the business primary key is negative. It can identify explicit physical deletion in the database and implicit deletion in the business, avoid missed judgments and false judgments, and improve the accuracy of deletion semantic recognition.

[0073] Example 3

[0074] Figure 3 This is a schematic diagram of a data processing device provided in Embodiment 3 of the present invention. This embodiment of the invention is applicable to the semantic transformation of CDC events in a streaming computing platform. The device can execute data processing methods in streaming computing scenarios. The data processing device can be implemented in hardware and / or software, and can be configured in an electronic device.

[0075] See Figure 3 The data processing device shown includes a first deserialization module 301, a determination module 302, a semantic conversion module 303, a second deserialization module 304, and an event generation module 305, wherein...

[0076] The first deserialization module 301 is used to deserialize the metadata of the original CDC event sent by the upstream database to obtain the structured metadata of the original CDC event.

[0077] The determination module 302 is used to determine whether the original CDC event is a deletion semantic event based on the operation type field value and the business primary key field value of the structured metadata;

[0078] The semantic conversion module 303 is used to perform semantic conversion processing on the structured metadata according to the event type of the structured metadata if the original CDC event is a deletion semantic event, so as to obtain semantically converted metadata.

[0079] The second deserialization module 304 is used to deserialize the business data of the original CDC event to obtain structured business data.

[0080] The event generation module 305 is used to generate RowData events based on the semantic transformation metadata and the structured business data and send them to downstream application consumption.

[0081] This invention deserializes the metadata of the original CDC event during the streaming computing platform processing; performs semantic recognition and transformation based on the metadata of the original CDC, and deserializes the business data of the original CDC data to obtain structured business data. Through the structured business data and semantic transformation metadata, a unified standardized RowData event is output, thereby unifying the semantic transformation processing scattered across various downstream applications into the streaming computing platform. This avoids multiple downstream applications repeatedly performing the same semantic transformation, reduces redundant code development and maintenance costs, and reduces the amount of data processed by downstream applications during operation, improving processing efficiency and achieving unified governance of the streaming computing platform.

[0082] Optionally, module 302 is determined, specifically for:

[0083] If the operation type field value of the structured metadata is a first string, or the business primary key field value of the structured metadata is a negative value, then the original CDC event is determined to be a deletion semantic event; the first string is a string representing the physical deletion type.

[0084] Optionally, the device further includes:

[0085] The first deletion type determination module is used to determine the event type of the original CDC event as the first deletion type if the operation type field value of the structured metadata is a first string and the business primary key field value of the structured metadata is not a negative value.

[0086] The second deletion type determination module is used to determine the event type of the original CDC event as the second deletion type if the value of the operation type field of the structured metadata is not the first string and the value of the business primary key field of the structured metadata is a negative value.

[0087] The third type determination module is used to determine the event type of the original CDC event as the third deletion type if the operation type field value of the structured metadata is a first string and the business primary key field value of the structured metadata is a negative value.

[0088] Optionally, the semantic conversion module 303 includes:

[0089] The first writing unit is configured to write a logical deletion identifier field into the structured metadata if the event type of the structured metadata is a first deletion type, and set the value of the logical deletion identifier field to a deletion status string.

[0090] The first modification unit is used to change the value of the operation type field in the structured metadata to a second string; the second string is a string representing the update type.

[0091] The first semantic conversion unit is used to perform type conversion on each field in the structured metadata to obtain semantically converted metadata.

[0092] Optionally, the semantic conversion module 303 includes:

[0093] The second writing unit is used to write a logical deletion identifier field into the structured metadata if the event type of the structured metadata is the second deletion type, and set the value of the logical deletion identifier field to a deletion status string.

[0094] The second modification unit is used to change the value of the business primary key field in the structured metadata to a positive number type numeric value;

[0095] The second conversion unit is used to perform type conversion on each field in the structured metadata to obtain semantically converted metadata.

[0096] Optionally, the device may also include:

[0097] The monitoring module is used to detect anomalies in the converted fields during the semantic transformation process of the structured metadata and to determine the anomaly type of the abnormal fields.

[0098] The processing module is used to query the processing type of the exception field according to the conversion exception type, and execute the processing operation corresponding to the processing type; the processing type includes alarm type, empty type, repair type and discard type.

[0099] The data processing apparatus provided in the embodiments of the present invention can execute the data processing method in the streaming computing scenario provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects for executing the data processing method in the streaming computing scenario.

[0100] Example 4

[0101] Figure 4A schematic diagram of a data processing apparatus 410 that can be used to implement embodiments of the present invention is shown. The data processing apparatus is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The data processing apparatus can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices (e.g., helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.

[0102] like Figure 4 As shown, the data processing device 410 includes at least one processor 411 and a memory, such as a read-only memory (ROM) 412 or a random access memory (RAM) 413, communicatively connected to the at least one processor 411. The memory stores computer programs executable by the at least one processor. The processor 411 can perform various appropriate actions and processes based on the computer program stored in the ROM 412 or loaded from storage unit 418 into the RAM 413. The RAM 413 may also store various programs and data required for the operation of the data processing device 410. The processor 411, ROM 412, and RAM 413 are interconnected via a bus 414. An input / output (I / O) interface 415 is also connected to the bus 414.

[0103] Multiple components in the data processing device 410 are connected to the I / O interface 415, including: an input unit 416, such as a keyboard, mouse, etc.; an output unit 417, such as various types of displays, speakers, etc.; a storage unit 418, such as a disk, optical disk, etc.; and a communication unit 419, such as a network card, modem, wireless transceiver, etc. The communication unit 419 allows the data processing device 410 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0104] Processor 411 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 411 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. Processor 411 performs the various methods and processes described above, such as data processing methods in streaming computing scenarios.

[0105] In some embodiments, the data processing method in a streaming computing scenario can be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 418. In some embodiments, part or all of the computer program can be loaded and / or installed on the data processing device 410 via ROM 412 and / or communication unit 419. When the computer program is loaded into RAM 413 and executed by processor 411, one or more steps of the data processing method in the streaming computing scenario described above can be performed. Alternatively, in other embodiments, processor 411 can be configured to perform the data processing method in the streaming computing scenario by any other suitable means (e.g., by means of firmware).

[0106] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations may include: implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0107] Computer programs used to implement the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be performed. The computer programs may be executed entirely on a machine, partially on a machine, or as a standalone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0108] In the context of this invention, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0109] To provide interaction with a user, the systems and techniques described herein can be implemented on a data processing device having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the data processing device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0110] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or middleware components (e.g., application servers), or frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0111] A computing system can include clients and servers. Clients and servers are generally geographically separated and typically interact via communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system. It addresses the shortcomings of traditional physical hosts and VPS (Virtual Private Server) services, such as high management difficulty and weak business scalability.

[0112] It should be understood that the various forms of processes shown above can be used to reorder, add, or delete steps. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and this is not limited herein.

[0113] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various corrections, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A data processing method in a streaming computing scenario, characterized in that, Executed by a streaming computing platform, the method includes: The metadata of the original CDC event sent from the upstream database is deserialized to obtain the structured metadata of the original CDC event; Based on the operation type field value and business primary key field value of the structured metadata, determine whether the original CDC event is a deletion semantic event; If the original CDC event is a deletion semantic event, then the structured metadata is semantically transformed according to the event type of the structured metadata to obtain semantically transformed metadata; The original CDC event's business data is deserialized to obtain structured business data; Based on the semantic transformation metadata and the structured business data, a RowData event is generated and sent to downstream applications for consumption.

2. The method according to claim 1, characterized in that, The step of determining whether the original CDC event is a deletion semantic event based on the operation type field value and business primary key field value of the structured metadata includes: If the operation type field value of the structured metadata is a first string, or the business primary key field value of the structured metadata is a negative value, then the original CDC event is determined to be a deletion semantic event; the first string is a string representing the physical deletion type.

3. The method according to claim 2, characterized in that, After determining that the original CDC event is a deletion semantic event, the process also includes: If the operation type field value of the structured metadata is a first string, and the business primary key field value of the structured metadata is not a negative value, then the event type of the original CDC event is determined to be the first deletion type. If the operation type field value of the structured metadata is not the first string, and the business primary key field value of the structured metadata is a negative value, then the event type of the original CDC event is determined to be the second deletion type. If the operation type field value of the structured metadata is a first string and the business primary key field value of the structured metadata is a negative value, then the event type of the original CDC event is determined to be the third deletion type.

4. The method according to claim 3, characterized in that, The step of performing semantic transformation processing on the structured metadata based on the event type to obtain semantically transformed metadata includes: If the event type of the structured metadata is the first deletion type, then write the logical deletion identifier field into the structured metadata and set the value of the logical deletion identifier field to the deletion status string; Change the value of the operation type field in the structured metadata to the second string; the second string is a string representing the update type; For each field in the structured metadata, the field is type-converted to obtain semantically converted metadata.

5. The method according to claim 3, characterized in that, The step of performing semantic transformation processing on the structured metadata based on the event type to obtain semantically transformed metadata includes: If the event type of the structured metadata is the second deletion type, then write the logical deletion identifier field into the structured metadata and set the value of the logical deletion identifier field to the deletion status string; Change the value of the business primary key field in the structured metadata to a positive number. For each field in the structured metadata, the field is type-converted to obtain semantically converted metadata.

6. The method according to any one of claims 1-5, characterized in that, In addition to performing semantic transformation processing on the structured metadata according to the event type to obtain semantically transformed metadata, the process also includes: During the semantic transformation of the structured metadata, anomaly detection is performed on the transformed fields, and the transformation anomaly type of the abnormal fields is determined. Based on the transformation exception type, query the processing type of the exception field and execute the processing operation corresponding to the processing type; the processing type includes alarm type, empty type, repair type and discard type.

7. A data processing apparatus, characterized in that, Configured on a streaming computing platform, the device includes: The first deserialization module is used to deserialize the metadata of the original CDC event sent by the upstream database to obtain the structured metadata of the original CDC event. The determination module is used to determine whether the original CDC event is a deletion semantic event based on the operation type field value and business primary key field value of the structured metadata; The semantic conversion module is used to perform semantic conversion processing on the structured metadata according to the event type of the original CDC event if the original CDC event is a deletion semantic event, so as to obtain semantically converted metadata. The second deserialization module is used to deserialize the business data of the original CDC event to obtain structured business data. The event generation module is used to generate RowData events based on the semantic transformation metadata and the structured business data, and send them to downstream applications for consumption.

8. An electronic device, characterized in that, The electronic device includes: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the data processing method in the streaming computing scenario according to any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that are used to cause a processor to execute the data processing method in the streaming computing scenario according to any one of claims 1-6.

10. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the data processing method in the streaming computing scenario as described in any one of claims 1-6.

Images