A binlog synchronization method and system supporting OceanBase unique data type access Chunjun

Abstract

The application relates to a Binlog synchronization method and system supporting OceanBase unique data type access Chunjun. The method comprises the following steps: receiving a Binlog log of an OceanBase database, obtaining Binlog events and corresponding Schema information therefrom; distinguishing native data types from OceanBase unique data types, and parsing the unique data types by a Parser Factory; converting the parsing result into a downstream compatible format, assembling all fields into RowData, and sending the RowData to a downstream system after processing. According to the scheme, the Chunjun Binlog Connector is deeply expanded to actively parse OceanBase unique data types, end-to-end full-type lossless synchronization of OceanBase unique data types can be realized without external tools, and the efficiency and conversion accuracy of data synchronization are improved.

Description

Technical Field

[0001] This application relates to the field of database synchronization and data migration technology, and in particular to a method, system, computer-readable storage medium, and electronic device that supports the access of OceanBase-specific data types to Chunjun's Binlog synchronization. Background Technology

[0002] In the field of real-time data synchronization technology, mainstream real-time data synchronization tools (such as Chunjun) typically adopt the technical approach of parsing database binary logs (Binlog) to achieve real-time capture and cross-system synchronization of database data changes. For MySQL databases, Chunjun provides a mature chunjun-connector-binlog plugin, which can efficiently parse the native MySQL Binlog format and fully support the synchronization processing of native MySQL data types. Its technical maturity has been widely verified through application.

[0003] OceanBase, a distributed relational database, provides the OBProxy (OceanBase Proxy) component to adapt to the MySQL ecosystem and reduce user migration costs. As a MySQL protocol compatibility layer for OceanBase, OBProxy exposes the OceanBase database service in the form of the MySQL protocol, enabling mainstream MySQL clients and tools (including Chunjun) to access OceanBase through standard MySQL connections and supporting the consumption of OceanBase's changed data via the MySQL Binlog protocol. However, OBProxy's simulation of the MySQL Binlog protocol has limitations: although it can convert OceanBase's internal change logs into a data stream conforming to the MySQL Binlog protocol format, it cannot fully map OceanBase's unique data type information.

[0004] Specifically, OceanBase, as a distributed relational database, extends the capabilities of MySQL's native data types in its data type design, forming several unique data types that MySQL does not possess, including but not limited to: (1) JSON type: OceanBase's JSON type has more powerful capabilities in data validation, index support and complex query processing, and has functional differences from MySQL's JSON type; (2) ARRAY type: Supports storing ordered arrays of homogeneous or heterogeneous elements, and provides an efficient array operation interface; (3) STRUCT / RECORD type: Supports storing structured data with named fields, and can flexibly represent multi-attribute related information; (4) BIT type: It has specific bit operation characteristics and storage optimization, and differs from MySQL's BIT type in bit width range and operation semantics; (5) DECIMAL type (precision and range): There may be differences in numerical precision and value range compared to MySQL (especially in cross-regional deployment scenarios), and more fine-grained numerical features need to be retained; (6) Specific encoding of TEXT / BLOB type: Although both OceanBase and MySQL support large text / binary data, OceanBase has specific characteristics in its internal storage encoding (such as compression algorithm, character set mapping) and read / write processing logic; (7) User-defined types (UDT): As OceanBase functionality evolves, it may support more user-defined extended data types in the future.

[0005] When the chunjun-connector-binlog plugin connects to OceanBase via OBProxy and attempts to parse its exposed Binlog, the plugin only has built-in parsing logic for native MySQL data types and cannot recognize and process the OceanBase-specific data types mentioned above, leading to the following key issues: 1. Data parsing error: The plugin cannot match the format definition of the specific data type, which may trigger a parsing exception and directly cause the synchronization task to be interrupted.

[0006] 2. Data loss: For unique data types that cannot be recognized, the plugin may ignore or truncate related fields by default, resulting in incomplete data after synchronization.

[0007] 3. Data distortion: Even if the plugin attempts to convert the unique type to a MySQL-compatible general type (such as converting ARRAY to VARCHAR), the difference in type semantics will lead to problems such as loss of numerical precision and destruction of structural information, making it impossible to restore the original OceanBase data.

[0008] 4. Precise synchronization failure: After receiving incorrect, incomplete or distorted data, downstream systems are unable to perform effective business analysis, data governance or transaction consistency assurance based on synchronized data.

[0009] In summary, the existing chunjun-connector-binlog plugin, when adapted to OceanBase database, lacks the ability to identify and process OceanBase-specific data types, thus failing to achieve complete and accurate capture and synchronization of data changes, significantly limiting the application of Chunjun in the OceanBase ecosystem. Summary of the Invention

[0010] To address the aforementioned problems in existing technologies, this application proposes a new method and system for supporting the access of OceanBase-specific data types to Chunjun's Binlog synchronization.

[0011] The core of this invention lies in the deep customization and extension of Chunjun's chunjun-connector-binlog plugin, enabling it to understand, parse, and correctly process the OceanBase-specific data types contained in the Binlog stream exposed by OBProxy.

[0012] Specifically, this invention achieves the following key functions: 1. OceanBase's unique data type recognition and mapping mechanism During the connection initialization phase of the Chunjun plugin, the schema information of the target OceanBase table is obtained through OBProxy.

[0013] Establish a data type mapping table that can map data types to intermediate representations within the Chunjun plugin based on OceanBase-specific data type identifiers provided by OBProxy (e.g., custom type strings or specific numeric encodings), or directly identify their special characteristics.

[0014] 2. Enhanced Binlog Event Parser General parser basics: Inherits and reuses Chunjun's original general parsing capabilities for Binlog events, and handles native MySQL data types.

[0015] Unique data type detection and parsing: When parsing data in a field of a Binlog event, the plugin queries the field's type information. If it is identified as an OceanBase-specific data type (such as JSON, ARRAY, STRUCT, BIT, etc.), it invokes a parser specifically designed for that type. These type-specific parsers can accurately extract the original value based on the specific encoding format (which may be binary or text representation) of the data type in the OceanBase Binlog.

[0016] 3. Data type conversion and serialization module The goal of generalized transformation is to convert values ​​of OceanBase-specific data types into general data structures that Chunjun can process internally, or into formats that downstream systems (such as Kafka, Flink, and other databases) can easily receive and parse.

[0017] Flexible conversion strategies: (1) JSON type: Parse OceanBase JSON data into a structured JSON object (e.g., represented as a JSON Node in Java), which can then be serialized into a UTF-8 encoded JSON string, or mapped to a predefined JSON field in the Schema, depending on downstream requirements.

[0018] (2) ARRAY type: Parse OceanBase ARRAY data into a List (or equivalent structure), and serialize it into a JSON array string as needed, or pass it as a List structure.

[0019] (3) STRUCT / RECORD type: Parse OceanBase STRUCT / RECORD data into a Map (or equivalent structure), where the Key is the field name and the Value is the field value, and then it can be serialized into a JSON object string.

[0020] (4) BIT type: Convert it into a byte array, Long type or other binary representation according to downstream requirements.

[0021] (5) DECIMAL type (precision and range): Ensure that the precision and range of OceanBase DECIMAL type are fully preserved during parsing and conversion. It is usually converted to a high-precision numeric type (such as Java BigDecimal) or a string.

[0022] Configurability: Allows users to select specific data type conversion methods (e.g., whether JSON output is a string or a structured field) through Chunjun task configuration parameters.

[0023] 4. Unified data output and formatting: The converted special data type values, along with other synchronized ordinary data type values, are integrated into Chunjun's standard data model (such as RowData).

[0024] Based on the configuration of the Chunjun Sink Connector, the final data is sent to the downstream data consumer in the target format.

[0025] To achieve the above objectives, the present invention employs the following technical strategies: 1. Modify the Chunjun Connector Binlog plugin to enable it to actively parse data. Instead of passively accepting Binlog data, it can actively identify, understand, and parse the internal structure of OceanBase's unique data types. This achieves native and deep adaptation and support at the Chunjun Binlog Connector level for the first time, rather than relying on external tools or indirect conversion.

[0026] 2. OBProxy Binlog's "Intelligent Secondary Processing": This invention fully utilizes OBProxy's basic capabilities in simulating the MySQL Binlog protocol, but goes further by using the Chunjun plugin for "secondary processing" to compensate for OBProxy's shortcomings in exposing specific types of information (or its limitations on downstream applications). This solution does not modify OBProxy, but rather enhances the parsing capabilities of downstream consumers to achieve more refined data processing. Simultaneously, it forms an efficient "OBProxy-Chunjun Adapter" model, enabling OBProxy's Binlog to more effectively serve Chunjun, thereby supporting more complex database synchronization scenarios.

[0027] 3. Construct a unified, scalable, interface-based framework for handling OceanBase-specific data types. This framework includes modules such as type identification, dedicated parsers, and general-purpose converters. This framework design offers excellent maintainability and scalability, allowing for the easy integration of any new OceanBase-specific data types that may be added in the future. Only the corresponding parsers and converters need to be implemented, without requiring large-scale modifications to the Chunjun core code.

[0028] 4. Schema-driven Binlog parsing: When parsing the Binlog, the plugin not only relies on the Binlog events themselves but also actively utilizes schema information to guide the parsing and conversion of specific types. Simultaneously, a polymorphic type conversion mechanism is established, employing object-oriented design principles to implement different conversion logics for different OceanBase-specific types. Dynamic invocation is achieved through a registration mechanism, improving code flexibility and maintainability. Specifically, this application provides the following technical solutions: The first aspect of this application provides a method for supporting the access of OceanBase-specific data types to Chunjun's Binlog synchronization, the method comprising: S1. Data source access: Receive Binlog logs from OceanBase database through OBProxy, and ChunjunBinlog Connector obtains Binlog events and their corresponding schema information from OBProxy; S2. Type Identification and Parsing Scheduling: The Binlog event and Schema information are transmitted to the Chunjun internal processing module. The Type Identifier module identifies the data type of the field in the Binlog event. If it is a native MySQL data type, Chunjun's general parsing logic is called for processing. If it is a unique OceanBase data type, the data of the field is scheduled to the Parser Factory. S3. Specific type parsing and format conversion: The Parser Factory parses the raw data of OceanBase's specific data types and transmits the parsing results to the ConverterFactory. The ConverterFactory converts the parsed data into a format compatible with downstream systems. All fields (including the converted specific data type fields) are assembled into RowData of the Chunjun standard data model. S4. Data forwarding: The RowData is transmitted to Chunjun Core for core logic processing, and the SinkConnector sends the processed data to the downstream system according to the configuration.

[0029] Furthermore, in the method of this application, step S1 further includes: during the Chunjun Binlog Connector connection initialization phase, obtaining the schema information of the target OceanBase table through OBProxy, establishing a data type mapping table, and mapping the OceanBase-specific data type identifiers provided by OBProxy to Chunjun internal intermediate representations or directly marking specific types; the Type Identifier module identifies the field data type based on the data type mapping table.

[0030] Furthermore, in the method of this application, the Parser Factory includes dedicated parsers for different OceanBase-specific data types, and the dedicated parsers include: A JSON parser is used to parse the binary or text representation of OceanBase JSON data into a JSON object; An ARRAY parser is used to parse OceanBase ARRAY data into a list structure. The STRUCT parser is used to parse OceanBase STRUCT data into a key-value pair mapping structure. The BIT parser is used to parse OceanBase BIT data into byte arrays or long integers. The DECIMAL parser is used to convert OceanBase DECIMAL data into a high-precision numeric type.

[0031] Furthermore, in the method of this application, the ConverterFactory includes converters configured for different OceanBase-specific data types; the converters support selecting the conversion method through Chunjun task configuration parameters, and the conversion methods include: Convert JSON data types to strings or structured fields; ARRAY types are serialized as JSON array strings or retain the list structure; Serialize the STRUCT type into a JSON object string; BIT type is converted to byte array or long integer; DECIMAL type can be converted to string or high-precision numeric type.

[0032] A second aspect of this application provides a Binlog synchronization system that supports OceanBase-specific data types for accessing Chunjun's Binlog, the system comprising: The data source access module is used to receive Binlog logs from the OceanBase database through OBProxy. ChunjunBinlog Connector obtains Binlog events and their corresponding schema information from OBProxy. The type identification and parsing scheduling module is used to transmit the Binlog event and Schema information to the Chunjun internal processing module. The Type Identifier module identifies the data type of the field in the Binlog event. If it is a native MySQL data type, Chunjun's general parsing logic is called for processing. If it is a unique OceanBase data type, the data of the field is scheduled to the Parser Factory. The unique type parsing and format conversion module is used to parse the original data of OceanBase's unique data types through Parser Factory, transmit the parsing results to ConverterFactory, and ConverterFactory converts the parsed data into a format compatible with downstream systems. All fields (including the converted unique data type fields) are assembled into RowData of the Chunjun standard data model. The data forwarding module is used to transmit the RowData to Chunjun Core for core logic processing, and the Sink Connector sends the processed data to the downstream system according to the configuration.

[0033] The system implements the aforementioned steps of supporting the access of OceanBase-specific data types to Chunjun's Binlog synchronization method during runtime.

[0034] Furthermore, the system in this application also includes: The type mapping module is used during the Chunjun Binlog Connector connection initialization phase to obtain the schema information of the target OceanBase table through OBProxy, establish a data type mapping table, and map the OceanBase-specific data type identifiers provided by OBProxy to Chunjun internal intermediate representations or directly mark the specific types; the Type Identifier module identifies the field data type based on the data type mapping table.

[0035] Furthermore, in the system of this application, the Parser Factory includes dedicated parsers for different OceanBase-specific data types, and the dedicated parsers include: A JSON parser is used to parse the binary or text representation of OceanBase JSON data into a JSON object; An ARRAY parser is used to parse OceanBase ARRAY data into a list structure. The STRUCT parser is used to parse OceanBase STRUCT data into a key-value pair mapping structure. The BIT parser is used to parse OceanBase BIT data into byte arrays or long integers. The DECIMAL parser is used to convert OceanBase DECIMAL data into a high-precision numeric type.

[0036] Furthermore, in this application's system, the ConverterFactory includes converters configured for different OceanBase-specific data types; the converters support selecting conversion methods through Chunjun task configuration parameters, and the conversion methods include: Convert JSON data types to strings or structured fields; ARRAY types are serialized as JSON array strings or retain the list structure; Serialize the STRUCT type into a JSON object string; BIT type is converted to byte array or long integer; DECIMAL type can be converted to string or high-precision numeric type.

[0037] A third aspect of this application provides an electronic device, including: a memory and a processor; Memory: Used to store computer programs; Processor: Used to execute the computer program to implement the aforementioned steps of supporting the access of OceanBase-specific data types to Chunjun's Binlog synchronization method.

[0038] A fourth aspect of this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the aforementioned steps of the Binlog synchronization method for supporting OceanBase-specific data types accessed by Chunjun.

[0039] In summary, this invention provides a systematic solution for the synchronization requirements of OceanBase's unique data types through targeted modifications to the Chunjun Connector Binlog. Its core advantages are reflected in the following aspects: 1. Achieve end-to-end lossless and complete synchronization of OceanBase-specific data types. This solution overcomes the limitations of traditional Binlog synchronization tools in supporting OceanBase's unique data types. By extending the type parsing module of Chunjun Connector Binlog, it enables accurate identification and processing of OceanBase's unique data types, such as JSON, ARRAY, STRUCT / RECORD, BIT, high-precision DECIMAL, and specifically encoded TEXT / BLOB. Specifically: Full type coverage: Supports the capture and parsing of all OceanBase-specific data types, eliminating the "data barrier" caused by type incompatibility in traditional solutions, and ensuring that the synchronization process covers all data fields of the OceanBase database; Lossless fidelity: Through customized type conversion rules and semantic preservation mechanisms, the accuracy loss, structural damage, or semantic distortion of data during the synchronization process is avoided. The data received by the downstream system and the original data of the source OceanBase are highly consistent in terms of type definition, numerical precision, and business semantics.

[0040] 2. Significantly enhance Chunjun's application value and coverage within the OceanBase ecosystem. This solution significantly expands Chunjun's applicability in hybrid data ecosystems by addressing the synchronization challenges of OceanBase's unique data types. Enables real-time data integration: Chunjun serves as a reliable bridge for real-time data synchronization between OceanBase and big data platforms (such as data lakes, data warehouses, and real-time analysis systems) and business systems (such as financial transactions and IoT data platforms), meeting the needs of scenarios with extremely high requirements for data integrity and consistency (such as real-time risk control, transaction reconciliation, and log chain tracking). Reduced user integration costs: Users do not need to develop customized extraction, transformation or adaptation logic for OceanBase-specific data types. They can simply deploy synchronization tasks through Chunjun's standard configuration interface, significantly reducing the technical threshold and development and maintenance costs.

[0041] 3. Enhance the depth and flexibility of downstream data processing and analysis. The OceanBase data synchronized to downstream systems using this solution fully retains the semantic information of its unique data types, providing richer input dimensions for downstream data processing. Expand data analysis capabilities: Downstream systems (such as Flink, Spark, and ClickHouse) can directly obtain the original type information of structured and semi-structured data (such as JSON objects and ARRAY arrays), and support complex queries (such as JSON field filtering and array element aggregation) and in-depth analysis (such as nested structure statistics and multidimensional association mining) based on specific types. Supporting diverse data applications: Databases or analytics platforms can implement more flexible data applications based on the parsed unique types. For example, they can quickly access related fields through the STRUCT type and directly process batch data through the ARRAY type, avoiding data flattening and information loss caused by type conversion.

[0042] 4. Improve data synchronization efficiency and system robustness This solution achieves differentiated advantages in synchronization efficiency and system reliability through native plugin integration and optimization. High performance and low latency: The parsing and conversion of unique data types are completed directly at the Chunjun plugin level, avoiding the extra overhead of secondary conversion through middleware or external services in traditional solutions, significantly reducing synchronization latency and improving real-time performance; Simplified architecture reduces risk: No need to introduce independent conversion middleware, reducing redundant links in the system architecture, reducing the risk of synchronization interruption caused by middleware failure, data format incompatibility and other issues, and enhancing the overall system stability and maintainability.

[0043] 5. Accelerate the deep integration of OceanBase with multiple data ecosystems. This solution addresses the core pain points of data synchronization, promoting interoperability between OceanBase and other data technology systems: Breaking down data silos: Providing OceanBase with a smooth data outflow channel, enabling it to seamlessly integrate into the enterprise's existing data platform, data lake warehouse, microservice architecture, and other ecosystems, achieving cross-system data sharing and collaboration; Unleash the potential value of data: Through complete and accurate synchronization capabilities, OceanBase’s unique data types (such as high-precision DECIMAL financial data and structured JSON logs) can be efficiently utilized by downstream systems, accelerating the implementation of business insights and innovative applications (such as real-time report generation and intelligent model training).

[0044] In summary, this invention achieves high-quality synchronization of OceanBase's unique data types through technical optimization, which not only enhances Chunjun's competitiveness in the hybrid data ecosystem, but also promotes the deep integration of OceanBase with enterprise data infrastructure, providing more reliable technical support for the digital transformation of various industries.

[0045] Other features and advantages of this application will be set forth in detail in the following description, or will become apparent through the implementation of the relevant technical solutions of this application. The objectives and other advantages of this application can be achieved through the technical features and means explicitly pointed out in the description, claims, and drawings, and will be obtained through the implementation of these technical contents. Attached Figure Description

[0046] To more clearly illustrate the technical solutions of this application, the accompanying drawings involved in the description of the embodiments will be briefly introduced below. It should be noted that the drawings only show some embodiments of this application. For those skilled in the art, other related drawings can be derived from these drawings without creative effort.

[0047] Figure 1 This is a flowchart illustrating the implementation of the method for integrating OceanBase-specific data types into Chunjun's Binlog synchronization in this application.

[0048] Figure 2 This is a diagram illustrating the overall design architecture and technical flow of the embodiments of this application.

[0049] Figure 3 This is a structural diagram illustrating the components of the Binlog synchronization system in Chunjun that support OceanBase-specific data types in this application.

[0050] Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0051] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. It should be noted that the described embodiments are only some embodiments of this application, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the protection scope of this application.

[0052] In this document, the term "comprising" and any variations thereof (such as "including," "including," etc.) are open-ended expressions and should be understood as "including but not limited to," meaning that the listed content is not exhaustive and may include other content not explicitly mentioned. The term "based on" should be understood as "at least partially based on," meaning that the basis or condition referred to may not be the only factor and may involve other relevant factors. The term "one embodiment" should be understood as "at least one embodiment," meaning that the described embodiment is not the only possible implementation, and other similar embodiments may exist.

[0053] In this application, the terms "a" and "a plurality of" are used to modify related elements or features, and their expression is illustrative rather than restrictive. Unless otherwise expressly stated in the context, "a" should be understood as "at least one," and "a plurality of" should be understood as "at least two." Those skilled in the art should reasonably interpret these terms based on the semantic and logical relationships of the context to ensure that they cover the possibility of "one or more."

[0054] Figure 1 The following is an implementation flow of the Binlog synchronization method for OceanBase-specific data types provided in this application, including the following steps: S1. Data source access: Receive Binlog logs from OceanBase database through OBProxy, and ChunjunBinlog Connector obtains Binlog events and their corresponding schema information from OBProxy; S2. Type Identification and Parsing Scheduling: The Binlog event and Schema information are transmitted to the Chunjun internal processing module. The Type Identifier module identifies the data type of the field in the Binlog event. If it is a native MySQL data type, Chunjun's general parsing logic is called for processing. If it is a unique OceanBase data type, the data of the field is scheduled to the Parser Factory. S3. Specific type parsing and format conversion: The Parser Factory parses the raw data of OceanBase's specific data types and transmits the parsing results to the ConverterFactory. The ConverterFactory converts the parsed data into a format compatible with downstream systems. All fields (including the converted specific data type fields) are assembled into RowData of the Chunjun standard data model. S4. Data forwarding: The RowData is transmitted to Chunjun Core for core logic processing, and the SinkConnector sends the processed data to the downstream system according to the configuration.

[0055] To more clearly illustrate the technical solution of this application, the following will provide further explanation through specific scenario embodiments.

[0056] The specific implementation process of this invention mainly focuses on the modification of the Chunjun Binlog Connector plugin and its collaborative cooperation with OBProxy.

[0057] 1. System Architecture and Components OceanBase Database: Source database, with Binlog enabled.

[0058] OBProxy: OceanBase's MySQL protocol compatibility layer, responsible for exposing OceanBase Binlog to downstream applications.

[0059] Chunjun Framework: Real-time data integration platform.

[0060] Chunjun Binlog Connector: A modified Chunjun Binlog connector plugin.

[0061] Downstream Systems: Target data consumers, such as Kafka, Flink, HDFS, and other databases.

[0062] 2. Implementation process Figure 2 The diagram shows the overall design architecture and technical flow of the present invention, which can be divided into three parts according to data flow and processing stages: 1. Data source access phase to Chunjun The data flow begins in the OceanBase database, where changed data is stored in the form of Binlog logs. OBProxy, acting as a MySQL protocol compatibility layer for OceanBase, exposes the Binlog logs in MySQL protocol format and forwards them to the Chunjun Binlog Connector. The Chunjun Binlog Connector receives Binlog events and corresponding schema information, completing the initial data access.

[0063] 2. Chunjun's internal processing phase The Chunjun Binlog Connector passes the received Binlog events and schema information to the internal processing module, which then performs the following processing steps: (a) Type identification and parsing: The Type Identifier module identifies the type of a field. If the field is a native MySQL type, Chunjun's original parsing logic is used; if the field is a OceanBase-specific type (such as JSON, ARRAY, STRUCT, etc.), it is handled by the Parser Factory.

[0064] (b) Data transformation and assembly: The Parser Factory passes the parsed results to the Converter Factory, which converts the parsed field values ​​into a format compatible with downstream systems (such as RowData). All fields (including converted fields of specific types) are assembled into a unified RowData data structure.

[0065] 3. Chunjun's forwarding stage to downstream The assembled RowData is transmitted to Chunjun Core for core logic processing. Chunjun Core calls SinkConnector according to the configuration to write the data to downstream target systems (such as data lakes, data warehouses, or real-time analytics platforms), completing end-to-end data synchronization.

[0066] The entire process enables end-to-end data synchronization from OceanBase data source to downstream systems, and supports precise parsing and conversion of OceanBase-specific data types.

[0067] More specifically, it includes the following processes: (1) Environment deployment and configuration Deploy an OceanBase cluster and ensure that the Binlog function is enabled.

[0068] Deploy OBProxy and configure it to connect to the OceanBase cluster, while listening to and forwarding OceanBase's Binlog streams.

[0069] Deploy the Chunjun framework and install the Chunjun Binlog Connector plugin.

[0070] (2) OBProxy Binlog Information Extraction Chunjun Binlog Connector connects to OceanBase via OBProxy.

[0071] After the connection is established, Chunjun first performs a schema synchronization operation to obtain detailed schema information of the target OceanBase table through OBProxy, which includes key metadata such as the table's field names, raw data types, precision, and length.

[0072] Key technical point: The Binlog events exposed by OBProxy need to include information sufficient to distinguish OceanBase-specific types when describing field types. For example, for JSON types, OBProxy may mark a specific type ID or type name string (such as "OB_JSON", "OB_ARRAY", etc.) in the Binlog header or payload.

[0073] (3) Chunjun Binlog Connector core modification a) Type Identifier Dictionary: Internally, the Chunjun plugin maintains a global dictionary of type identifiers. This dictionary maps OceanBase-specific data type identifiers exposed by OBProxy (such as the strings "JSON", "ARRAY", and "STRUCT") to an internal enumeration value or a specific processing identifier.

[0074] For example: Map<String, InternalType> OB_TYPE_MAP = new HashMap<>(); OB_TYPE_MAP.put("JSON", InternalType.OB_JSON); OB_TYPE_MAP.put("ARRAY",InternalType.OB_ARRAY); b) Schema metadata parsing and unique type tagging: When Chunjun Binlog Connector parses the table schema obtained from OBProxy, it iterates through all fields.

[0075] For each field, look up OB_TYPE_MAP based on its type description string (obtained from OBProxy).

[0076] If a match is found, the field is marked as an OceanBase-specific type, and its corresponding internal processing identifier is recorded (e.g., field.setOceanBaseType(InternalType.OB_JSON)).

[0077] c) Enhanced Binlog event data parsing: Binlog Event Reception and Parser Distribution: Chunjun Binlog Connector receives Binlog events from OBProxy.

[0078] Field-by-field parsing: For each modified field in the Binlog event, the following processing is performed based on its pre-marked (or dynamically queried at runtime) type information: Raw type handling: If a field is marked as a MySQL raw type, it will be processed using Chunjun's original general parser.

[0079] Special type resolution: If a field is marked as an OceanBase-specific type (such as OB_JSON), the data for that field (which may be a byte array or a specially encoded string) is passed to a dedicated special type resolver, for example: JSON parser: Receives binary / text representations of JSON data and parses them into JSON objects using JSON libraries (such as Jackson and Gson).

[0080] ARRAY parser: Parses the array structure, identifies the element type and quantity, and parses it into a List.

[0081] STRUCT parser: Parses struct fields and converts them into Maps.<String, Object> .

[0082] BIT parser: Based on the definition of OceanBase BIT, it parses it into byte[] or Long.

[0083] DECIMAL parser: processes the bytes representation of OceanBase DECIMAL and converts it precisely to BigDecimal.

[0084] d) Data conversion and serialization module: Converter registration: Register a converter with a unified interface for each OceanBase-specific type (identified by InternalType).

[0085] Dynamic converter invocation: After the specific type parser resolves the raw value, the plugin will invoke the corresponding converter based on the field's InternalType, for example: JSON Converter: Receives a JSON object and, depending on the Chunjun task configuration (e.g., json_output_format="STRING" or json_output_format="SCHEMA"), serializes it into a JSON string or attempts to map it to a predefined JSON field in Chunjun's Schema.

[0086] ARRAY converter: Receives a List and serializes it into a JSON array string.

[0087] STRUCT converter: Receives a Map and serializes it into a JSON object string.

[0088] BIT converter: Converts to Long or byte[] depending on the configuration.

[0089] DECIMAL converter: Converts to string or BigDecimal.

[0090] Result encapsulation: The converted data (usually a String, Long, BigDecimal, byte[], or String representation of a JSON object / array) will be used as field values ​​in Chunjun's RowData.

[0091] e) Data output and Sink interface: The Chunjun Binlog Connector passes RowData, which contains data of a specific type that has been transformed, to the Chunjun Sink Connector.

[0092] Sink Connectors (such as Kafka Sink) serialize RowData into the final output format (such as JSON string, AvroRecord, etc.) according to their own configuration and send it to the target system (Downstream Systems).

[0093] (4) Chunjun task configuration In Chunjun's JSON configuration file, the user specifies the OBProxy connection address, OceanBase database name, username, password, and target table.

[0094] New configuration items: You can add configuration parameters to control how specific data types are handled, for example: oceanbase_special_type_handling.json_as_string: true (defaults to serializing JSON as a string) oceanbase_special_type_handling.array_output_format: "JSON_ARRAY" oceanbase_special_type_handling.struct_output_format: "JSON_OBJECT".

[0095] (5) Data validation Perform INSERT / UPDATE / DELETE operations with specific data types in OceanBase.

[0096] Check the synchronized data in downstream systems (such as messages in Kafka Topic) to verify that types such as JSON, ARRAY, and STRUCT have been converted to strings or other formats as expected, and that the content is accurate.

[0097] Figure 3 The figure shown is a Binlog synchronization system for Chunjun that supports OceanBase-specific data types. The system includes: The data source access module is used to receive Binlog logs from the OceanBase database through OBProxy. ChunjunBinlog Connector obtains Binlog events and their corresponding schema information from OBProxy. The type identification and parsing scheduling module is used to transmit the Binlog event and Schema information to the Chunjun internal processing module. The Type Identifier module identifies the data type of the field in the Binlog event. If it is a native MySQL data type, Chunjun's general parsing logic is called for processing. If it is a unique OceanBase data type, the data of the field is scheduled to the Parser Factory. The unique type parsing and format conversion module is used to parse the original data of OceanBase's unique data types through Parser Factory, transmit the parsing results to ConverterFactory, and ConverterFactory converts the parsed data into a format compatible with downstream systems. All fields (including the converted unique data type fields) are assembled into RowData of the Chunjun standard data model. The data forwarding module is used to transmit the RowData to Chunjun Core for core logic processing, and the Sink Connector sends the processed data to the downstream system according to the configuration. The type mapping module is used during the Chunjun Binlog Connector connection initialization phase to obtain the schema information of the target OceanBase table through OBProxy, establish a data type mapping table, and map the OceanBase-specific data type identifiers provided by OBProxy to Chunjun internal intermediate representations or directly mark the specific types; the Type Identifier module identifies the field data type based on the data type mapping table.

[0098] The above system implements the steps disclosed in this application for supporting the access of OceanBase-specific data types to Chunjun's Binlog synchronization method during runtime.

[0099] The flowcharts and block diagrams in the accompanying drawings illustrate possible implementations of systems, methods, and computer program products according to various embodiments of this application, including architecture, functionality, and operation. In these figures, each block may represent a module, program segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should be noted that each block in the block diagrams and / or flowcharts, and combinations thereof, can be implemented using either a dedicated hardware-based system or a combination of dedicated hardware and computer instructions to achieve the specified function or operation.

[0100] like Figure 4 As shown, embodiments of this application also disclose an electronic device, including: a processor 310, a communication interface 320, a memory 330 for storing a processor-executable computer program, and a communication bus 340. The processor 310, communication interface 320, and memory 330 communicate with each other via the communication bus 340. The processor 310 executes the executable computer program to implement the steps described above for supporting the access of OceanBase-specific data types to Chunjun's Binlog synchronization method.

[0101] It is understood that, in addition to memory and a processor, this electronic device may also include input devices (such as a keyboard), output devices (such as a display), and other communication modules. These input devices, output devices, and other communication modules all communicate with the processor through I / O interfaces (i.e., input / output interfaces).

[0102] The operations described in this application can be implemented by writing computer program code using one or more programming languages ​​or a combination thereof. The programming languages ​​include, but are not limited to, the following types: Object-oriented programming languages, such as Java, Smalltalk, C++, etc. Conventional procedural programming languages, such as "C" or similar programming languages.

[0103] The execution methods of program code include, but are not limited to: It runs entirely on the user's computer; Part of it executes on the user's computer, and part of it executes on a remote computer; Execute as a standalone software package; It is executed entirely on a remote computer or server.

[0104] In scenarios involving remote computers, the remote computer can connect to the user computer via any type of network, including but not limited to a local area network (LAN) or a wide area network (WAN). Furthermore, the remote computer can also connect to external computers via an internet service provider, for example, by utilizing the internet for connection.

[0105] Furthermore, this application also discloses a computer-readable storage medium, which, when the instructions in the computer-readable storage medium are executed by the processor of an electronic device, enables the electronic device to perform the various steps of the Binlog synchronization method for accessing Chunjun with OceanBase-specific data types disclosed in this application.

[0106] In the context of this application, a computer-readable storage medium refers to a tangible medium capable of storing computer program code and related data. Specific examples include, but are not limited to, the following: (1) Portable computer disk: such as floppy disks and other removable magnetic storage media.

[0107] (2) Hard disk: including mechanical hard disks and solid-state hard disks and other fixed storage devices.

[0108] (3) Random Access Memory (RAM): A volatile storage medium used for temporary storage of data and program code.

[0109] (4) Read-only memory (ROM): a non-volatile storage medium used to store fixed programs and data.

[0110] (5) Erasable programmable read-only memory (EPROM) or flash memory: non-volatile storage media that supports multiple erasures and reprogrammings.

[0111] (6) Fiber optic storage devices: storage media based on fiber optic technology.

[0112] (7) Portable compact disc read-only memory (CD-ROM): a read-only medium that stores data in the form of an optical disc.

[0113] (8) Optical storage devices: such as DVDs, Blu-ray discs and other storage media based on optical principles.

[0114] (9) Magnetic storage devices: such as magnetic tapes, disks and other storage media based on magnetic principles.

[0115] (10) Any suitable combination of the above: for example, combining multiple storage media to meet different storage needs.

[0116] These computer-readable storage media can be used to store the program code and related data described in this application to support program execution and persistent data storage.

[0117] Specifically, according to embodiments of this application, the processes described in the flowcharts can be implemented as computer software programs. For example, embodiments of this application relate to a computer program product comprising a computer program carried on a non-transitory computer-readable medium. This computer program includes program code for executing the Binlog synchronization method disclosed in this application that supports OceanBase-specific data types for accessing Chunjun. When this computer program is executed by a processing system, it can achieve the functions defined in the embodiments of this application.

[0118] While the foregoing discussion contains several specific implementation details, these details should not be construed as limiting the scope of this application. The above description is merely a preferred embodiment of this application and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of this application is not limited to technical solutions formed by specific combinations of the above-described technical features. Furthermore, this application should also cover other technical solutions formed by any combination of the above-described technical features or their equivalents without departing from the foregoing disclosed concept.

[0119] Those skilled in the art should also understand that modifications can be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features, without departing from the spirit and scope of the technical solutions of the embodiments of this application. These modifications or substitutions will not cause the essence of the corresponding technical solutions to deviate from the core spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A method for synchronizing Chunjun's Binlog that supports OceanBase-specific data types, characterized in that, The method includes: S1. Data source access: Receive Binlog logs from OceanBase database through OBProxy, and Chunjun BinlogConnector obtains Binlog events and their corresponding schema information from OBProxy; S2. Type Identification and Parsing Scheduling: The Binlog event and Schema information are transmitted to the Chunjun internal processing module. The Type Identifier module identifies the data type of the field in the Binlog event. If it is a native MySQL data type, Chunjun's general parsing logic is called for processing. If it is a unique OceanBase data type, the data of the field is scheduled to the Parser Factory. S3. Specific type parsing and format conversion: The Parser Factory parses the raw data of OceanBase's specific data types and transmits the parsing results to the ConverterFactory. The ConverterFactory converts the parsed data into a format compatible with downstream systems, and all fields are assembled into RowData of the Chunjun standard data model. The Parser Factory contains dedicated parsers for different OceanBase-specific data types, and the dedicated parsers include: A JSON parser is used to parse the binary or text representation of OceanBase JSON data into a JSON object; An ARRAY parser is used to parse OceanBase ARRAY data into a list structure. The STRUCT parser is used to parse OceanBase STRUCT data into a key-value pair mapping structure. The BIT parser is used to parse OceanBase BIT data into byte arrays or long integers. The DECIMAL parser is used to convert OceanBase DECIMAL data into a high-precision numeric type. S4. Data forwarding: The RowData is transmitted to Chunjun Core for core logic processing, and the SinkConnector sends the processed data to the downstream system according to the configuration.

2. The method according to claim 1, characterized in that, Step S1 also includes: during the Chunjun BinlogConnector connection initialization phase, obtaining the schema information of the target OceanBase table through OBProxy, establishing a data type mapping table, and mapping the OceanBase-specific data type identifiers provided by OBProxy to Chunjun internal intermediate representations or directly marking specific types; the Type Identifier module identifies the field data type based on the data type mapping table.

3. The method according to claim 1, characterized in that, The ConverterFactory contains converters configured for different OceanBase-specific data types; the converters support selecting the conversion method through Chunjun task configuration parameters, and the conversion methods include: Convert JSON data types to strings or structured fields; ARRAY types are serialized as JSON array strings or retain the list structure; Serialize the STRUCT type into a JSON object string; BIT type can be converted to byte array or long integer; DECIMAL type can be converted to string or high-precision numeric type.

4. A Binlog synchronization system that supports OceanBase-specific data types for accessing Chunjun, characterized in that, The system includes: The data source access module is used to receive Binlog logs from the OceanBase database through OBProxy. ChunjunBinlog Connector obtains Binlog events and their corresponding schema information from OBProxy. The type identification and parsing scheduling module is used to transmit the Binlog event and Schema information to the Chunjun internal processing module. The Type Identifier module identifies the data type of the field in the Binlog event. If it is a native MySQL data type, Chunjun's general parsing logic is called for processing. If it is a unique OceanBase data type, the data of the field is scheduled to the Parser Factory. The unique type parsing and format conversion module is used to parse the raw data of OceanBase's unique data types through Parser Factory, transmit the parsing results to ConverterFactory, and ConverterFactory converts the parsed data into a format compatible with downstream systems. All fields are assembled into RowData of the Chunjun standard data model. The Parser Factory contains dedicated parsers for different OceanBase-specific data types, and the dedicated parsers include: A JSON parser is used to parse the binary or text representation of OceanBase JSON data into a JSON object; An ARRAY parser is used to parse OceanBase ARRAY data into a list structure. The STRUCT parser is used to parse OceanBase STRUCT data into a key-value pair mapping structure. The BIT parser is used to parse OceanBase BIT data into byte arrays or long integers. The DECIMAL parser is used to convert OceanBase DECIMAL data into a high-precision numeric type. The data forwarding module is used to transmit the RowData to Chunjun Core for core logic processing, and the SinkConnector sends the processed data to the downstream system according to the configuration.

5. The system according to claim 4, characterized in that, The system also includes: The type mapping module is used during the Chunjun Binlog Connector connection initialization phase to obtain the schema information of the target OceanBase table through OBProxy, establish a data type mapping table, and map the OceanBase-specific data type identifiers provided by OBProxy to Chunjun internal intermediate representations or directly mark the specific types; the Type Identifier module identifies the field data type based on the data type mapping table.

6. The system according to claim 4, characterized in that, The ConverterFactory contains converters configured for different OceanBase-specific data types; the converters support selecting the conversion method through Chunjun task configuration parameters, and the conversion methods include: Convert JSON data types to strings or structured fields; ARRAY types are serialized as JSON array strings or retain the list structure; Serialize the STRUCT type into a JSON object string; BIT type can be converted to byte array or long integer; DECIMAL type can be converted to string or high-precision numeric type.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the Binlog synchronization method for supporting OceanBase-specific data types accessed by Chunjun as described in any one of claims 1-3.

8. An electronic device, characterized in that, include: Memory and processor; Memory: Used to store computer programs; Processor: Used to execute the computer program to implement the steps of the Binlog synchronization method for supporting OceanBase-specific data types accessed by Chunjun as described in any one of claims 1-3.

Images