Real-time acquisition method based on cache database log and related equipment

Abstract

The application discloses a kind of real-time acquisition method and related equipment based on Cache database log.The method includes: real-time analysis Cache database's Journal change log and extract change operation record, including change type identification, encryption field identification and encryption field value;Escaping decoding is carried out to encryption field identification and encryption field value to obtain escaped field identification and escaped field value, and escaped field identification is mapped to actual field name;According to the data type identification corresponding to actual field name, type mark is carried out to escaped field value, generates structured change data and is transmitted to message queue;Structured change data in message queue is parsed to obtain database source information and table structure information, and is written into target storage system.The application realizes the streaming real-time acquisition of Cache database, supports single data instant processing, significantly reduces data delay, meets real-time business needs.

Description

Technical Field

[0001] This invention relates to the field of database technology, and in particular to a real-time acquisition method and related equipment based on cache database logs. Background Technology

[0002] With the rapid development of big data and real-time business, enterprises are increasingly demanding faster data acquisition. Cache databases, as high-performance relational database systems, are widely used in critical business areas such as healthcare and finance, supporting standard data querying and access via JDBC. In big data scenarios, existing technologies typically employ batch acquisition methods based on JDBC connections, using full or incremental acquisition strategies to achieve periodic data synchronization. This batch processing mode relies on a scheduling engine to start acquisition tasks at fixed intervals, packaging and processing multiple data entries before transmitting them to the target system. It performs well in scenarios with small data volumes and low timeliness requirements. Some database systems, such as MySQL, have implemented binlog-based CDC mechanisms, enabling real-time capture and streaming processing of changed data using tools like FlinkCDC.

[0003] However, Cache databases do not offer a real-time data acquisition solution similar to FlinkCDC for MySQL, and existing batch processing models exhibit significant limitations in practical applications. First, the nature of batch processing dictates that multiple data entries must be processed simultaneously, making it impossible to respond instantly to changes in individual data entries. This limitation in processing granularity makes it difficult for the system to meet the refined business needs requiring data processing on a per-entry basis. Second, periodic fetching based on a scheduling engine is a passive acquisition mode; the inherent delay from data generation to collection due to the scheduling cycle significantly increases system resource consumption, even if the scheduling interval is shortened. With the booming development of real-time businesses such as the Internet of Things and financial risk control, more and more application scenarios require rapid detection of data changes and immediate triggering of subsequent processing flows. The data latency of existing batch processing models can no longer meet these real-time requirements. Therefore, how to achieve streaming real-time data change acquisition in a Cache database environment has become an urgent technical problem to be solved. Summary of the Invention

[0004] The main objective of this invention is to solve the problems of high data latency and inability to respond to data changes in real time caused by the batch processing mode of existing cache database acquisition methods.

[0005] The first aspect of this invention provides a real-time data collection method based on cache database logs. The method includes: real-time parsing of the Journal change log of the cache database and extracting change operation records from the Journal change log, wherein the change operation records include a change type identifier, an encrypted field identifier, and an encrypted field value; escaping and decoding the encrypted field identifier and encrypted field value to obtain escaped field identifiers and escaped field values, and mapping the escaped field identifiers to actual field names; labeling the escaped field values ​​with data types according to the data type identifier corresponding to the actual field names, generating structured change data and transmitting it to a message queue; parsing the structured change data in the message queue to obtain database source information and table structure information, and writing the structured change data into a data table corresponding to a preset target storage system according to the database source information and the table structure information.

[0006] Optionally, in a first implementation of the first aspect of the present invention, the real-time parsing of the Journal change log of the Cache database and the extraction of change operation records from the Journal change log includes: reading the binary format Journal change log in the Journal log storage path of the Cache database; performing type identification on the Journal change log and filtering out data setting records with change type identifier SET from the type identification results; and extracting the encrypted field identifier and encrypted field value from the data setting record.

[0007] Optionally, in a second implementation of the first aspect of the present invention, the step of escaping and decoding the encrypted field identifier and the encrypted field value to obtain the escaped field identifier and the escaped field value, and mapping the escaped field identifier to the actual field name, includes: decrypting and converting the encrypted field identifier and the encrypted field value respectively using the built-in formatting function of the Cache database to obtain the escaped field identifier and the escaped field value; obtaining the data table identifier from the change operation record and querying the class definition metadata corresponding to the data table identifier; parsing the attribute definition list from the class definition metadata and parsing the mapping relationship between the internal field identifier and the field name of each attribute in the attribute definition list; and mapping the escaped field identifier to the corresponding field name according to the mapping relationship to obtain the actual field name.

[0008] Optionally, in a third implementation of the first aspect of the present invention, the step of annotating the escaped field value with a data type identifier corresponding to the actual field name includes: extracting the database name, table name, primary key attribute identifier, and the Cache native data type corresponding to the actual field name from the class definition metadata; converting the Cache native data type into a basic data type as type annotation information according to a preset type mapping relationship; and organizing the database name, table name, primary key attribute identifier, actual field name, escaped field value, type annotation information, and corresponding change type identifier into structured change data based on a preset structure format.

[0009] Optionally, in a fourth implementation of the first aspect of the present invention, generating structured change data and transmitting it to a message queue includes: writing the structured change data into a preset local log file; scanning the local log file using a preset log collection component; when new data is detected in the local log file, reading the new data to obtain data to be transmitted; and sending the data to be transmitted to a message queue.

[0010] Optionally, in the fifth implementation of the first aspect of the present invention, the step of scanning the local log file through a preset log collection component, and reading the new data to obtain the data to be transmitted when new data is detected in the local log file includes: obtaining the current file size and historical read offset of the local log file; when the current file size is greater than the historical read offset, reading the data content between the start and end positions in the local log file using the preset log collection component with the historical read offset as the start position and the current file size as the end position to obtain incremental data; identifying the newline character in the incremental data, and splitting the incremental data into multiple data records according to the newline character to generate the data to be transmitted.

[0011] Optionally, in the sixth implementation of the first aspect of the present invention, parsing the structured change data in the message queue to obtain database source information and table structure information includes: obtaining structured change data from the message queue in real time; extracting the database name and table name from the structured change data to obtain database source information; and extracting the actual field names, type labeling information, and primary key attribute identifier from the structured change data to generate table structure information.

[0012] A second aspect of the present invention provides a real-time acquisition device based on cache database logs. The device comprises: a log parsing unit, configured to parse the Journal change logs of the cache database in real time and extract change operation records from the Journal change logs, the change operation records including a change type identifier, an encrypted field identifier, and an encrypted field value; a field mapping unit, configured to escape and decode the encrypted field identifier and encrypted field value to obtain escaped field identifiers and escaped field values, and map the escaped field identifiers to actual field names; a data organization unit, configured to annotate the escaped field values ​​with data type identifiers corresponding to the actual field names, generate structured change data, and transmit it to a message queue; and a data writing unit, configured to parse the structured change data in the message queue to obtain database source information and table structure information, and write the structured change data into a data table corresponding to a preset target storage system based on the database source information and the table structure information.

[0013] A third aspect of the present invention provides a real-time acquisition device based on cache database logs, comprising: a memory and at least one processor, wherein the memory stores instructions; the at least one processor invokes the instructions in the memory to cause the real-time acquisition device based on cache database logs to execute the various steps of the above-described real-time acquisition method based on cache database logs.

[0014] A fourth aspect of the present invention provides a computer-readable storage medium storing instructions that, when executed on a computer, cause the computer to perform the steps of the above-described real-time acquisition method based on cache database logs.

[0015] The above-described real-time data collection method and related equipment based on Cache database logs. In this embodiment of the invention, change operation records containing change type identifiers, encrypted field identifiers, and encrypted field values ​​are extracted by real-time parsing of the Journal change logs of the Cache database; the encrypted field identifiers and encrypted field values ​​are escaped and decoded to obtain escaped field identifiers and escaped field values, and the escaped field identifiers are mapped to actual field names; the escaped field values ​​are labeled with data types according to the data type identifiers corresponding to the actual field names, generating structured change data and transmitting it to a message queue; the structured change data in the message queue is parsed to obtain database source information and table structure information, and the structured change data is written into the data table corresponding to the preset target storage system according to the database source information and table structure information. This application constructs a streaming processing chain that includes real-time parsing of journal logs, encrypted field escaping mapping, data type annotation conversion, and message queue transmission. This enables end-to-end real-time collection of cache database change data from generation to storage, solving the technical problems of high data latency and inability to respond to data changes in real time caused by the batch processing mode of existing cache database collection methods. It also eliminates the limitation of passively pulling data by the scheduling engine, supports the instant processing of single data entries, effectively improves the system's response speed and processing accuracy to data changes, and meets the strict requirements of data timeliness in real-time business scenarios.

[0016] Other features and advantages of the invention will be set forth in the description which follows, and will be apparent in part from the description, or may be learned by practicing the invention. The objects and other advantages of the invention are realized and obtained in accordance with the structures particularly pointed out in the description, claims and drawings.

[0017] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0018] Figure 1 This is a schematic diagram of the first embodiment of the real-time data collection method based on Cache database logs in this invention. Figure 2 This is a schematic diagram of an embodiment of the real-time acquisition device based on Cache database logs in this invention. Figure 3 This is a schematic diagram of an embodiment of a real-time acquisition device based on a cache database log in this invention. Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0020] The terms "comprising" and "having," and any variations thereof, used in the embodiments of this invention are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or units is not limited to the steps or units listed, but may optionally include other steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or devices.

[0021] To facilitate understanding of this embodiment, the specific process of this embodiment is described below. Please refer to [link / reference]. Figure 1 The first embodiment of the real-time data collection method based on cache database logs in this invention includes: 101. Real-time parsing of the Journal change log in the Cache database, and extraction of change operation records from the Journal change log. The change operation records include change type identifier, encrypted field identifier, and encrypted field value. In this embodiment, the binary format Journal change log is read from the Journal log storage path of the Cache database; the Journal change log is type-identified, and data setting records with change type identifier SET are filtered out from the type identification results; the encrypted field identifier and encrypted field value are extracted from the data setting records.

[0022] Specifically, the Cache database first stores all database operation records in a Journal log file. This log file is located by default in the mgr / journal path under the Cache installation directory and is stored in binary format. This binary file cannot be directly read and parsed by external programs; it must be accessed through an internal system class provided by the Cache database. M-language code is written in the Cache database's built-in Studio editor to call the `##class(%SYS.Journal.File)` system class to obtain a Journal file object. This system class encapsulates the underlying interface for accessing the Journal binary file, enabling the reading of database operation events recorded in the log sequentially. For example, when the database performs an INSERT or UPDATE operation to modify data in the Person table, Cache generates a corresponding operation record in the Journal log, including the operation timestamp, operation type, and information on the affected tables and fields. Real-time monitoring of database changes is achieved by continuously reading the Journal file object.

[0023] Secondly, each log record read from the Journal file object contains an InTransaction field and a Global node field. The InTransaction field has values ​​including status indicators such as "No," "Yes," and "Completed," while the Global node field records the specific operation type. When the content of the Global node field is a system variable assignment like "^%SYS("SERVICE","ECPCluster")", it corresponds to a SET type operation, indicating an assignment to a global variable or data table field. When the log record contains the BeginTrans or CommitTrans identifiers, it indicates the start and commit of a transaction; these records do not contain actual data change information. In the code implementation, the log record sequence obtained through the Read method is traversed, and the operation type characteristics of each record are checked to identify records of the SET type. The determination method is to check whether the log record's details field contains assignment operation characteristics, such as whether the new value field exists. All SET type records are filtered out, while records of transaction control types such as BeginTrans and CommitTrans are filtered out. SET type records correspond to field value changes caused by INSERT and UPDATE statements. They are a key operation type for capturing actual data changes in the database. Filtering with this type avoids processing records that do not involve data changes, such as transaction control.

[0024] Finally, the field identifiers and values ​​stored internally in the filtered SET type records use the internal encoding format of the Cache database. The encrypted field identifiers referred to here are attribute encoding identifiers used internally by the Cache object-oriented database, not the actual column names of the table. The encrypted field values ​​are numerical representations after being converted by the Cache's internal encoding. The field identifier and value portions are directly extracted from the SET type records to obtain the encrypted field identifiers and values. These encrypted field identifiers and values ​​still maintain the internal encoding format of the Journal log. For example, for changes to the Name field in the Person table, the extracted field identifier is presented in the Cache's internal attribute encoding form. This encoding is the Cache system's internal representation and not the readable "Name" field name. The encrypted field value is also stored in the internal encoding format. The extracted encrypted field identifiers and values ​​need to be escaped and decoded using Cache's built-in functions in subsequent steps to be converted into an operable text format and further mapped to the actual field names by querying metadata.

[0025] 102. Escape and decode the encrypted field identifier and encrypted field value to obtain the escaped field identifier and escaped field value, and map the escaped field identifier to the actual field name; In this embodiment, the encrypted field identifier and encrypted field value are decrypted and converted using the built-in formatting function of the Cache database to obtain the escaped field identifier and escaped field value; the data table identifier is obtained from the change operation record, and the class definition metadata corresponding to the data table identifier is queried; the attribute definition list is parsed from the class definition metadata, and the mapping relationship between the internal field identifier and field name of each attribute in the attribute definition list is parsed; according to the mapping relationship, the escaped field identifier is mapped to the corresponding field name to obtain the actual field name.

[0026] Specifically, the encrypted field identifier and encrypted field value extracted from the SET type record in the previous steps are passed as input parameters to the $$Format^%qcr function. $$Format^%qcr is a dedicated function for formatting Journal logs provided by the Cache database, which can convert the binary encoded data inside the Journal into an operable text format. This function parses the input encrypted data and returns the escaped result. The escaped field identifier refers to the internal identifier of the field that has been converted from binary encoding to character form, but this identifier is still an attribute reference form used internally by the Cache system, rather than a field name that can be directly recognized externally. The escaped field value is the actual data content restored from the encoded format. For example, for an update operation on the Name field of a record in the Person table, the escaped field value can obtain the specific text content such as "Garcia5". After escaping and decoding, the data has been converted from the proprietary binary format of the Journal log into a text representation form that the Cache system can further process. However, since the field identifier still uses an internal reference method, the external system cannot recognize its corresponding actual field name. Therefore, it is necessary to query the metadata for further mapping.

[0027] Secondly, the Global node field in the change operation log records the complete path of the global variable where the data change occurred. For example, "^%SYS("SERVICE","ECPCluster")" shown in the Journal log indicates a change to the system service class, or "^Sample.PersonD" indicates the data node of the Person table in the Sample database. The data table identifier is extracted from the Global node field by identifying the part after the "^" symbol and before "D" or other suffixes. For example, "Sample.Person" is extracted from "^Sample.PersonD" as the data table identifier, where "Sample" is the database name and "Person" is the table name. Class definition metadata is system data stored in the Cache object-oriented database that stores class structure information, recording a complete structural description of all class attribute names, attribute data types, and the order in which attributes are defined within the class. In the Studio editor, calling the %OpenId method of the system class `##class(%Dictionary.ClassDefinition)` and passing the extracted data table identifier "Sample.Person" as a parameter opens and returns the class definition object corresponding to the Person class. This class definition object allows access to the complete structure information of the Person table, including which fields are defined and what the internal attribute identifier for each field is. Obtaining class definition metadata enables querying from the table identifier to detailed table structure information, providing a data source for resolving the mapping between the internal identifiers of fields and their actual names.

[0028] Then, the `Properties` method of the obtained class definition object is called. This method returns a collection of all properties defined in the class; this collection is the property definition list. Each element in the property definition list represents the complete definition information of a field. The property definition list is traversed, and for each property object in the list, its `Name` method is called to obtain the field name corresponding to that property. Simultaneously, the internal identifier used by the property in the Journal log is obtained. The internal field identifier is the field reference method used by the Cache system when recording Journal logs, and it differs from the field names used externally. For example, for the `Person` class, when traversing its property definition list, the `Name` of the first property object returns "ID", the `Name` of the second property object returns "Name", the third property object returns "SSN", and so on. At the same time, the position or internal reference mark of each property in the class definition is recorded, establishing a mapping relationship between the internal field identifier and the field name. For example, a mapping table is established, mapping the first property position in the `Person` class to the field name "ID", the second position to "Name", and the third position to "SSN". This mapping relationship solves the need for the Journal log to use internal identifiers to record field changes, while the external system requires the conversion of the actual field names.

[0029] Finally, the internal identifier characteristics are identified from the escaped field identifiers obtained in the first step. For example, if the identifier points to the second attribute in the class definition, the corresponding field name is searched in the mapping table established above using this internal identifier as an index. The mapping table records the correspondence from attribute definition positions or internal identifiers to actual field names, and the field name can be directly obtained through table lookup. For example, when the escaped field identifier points to the second attribute of the Person class, the actual field name "Name" is obtained by searching for the field name corresponding to the second position in the mapping table. The same mapping conversion is performed on all fields involved in a change record in the Journal log, converting all internal identifiers to actual field names. For example, if an UPDATE operation modifies the Name, DOB, and Home_City fields of the Person table, the corresponding three escaped field identifiers are mapped to "Name", "DOB", and "Home_City" respectively. The actual field names obtained after mapping are standard database field names that external systems can directly understand and use. Combined with the escaped field values ​​obtained in the first step, it is possible to clearly identify which field has undergone data changes, providing a readable field information foundation for building standardized data change records.

[0030] 103. Based on the data type identifier corresponding to the actual field name, mark the data type of the escaped field value, generate structured change data, and transmit it to the message queue; In this embodiment, the cache native data types corresponding to the database name, table name, primary key attribute identifier, and actual field name are extracted from the class definition metadata; according to the preset type mapping relationship, the cache native data types are converted into basic data types as type annotation information; based on the preset structure format, the database name, table name, primary key attribute identifier, actual field name, escaped field value, type annotation information, and corresponding change type identifier are organized into structured change data. Generating structured change data and transmitting it to the message queue includes: writing the structured change data to a preset local log file; scanning the local log file using a preset log collection component, and when new data is detected in the local log file, reading the new data to obtain the data to be transmitted (scanning the local log file using the preset log collection component, and when new data is detected in the local log file, reading the new data to obtain the data to be transmitted includes: obtaining the current file size and historical read offset of the local log file; when the current file size is greater than the historical read offset, using the preset log collection component, starting from the historical read offset and ending at the current file size, reading the data content between the start and end positions in the local log file to obtain incremental data; identifying newline characters in the incremental data, and splitting the incremental data into multiple data records based on the newline characters to generate the data to be transmitted.); and sending the data to be transmitted to the message queue.

[0031] Specifically, the class definition object obtained above allows access to the complete structural information of the class. The relevant methods provided by the class definition object are called to obtain the database name and table name; for example, for the Person class, the database name is "Sample" and the table name is "Person". The attribute set in the class definition object is traversed, and each attribute is checked to see if it is marked as a primary key. The primary key attribute is a field in the data table used to uniquely identify each record. For example, if the ID field in the Person table is marked as a primary key, then the primary key attribute is marked as "ID". For the actual field name obtained through mapping, the attribute definition corresponding to that field name is searched in the attribute set of the class definition object, and the data type identifier of that field is read from the attribute definition. Cache native data types are data type representations defined by the Cache database system, using a naming format starting with a percent sign "%", such as "%String" for string type, "%Integer" for integer type, and "%BigInt" for large integer type. For example, the Cache native data type for the Name field in the Person table is "%String", the ID field is "%Integer", the DOB field is "%BigInt", and the Home_Zip field is "%Integer". This information is extracted to create a complete metadata description for each changed field, including the database to which the field belongs, the table to which it belongs, the primary key of the table, and the data type of the field. This metadata is the necessary information foundation for building a standardized change data structure.

[0032] Secondly, the preset type mapping relationship is a predefined conversion rule table from cache-specific data types to general data types. This mapping table records the correspondence between cache native data types and basic data types recognized by stream processing systems such as FlinkCDC. The type mapping relationship includes conversion rules such as "%String" to "string", "%Integer" to "int32", and "%BigInt" to "int64". For the cache native data type of each field extracted above, the corresponding basic data type is looked up in the type mapping relationship table for conversion. For example, when the cache native data type of the Name field is "%String", the corresponding basic data type "string" is found in the mapping table; when the type of the ID field is "%Integer", it is converted to "int32"; and when the type of the DOB field is "%BigInt", it is converted to "int64". The converted basic data type serves as type labeling information, identifying the type attribute of the field in the general data exchange format. For example, the type annotation information for each field in the Person table after conversion is as follows: ID field corresponds to "int32", Name field corresponds to "string", SSN field corresponds to "string", DOB field corresponds to "int64", and Home_Zip field corresponds to "int64". This type conversion eliminates the type representation method unique to the cache database and adopts standard type identifiers that can be directly recognized by stream processing frameworks such as FlinkCDC, enabling changed data from the cache database to be seamlessly integrated into the subsequent stream processing system.

[0033] Furthermore, the default structure format here adopts JSON format, containing two top-level fields: schema and payload. In the schema field, type is defined as "struct" to indicate that this is structured data. The fields array contains an after sub-object, and the fields array of the after sub-object creates a field definition item for each actual field name. Each field definition item contains a field attribute recording the actual field name, a type attribute recording type annotation information, and an optional attribute indicating whether the field allows null values. In the payload field, the source sub-object contains a db attribute recording the database name and a table attribute recording the table name. The after sub-object contains key-value pair mappings between each actual field name and its escaped field value. The op attribute records the change type identifier, using "c" for insert operations, "u" for update operations, and "d" for delete operations. For example, for an inserted record with ID 204 in the Person table, the reorganized structured change data includes the schema part containing the type definitions of each field, such as the ID field being defined as int32 and the Name field as string. The payload part contains the source specifying the database as Sample and the table as Person, and the after part containing the specific field values, such as ID as 204, Name as Garcia5, and SSN as 842-46-9429. The op identifier "c" indicates an insert operation. The reorganized structured change data completely records all the information of a data change event, including the database and table where the change occurred, the fields involved and their types, the specific values ​​of the fields, and the type of change operation performed, forming data in the debezium-json format conforming to the FlinkCDC standard.

[0034] Furthermore, the local log files here are pre-created text files in the local file system of the Cache database server, used to store structured change data in JSON format. The file storage path is set in an accessible directory, such as "E:\temp", and the file names use date-based serial numbers, such as "20250718.001.log", "20250721.001.log", "20250722.001.log", etc., facilitating time-based log file management. The M-language program written in the Cache Studio editor runs continuously. After each change record's structured data organization is completed, the JSON-formatted data is appended to the end of the local log file corresponding to the current date using file operation functions. Each structured change data occupies a single line, and records are separated by newline characters. For example, after processing an insert operation on the Person table, the corresponding complete JSON data is written to the 20250718.001.log file, which contains all the field and value information of this insert operation. To prevent the program from consuming excessive system resources and degrading database performance due to continuous operation, an appropriate sleep interval is set in the program. When no new changes are detected in the Journal log, the program pauses execution for a period of time before resuming its checks. This intermittent processing reduces CPU usage. Writing structured change data to a local log file enables data export from internal cache processing to an external file, establishing a file interface for subsequent data reading using standard log collection tools.

[0035] Therefore, the log collection component here uses Filebeat, a lightweight tool specifically designed for monitoring file changes and collecting log data. In Filebeat's configuration file journal2kafka.yml, the path pattern to be monitored is specified through the paths parameter, for example, "E:\temp\20 The `.log` directive matches all log files that conform to the naming rules. The `scan_frequency` parameter sets the file scan frequency to 5 seconds, meaning Filebeat checks the target file for changes every 5 seconds. Filebeat maintains a registry file recording the read status of each monitored file, including the byte position already read; this position is called the historical read offset. During each scan, Filebeat obtains the current file size of the local log file and reads the file's byte length using the `stat` system call. It compares the current file size with the historical read offset; if the current file size is greater than the historical read offset, it indicates that new content has been added to the file. Filebeat uses file read operations, starting from the historical read offset and reading up to the current file size, obtaining the data within this byte range; this data is the incremental data. For example, if a log file was previously read up to byte 1500, and the current file size is 3200 bytes, Filebeat reads 1700 bytes between byte 1500 and byte 3200 as incremental data. Filebeat performs line parsing on the read incremental data, identifying newline characters "\n" as record separators, and then... The filebeat splits a continuous byte stream into multiple independent text lines. Each text line corresponds to a complete JSON-formatted structured change data. The resulting multiple JSON records constitute the data set to be transmitted. Filebeat updates the registry file, recording the end position of the current read (3200 bytes) as the new historical read offset for use in the next scan. This mechanism, based on file size comparison and offset recording, ensures that only newly added data is read each time, avoiding repeated reading of already processed data, thus achieving incremental collection of local log files. Furthermore, in actual operation, Filebeat also needs to handle file anomalies. When the current file size is detected to be smaller than the historical read offset, it indicates that the file has been truncated, cleared, or reset. In this case, Filebeat resets the historical read offset to 0 and restarts reading from the beginning of the file, avoiding data reading failures due to offset errors. When a monitored log file is renamed, moved, or deleted, Filebeat detects the file rotation event through its file monitoring mechanism, automatically identifies the newly generated log file and includes it in the monitoring scope, while maintaining tracking of the original file until reading is complete, ensuring no data loss during file switching. These anomaly handling mechanisms guarantee the stability and data integrity of log collection.

[0036] Finally, in the Filebeat configuration file, the `output.kafka` section is configured, specifying the list of Kafka cluster server addresses via the `hosts` parameter. For example, `["192.168.2.160:9092","192.168.2.175:9092"]` indicates that the Kafka cluster contains two nodes. The target topic name is specified as "cache-journal-data" via the `topic` parameter, and all collected change data is sent to this topic. The `required_acks` parameter is set to 1, indicating that only the Kafka master node needs to confirm receipt to return success, without waiting for all replica nodes to synchronize, thus improving sending efficiency while ensuring no data loss. Filebeat treats each JSON record in the data to be transmitted as a message and calls the Kafka client's send interface to send the message to the `cache-journal-data` topic. Each JSON record is kept intact during sending, without splitting or merging. For example, when the data to be transmitted contains three changed records from the Person table, Filebeat sends three independent messages sequentially to a Kafka topic. Each message contains a complete JSON-formatted structured change data. Upon receiving the messages, Kafka persists them in a partition of the topic. Subsequent stream processing systems such as FlinkCDC can subscribe to this topic and consume this changed data. Sending to the message queue completes the data transfer from file storage to the message stream, allowing the changed data from the cache database to enter the real-time data processing pipeline, enabling it to be consumed and processed in real time by the stream processing framework.

[0037] 104. Parse the structured change data in the message queue to obtain the database source information and table structure information, and write the structured change data into the data table corresponding to the preset target storage system based on the database source information and table structure information.

[0038] In this embodiment, structured change data is obtained in real time from the message queue; the database name and table name are extracted from the structured change data to obtain the database source information; the actual field names, type label information and primary key attribute identifiers are extracted from the structured change data and combined to generate table structure information.

[0039] Specifically, the FlinkCDC framework is first used to subscribe to the `cache-journal-data` topic in the Kafka message queue. FlinkCDC is a change data capture component provided by Apache Flink, capable of consuming data streams from Kafka topics in real time. A `kafka_sync_database` synchronization task is started in the Flink cluster, configuring the Kafka cluster address, topic name, consumer group identifier, and specifying the data format as `debezium-json`. After starting, FlinkCDC continuously listens to the `cache-journal-data` topic. When Filebeat sends a new message to this topic, FlinkCDC immediately receives the message content. Each received message is a complete JSON-formatted structured change data. For example, if FlinkCDC receives a message about an inserted record with ID 204 from the `Person` table, this message contains the complete schema field definition and payload field content. FlinkCDC parses the JSON message into an internal data object for subsequent processing. This real-time acquisition mechanism ensures that change data from the cache database enters the stream processing system immediately after its generation, eliminating the inherent latency caused by scheduling cycles in traditional batch data collection.

[0040] Next, FlinkCDC parses the JSON-formatted structured change data and locates the `source` sub-object under the `payload` field. It reads the `db` attribute value from the `source` sub-object as the database name and the `table` attribute value as the table name. For example, for a change record in the `Person` table, the `db` value is extracted as "Sample" and the `table` value as "Person". The database name and table name are combined to form the database source information "Sample.Person", which identifies the source location of the change data. This database source information is used for subsequent data routing, determining which corresponding table in the target storage system the change data should be written to. For example, data with the source information "Sample.Person" needs to be routed to the `ods_person` table in the `paimon_cache` database of Paimon. Extracting the database source information enables accurate identification and classification of change data from different sources.

[0041] Next, FlinkCDC parses the schema field of the JSON data, locates the fields array of the after sub-object, and iterates through each field definition in the array. From each field definition, it reads the field attribute to obtain the actual field name and the type attribute to obtain type annotation information. The primary key attribute identifier was already annotated in the schema during the aforementioned organizational structure change of data; it is obtained by recognizing the primary key marker in the field definition. For example, for the Person table, it extracts the actual field name "ID" and the type annotation "int32" for the ID field and identifies it as the primary key; it extracts the actual field name "Name" and the type annotation "string" for the Name field. It combines the actual field names, type annotation information, and primary key attribute identifiers of all fields to generate table structure information. The table structure information describes the complete structure definition of the table corresponding to the changed data, including the field list, field types, and primary key information. The table structure information is used to verify the consistency between the target table structure and the changed data structure, ensuring that the data can be written correctly.

[0042] Finally, the default target storage system here is Apache Paimon. The Paimon catalog and paimon_cache databases are pre-created in Flink SQL, and the ods_person target table is created based on the Person table structure. This table contains fields such as ID, Name, SSN, and DOB, with the primary key set to ID. FlinkCDC uses the database source information Sample.Person to find the pre-configured table_mapping mapping relationship and determines the corresponding target table as ods_person. FlinkCDC verifies whether the field definitions of the ods_person table match the field structure of the changed data based on the table structure information. The actual values ​​of each field are extracted from the after sub-object of the JSON data payload field, and the change type identifier is identified from the op field. When op is "c", an INSERT operation is performed to insert a new record; when op is "u", an UPDATE operation is performed to update a record; and when op is "d", a DELETE operation is performed to delete a record. For example, for the Person record with ID 204, an op value of "c" indicates an insert operation. FlinkCDC inserts the ID value 204, Name value Garcia5, and SSN value 842-46-9429 into the ods_person table. Write operations are performed through Flink's paimon-flink-action component, which handles data writing and primary key conflict detection. After the write operation is complete, the data is persistently stored in Paimon, achieving real-time data synchronization from the cache database to the target storage system and realizing streaming data transfer from the relational database to the lakehouse storage.

[0043] In this embodiment of the invention, the Journal change log of the Cache database is parsed in real time to extract change operation records, including change type identifier, encrypted field identifier, and encrypted field value. The encrypted field identifier and encrypted field value are escaped and decoded to obtain escaped field identifier and escaped field value, and the escaped field identifier is mapped to the actual field name. The escaped field value is then labeled with its type according to the data type identifier corresponding to the actual field name, generating structured change data and transmitting it to a message queue. The structured change data in the message queue is parsed to obtain database source information and table structure information, which is then written to the target storage system. This application achieves real-time streaming acquisition of the Cache database, supports instant processing of single data entries, significantly reduces data latency, and meets real-time business requirements.

[0044] The above describes the real-time data collection method based on cache database logs in the embodiments of the present invention. The following describes the real-time data collection device based on cache database logs in the embodiments of the present invention. Please refer to [link / reference]. Figure 2 One embodiment of the real-time acquisition device based on cache database logs in this invention includes: Log parsing unit 201 is used to parse the Journal change log of the Cache database in real time and extract change operation records from the Journal change log. The change operation records include change type identifier, encrypted field identifier and encrypted field value. The field mapping unit 202 is used to escape and decode the encrypted field identifier and the encrypted field value to obtain the escaped field identifier and the escaped field value, and to map the escaped field identifier to the actual field name; The data organization unit 203 is used to annotate the escaped field value with data type according to the data type identifier corresponding to the actual field name, generate structured change data and transmit it to the message queue; The data writing unit 204 is used to parse the structured change data in the message queue to obtain database source information and table structure information, and write the structured change data into the data table corresponding to the preset target storage system according to the database source information and the table structure information.

[0045] In this embodiment of the invention, the Journal change log of the Cache database is parsed in real time to extract change operation records, including change type identifier, encrypted field identifier, and encrypted field value. The encrypted field identifier and encrypted field value are escaped and decoded to obtain escaped field identifier and escaped field value, and the escaped field identifier is mapped to the actual field name. The escaped field value is then labeled with its type according to the data type identifier corresponding to the actual field name, generating structured change data and transmitting it to a message queue. The structured change data in the message queue is parsed to obtain database source information and table structure information, which is then written to the target storage system. This application achieves real-time streaming acquisition of the Cache database, supports instant processing of single data entries, significantly reduces data latency, and meets real-time business requirements.

[0046] above Figure 2 The real-time acquisition device based on cache database logs in this embodiment of the invention will be described in detail from the perspective of modular functional entities. The real-time acquisition device based on cache database logs in this embodiment of the invention will be described in detail from the perspective of hardware processing.

[0047] Figure 3 This is a schematic diagram of the structure of a real-time acquisition device based on a cache database log provided in an embodiment of the present invention. The real-time acquisition device 300 based on the cache database log can vary significantly due to different configurations or performance. It may include one or more central processing units (CPUs) 310 (e.g., one or more processors) and a memory 320, and one or more storage media 330 (e.g., one or more mass storage devices) for storing application programs 333 or data 332. The memory 320 and storage media 330 can be temporary or persistent storage. The program stored in the storage media 330 may include one or more modules (not shown in the diagram), each module may include a series of instruction operations on the real-time acquisition device 300 based on the cache database log. Furthermore, the processor 310 may be configured to communicate with the storage media 330 and execute the series of instruction operations in the storage media 330 on the real-time acquisition device 300 based on the cache database log.

[0048] The real-time data acquisition device 300 based on cache database logs may also include one or more power supplies 340, one or more wired or wireless network interfaces 350, one or more input / output interfaces 360, and / or one or more operating systems 331, such as Windows Server, Mac OS X, Unix, Linux, FreeBSD, etc. Those skilled in the art will understand that... Figure 3The illustrated structure of a real-time acquisition device based on cache database logs does not constitute a limitation on real-time acquisition devices based on cache database logs. It may include more or fewer components than illustrated, or combine certain components, or have different component arrangements.

[0049] The present invention also provides a real-time acquisition device based on cache database logs. The computer device includes a memory and a processor. The memory stores computer-readable instructions. When the computer-readable instructions are executed by the processor, the processor performs each step of the real-time acquisition method based on cache database logs in the above embodiments.

[0050] The present invention also provides a computer-readable storage medium, which can be a non-volatile computer-readable storage medium or a volatile computer-readable storage medium, wherein the computer-readable storage medium stores instructions that, when the instructions are executed on a computer, cause the computer to perform each step of the real-time acquisition method based on cache database logs.

[0051] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0052] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0053] This application can be used in a wide variety of general-purpose or special-purpose computer system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices. This application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. This application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0054] The above-described embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A real-time data collection method based on cache database logs, characterized in that, The real-time data collection method based on cache database logs includes: The Journal change log of the Cache database is parsed in real time, and change operation records are extracted from the Journal change log. The change operation records include change type identifier, encrypted field identifier and encrypted field value. The encrypted field identifier and encrypted field value are escaped and decoded to obtain the escaped field identifier and escaped field value, and the escaped field identifier is mapped to the actual field name; Based on the data type identifier corresponding to the actual field name, the escaped field value is labeled with the data type, generating structured change data and transmitting it to the message queue; The structured change data in the message queue is parsed to obtain database source information and table structure information. Based on the database source information and the table structure information, the structured change data is written into the data table corresponding to the preset target storage system.

2. The real-time data collection method based on cache database logs according to claim 1, characterized in that, The real-time parsing of the Journal change log in the Cache database and the extraction of change operation records from the Journal change log include: Read the binary format Journal change log from the Journal log storage path in the Cache database; The Journal change log is type-identified, and data setting records with change type identifier SET are filtered out from the type identification results; Extract the encrypted field identifier and encrypted field value from the data settings record.

3. The real-time data collection method based on cache database logs according to claim 1, characterized in that, The step of escaping and decoding the encrypted field identifier and encrypted field value to obtain the escaped field identifier and escaped field value, and mapping the escaped field identifier to the actual field name, includes: The encrypted field identifier and the encrypted field value are decrypted and converted using the built-in formatting functions of the Cache database to obtain the escaped field identifier and the escaped field value. Obtain the data table identifier from the change operation record, and query the class definition metadata corresponding to the data table identifier; Parse the attribute definition list from the class definition metadata, and then parse the mapping relationship between the internal field identifiers and field names of each attribute in the attribute definition list; Based on the mapping relationship, the escaped field identifier is mapped to the corresponding field name to obtain the actual field name.

4. The real-time data collection method based on cache database logs according to claim 3, characterized in that, The step of annotating the escaped field value with its data type identifier based on the actual field name includes: Extract the database name, table name, primary key attribute identifier, and Cache native data type corresponding to the actual field name from the class definition metadata; According to the preset type mapping relationship, the Cache native data type is converted into the basic data type as type annotation information; Based on a preset structure format, the database name, the table name, the primary key attribute identifier, the actual field name, the escaped field value, the type annotation information, and the corresponding change type identifier are organized into structured change data.

5. The real-time data collection method based on cache database logs according to claim 1, characterized in that, The generation of structured change data and its transmission to the message queue includes: Write the structured change data to a preset local log file; The local log file is scanned by a preset log collection component. When new data is detected in the local log file, the new data is read to obtain the data to be transmitted. The data to be transmitted is sent to the message queue.

6. The real-time data collection method based on Cache database logs according to claim 5, characterized in that, The process involves scanning the local log file using a preset log collection component. When new data is detected in the local log file, the new data is read to obtain the data to be transmitted, including: Obtain the current file size and historical read offset of the local log file; When the current file size is greater than the historical read offset, the preset log collection component reads the data content between the start and end positions in the local log file, starting from the historical read offset and ending from the current file size, to obtain incremental data. Identify newline characters in the incremental data, and based on the newline characters, split the incremental data into multiple data records to generate data to be transmitted.

7. The real-time data collection method based on cache database logs according to claim 1, characterized in that, The process of parsing the structured change data in the message queue to obtain database source information and table structure information includes: Retrieve structured change data in real time from the message queue; The database name and table name are extracted from the structured change data to obtain the database source information; The actual field names, type annotation information, and primary key attribute identifiers are extracted from the structured change data and combined to generate table structure information.

8. A real-time data acquisition device based on cache database logs, characterized in that, The real-time data collection device based on cache database logs includes: The log parsing unit is used to parse the Journal change log of the Cache database in real time and extract change operation records from the Journal change log. The change operation records include change type identifier, encrypted field identifier and encrypted field value. The field mapping unit is used to escape and decode the encrypted field identifier and the encrypted field value to obtain the escaped field identifier and the escaped field value, and to map the escaped field identifier to the actual field name. The data organization unit is used to annotate the escaped field value with data type according to the data type identifier corresponding to the actual field name, generate structured change data and transmit it to the message queue; The data writing unit is used to parse the structured change data in the message queue to obtain database source information and table structure information, and write the structured change data into the data table corresponding to the preset target storage system according to the database source information and the table structure information.

9. A real-time data acquisition device based on cache database logs, characterized in that, The real-time acquisition device based on cache database logs includes: a memory and at least one processor, wherein the memory stores instructions; The at least one processor invokes the instructions in the memory to cause the real-time acquisition device based on the cache database log to perform each step of the real-time acquisition method based on the cache database log as described in any one of claims 1-7.

10. A computer-readable storage medium storing instructions thereon, characterized in that, When the instructions are executed by the processor, they implement each step of the real-time acquisition method based on cache database logs as described in any one of claims 1-7.

Images