Data direct connection writing method and system and electronic equipment

Abstract

The invention provides a data direct-connection writing method and system and electronic equipment, and belongs to the technical field of data processing, and the method comprises the following steps: writing to-be-written data into a buffer channel of a streaming acquisition framework; extracting the to-be-written data from the buffer channel according to a preset batch size to form a data batch, and determining a target database table according to information contained in the data batch; according to the target database table, constructing the data batch into a write-in request conforming to a target database batch write-in interface specification; and sending the write-in request to the target database table through the connection pool of the preset transmission protocol. Through data batch extraction based on the buffer channel and dynamic determination of the target database table, the batch writing requests are constructed and are directly connected and transmitted through the connection pool, so that the real-time performance and stability of data writing are improved in a large-scale real-time data writing scene.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a data direct writing method, system and electronic equipment. Background Art

[0002] In the current field of real-time data processing technology, the scale and types of data are growing significantly with the increasing complexity of business needs, and the demand for data access speed and real-time analytical processing is gradually increasing. Existing technical solutions typically use message queues (such as Kafka) to receive data, then write the data to analytical databases (such as StarRocks) through intermediate storage and secondary loading mechanisms (such as the Kafka Load mechanism). However, due to the multi-stage storage and transformation of data, this solution consumes additional storage resources, wastes network bandwidth, and suffers significant processing delays. Furthermore, due to the lack of flexible and efficient data batch processing mechanisms and dynamic routing table partitioning strategies, it is difficult to adapt to complex multi-business and multi-tenant application scenarios. Furthermore, traditional network transmission methods lack efficient connection management and reuse strategies, making connection congestion and performance jitter prone to problems when writing large amounts of concurrent data, seriously affecting the real-time and stable performance of data processing.

[0003] Therefore, how to improve the real-time and stability of data writing in large-scale real-time data writing scenarios has become a technical problem that needs to be solved urgently. Summary of the Invention

[0004] The present invention provides a data direct write method, system, electronic device and storage medium to address the defects in the prior art and improve the real-time performance and stability of data writing in large-scale real-time data writing scenarios.

[0005] The present invention provides a data direct writing method, comprising the following steps: Write the data to be written into the buffer channel of the streaming acquisition framework; Extracting the data to be written from the buffer channel according to a preset batch size to form a data batch, and determining a target database table based on information contained in the data batch; According to the target database table, constructing the data batch into a write request that complies with the target database batch write interface specification; The write request is sent to the target database table through a connection pool with a preset transmission protocol.

[0006] According to a data direct write method provided by the present invention, the buffer channel of the streaming acquisition framework is a buffer channel with a transaction mechanism; extracting the data to be written from the buffer channel according to a preset batch size to form a data batch includes: Executing a transaction corresponding to the buffer channel; Sequentially extracting a quantity of data to be written corresponding to the preset batch size from the buffer channel until the preset batch size is reached or the data in the buffer channel is empty; The data to be written is temporarily stored and extracted according to the transaction mechanism to form the data batch.

[0007] According to a data direct write method provided by the present invention, determining a target database table based on information included in the data batch includes: Invoking a dynamic routing decision engine to match predefined fields in message header information or message body of each data in the data batch, wherein the dynamic routing decision engine is configured with at least one set of routing rules, each set of routing rules including a matching condition and a database table corresponding to the matching condition; When the data in the data batch meets the matching condition corresponding to a set of routing rules in the dynamic routing decision engine, the data is routed to the database table corresponding to the matching condition; When the data in the data batch meets the matching conditions of multiple groups of routing rules, the target database table is determined according to a preset priority.

[0008] A data direct writing method provided by the present invention further includes: When the routing rules in the dynamic routing decision engine change, the routing rules are hot updated, wherein the dynamic routing decision engine or the streaming acquisition framework does not need to be restarted during the hot update.

[0009] According to a data direct write method provided by the present invention, the data batch is constructed into a write request that complies with the target database batch write interface specification based on the target database table, including: Selecting a serialization format corresponding to the data batch according to the batch write interface specification of the target database table; Serializing the data batch according to the selected serialization format to obtain serialized batch data; A write request that complies with the target database batch write interface specification is constructed based on the serialized batch data, wherein the write request includes a request header and a request body, wherein the request header includes authentication information, a target database table name, a request type, and data format information, and the request body includes the serialized batch data.

[0010] According to a data direct writing method provided by the present invention, the serialization format includes a CSV format and a CSV format with header information; wherein, the CSV format uses a comma as a field separator and a line break as a record separator; the header information includes the column name and data type of the target database table.

[0011] According to a data direct write method provided by the present invention, after the write request is sent to the target database table through the connection pool of the preset transmission protocol, the method further includes: Obtaining a response result from the target database; When the response result indicates that the write request fails to be sent, resending the write request to the target database table according to an exponential backoff mechanism until the sending is successful or the number of resends reaches a sending number threshold; When the number of resends reaches the sending number threshold, a retry exception log is generated and the corresponding data batch is returned to the buffer channel.

[0012] According to a data direct writing method provided by the present invention, before writing the data to be written into the buffer channel of the streaming acquisition framework, the method further includes: Receive raw streaming data from the message queue; The original streaming data is input into a custom interceptor for at least one of field verification, time format conversion, or label generation to obtain the data to be written.

[0013] The present invention also provides a data direct writing system, comprising the following modules: A first processing module is used to write the data to be written into the buffer channel of the streaming acquisition framework; a second processing module, configured to extract the data to be written from the buffer channel according to a preset batch size to form a data batch, and determine a target database table based on information contained in the data batch; A third processing module is configured to construct the data batch into a write request that complies with a target database batch write interface specification according to the target database table; A fourth processing module is configured to send the write request to the target database table through a connection pool of a preset transmission protocol.

[0014] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, any of the above-described direct data writing methods is implemented.

[0015] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any of the above-described direct data writing methods.

[0016] The present invention also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the computer program implements any of the above-mentioned direct data writing methods.

[0017] In summary, one or more technical solutions provided in the embodiments of the present application have at least the following technical effects or advantages: By writing the data to be written into the buffer channel of the streaming acquisition framework, real-time data access and efficient buffering are achieved, ensuring stability and continuity during data access; by extracting the data to be written from the buffer channel according to the preset batch size to form data batches, and determining the target database table based on the information contained in the data batches, the batch processing capability of data is improved, the system overhead caused by processing one by one is reduced, and data diversion is realized, so that data of different types or sources can be automatically and accurately written to different database tables; by constructing data batches into write requests that comply with the batch write interface specifications of the target database according to the target database table, and sending the write requests to the target database table through the connection pool of the preset transmission protocol, the standardization and compatibility of data write requests are improved, and write failures caused by format mismatch or field misalignment are avoided. At the same time, the connection pool is used to significantly improve network transmission efficiency and reduce write latency, thereby improving the real-time and stability of data writing in large-scale real-time data writing scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions in the present invention or the prior art, a brief introduction is given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0019] Figure 1 This is one of the flow charts of the direct data writing method provided by the present invention.

[0020] Figure 2 This is the second flow chart of the direct data writing method provided by the present invention.

[0021] Figure 3 This is the third flow chart of the direct data writing method provided by the present invention.

[0022] Figure 4 This is the fourth flow chart of the direct data writing method provided by the present invention.

[0023] Figure 5 This is the fifth flow chart of the direct data writing method provided by the present invention.

[0024] Figure 6 This is the sixth flow chart of the direct data writing method provided by the present invention.

[0025] Figure 7 This is the seventh flow chart of the direct data writing method provided by the present invention.

[0026] Figure 8 It is a structural diagram of the data direct writing system provided by the present invention.

[0027] Figure 9 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0028] To make the objectives, technical solutions, and advantages of the present invention more clear, the technical solutions of the present invention will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the embodiments described are only some embodiments of the present invention, not all embodiments. All other embodiments obtained by ordinary technicians in this field without making creative efforts based on the embodiments of the present invention are within the scope of protection of the present invention.

[0029] It should be noted that, in the description of the present invention, the terms "comprise", "include" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device comprising a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the presence of other identical elements in the process, method, article or device comprising the element. The orientation or positional relationship indicated by the terms "upper", "lower", etc. is based on the orientation or positional relationship shown in the accompanying drawings, and is only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the system or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present invention. For those of ordinary skill in the art, the specific meanings of the above terms in the present invention can be understood according to the specific circumstances.

[0030] The terms "first," "second," and so forth, used herein are used to distinguish similar objects, not to describe a specific order or precedence. It should be understood that such terms are interchangeable where appropriate, allowing embodiments of the present invention to be implemented in an order other than that illustrated or described herein. Furthermore, the terms "first," "second," and so forth generally distinguish objects of a single type, and do not limit the number of objects. For example, the first object may be one or more. Furthermore, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates an "or" relationship between the connected objects.

[0031] The following combination Figures 1-9 The present invention describes a method, system, electronic device and storage medium for direct data writing.

[0032] Figure 1 This is one of the flow charts of the data direct write method provided by the present invention, such as Figure 1 As shown, including but not limited to the following steps: Step 101: Write the data to be written into the buffer channel of the streaming acquisition framework.

[0033] In this embodiment, to achieve real-time cleansing and high-speed storage of streaming data, step 101 is first executed: acquiring the data to be written and writing it into the buffer channel of the streaming acquisition framework. This step, as the starting point of the entire direct data writing process, aims to uniformly convert the original unstructured or semi-structured streaming data into structured, standardized data to be written, and then connect it to the streaming data buffer system with a transaction mechanism to facilitate subsequent operations such as batch processing, dynamic routing, and database writing.

[0034] To ensure throughput and fault tolerance, the embodiment uses Apache Flume to build a streaming collection framework: KafkaSource is configured in the Flume task, so that the Source subscribes to the specified Topic and pulls messages in real time; each message will carry complete Kafka offset information before entering the Channel to support downstream transaction rollback.

[0035] Next, the data to be written is further written to the buffer channel (Channel) of the streaming collection framework (Flume). This embodiment prefers a combination of Memory Channel and File Channel: Memory Channel is used in general high-throughput scenarios to achieve near-millisecond write latency; when the Channel water level is detected approaching the threshold or a node failure occurs, the data is automatically switched to the File Channel to store on disk, ensuring "zero loss" disaster recovery. Flume has built-in transaction mechanisms at all three levels: Source, Channel, and Sink. This embodiment initiates transaction writes at the Source-Channel interaction point and commits the transaction after the Channel confirms successful persistence. Otherwise, the transaction is rolled back and retried, ensuring "connection is reliable."

[0036] By injecting data into the buffer channel, not only can the data processing process be decoupled, but it can also ensure that data is not discarded in abnormal situations such as the target database being temporarily unreachable or write failure, while providing a unified entry point for subsequent data batch construction and routing decisions.

[0037] In one possible implementation, Figure 2 This is the second flow chart of the data direct write method provided by the present invention, such as Figure 2 As shown, before step 101, the following steps are also included: Step 201: Receive original streaming data from a message queue.

[0038] Step 202: The original streaming data is input into the custom interceptor for at least one of field verification, time format conversion, or label generation to obtain data to be written.

[0039] In this embodiment, steps 201 and 202 are used to receive and preprocess the original streaming data. The purpose is to convert the unstructured, inconsistently formatted, and incomplete message queue data into structured and normalized data that meets the downstream batch processing and writing requirements, thereby ensuring the stability and efficiency of data batch construction, dynamic routing matching, and database writing in subsequent steps.

[0040] In step 201, the system configures a Kafka consumer component as a data access module to continuously receive raw streaming data from the message queue. This raw streaming data primarily originates from non-standard JSON data, such as operational status and user behavior logs, reported at a fixed frequency (e.g., every five seconds) by servers, clients, or third-party systems. This raw data has complex formats, diverse fields, and significant semantic differences. Directly feeding this data into subsequent batch processing or database interfaces can result in write failures, missing fields, and difficulty parsing data. Therefore, preprocessing is necessary immediately after receipt.

[0041] In step 202, the original streaming data is input into a custom interceptor configured in the streaming acquisition framework. The interceptor works in a chain structure and includes three processing units: a field verification module, a time format conversion module, and a label generation module. First, the field verification module determines whether the original data contains predefined key fields (such as "timestamp", "device_id", and "event_type"), and verifies the integrity of the fields using regular matching and field existence verification. For missing or illegal fields, a "status=invalid" tag is written to the event header for subsequent processes to discard or bypass as needed. Secondly, the time format conversion module standardizes the "timestamp" field and supports converting the ISO 8601 format (such as "2024-04-22T10:12:00Z") into the localized "yyyy-MM-dd HH:mm:ss" format to meet the target database's daily partitioned writing, query, and other requirements. Finally, the label generation module injects the business label field (for example, "event_class=user_behavior") based on the value of the "event_type" field through a lookup table or decision tree model. This label information will be used for fast matching of the target table in the dynamic routing stage.

[0042] After this processing, the data meets the requirements for writeability, with a complete structure, valid fields, a uniform format, and the necessary business semantic identifiers. This data can then be written directly to the buffer channel and participate in batch building, target table matching, and database request generation. This avoids the delays and resource overhead of secondary cleansing and secondary storage required in traditional solutions, improving overall system processing efficiency and accuracy.

[0043] Step 102: extracting data to be written from the buffer channel according to a preset batch size to form a data batch, and determining the target database table based on the information contained in the data batch.

[0044] In this embodiment, step 102 extracts the data to be written from the buffer channel and organizes it into data batches. Based on the information contained in the data batches, dynamic routing rules are used to determine the corresponding target database tables. This step aims to achieve structural batch organization and dynamic data diversion, improving the efficiency and accuracy of subsequent database writes and supporting complex business requirements such as concurrent writing to multiple tables and table partitioning by business dimension.

[0045] In one possible implementation, Figure 3 This is the third flow chart of the data direct write method provided by the present invention, such as Figure 3As shown, in step 102, extracting the data to be written from the buffer channel according to the preset batch size, and forming a data batch specifically includes the following steps: Step 301: Execute the transaction corresponding to the buffer channel.

[0046] Step 302: sequentially extracting data to be written in a quantity corresponding to a preset batch size from the buffer channel until the preset batch size is reached or the data in the buffer channel is empty.

[0047] Step 303: Temporarily store the extracted data to be written according to the transaction mechanism to form a data batch.

[0048] In this embodiment, steps 301 through 303 implement the transaction-based process of extracting data to be written from the buffer channel and forming data batches. This is a specific implementation of "extracting data to be written from the buffer channel according to a preset batch size to form data batches" in step 102. The core design of this process is to integrate the transaction control capabilities of the streaming ingestion framework to ensure rollback and consistency of the data extraction process, effectively preventing data loss or duplicate writes.

[0049] In step 301, the system starts a transaction corresponding to a buffered channel. A buffered channel is a component in streaming ingestion frameworks (such as Flume) that caches data events. It supports multiple channel types, including memory channels and file channels. To ensure data consistency during batch ingestion, this embodiment starts a transaction before channel read operations begin, ensuring that all data operations within the transaction's scope are controlled. The transaction mechanism in Flume is supported by the Transaction.begin() and Transaction.commit() / rollback() interfaces, offering low latency and high reliability.

[0050] After entering step 302, the system begins to sequentially extract the number of data to be written corresponding to the preset batch size from the buffer channel. During each extraction, the system obtains a piece of data to be written through the channel interface channel.take(); if the current channel is empty, the current extraction process is terminated early. This process continues until the number of extracted data pieces equals the preset batch size, or there is no more data to read in the channel, thus balancing throughput efficiency and real-time performance. In order to cope with the problem of uneven arrival of data sources, the system supports dynamic judgment of the availability of data in the current extraction window to avoid long-term blocking and waiting that causes batch processing delays.

[0051] In step 303, the system temporarily stores the extracted data to be written according to the transaction mechanism, forming a data batch. All events extracted in the current transaction are organized into an ordered set into a data batch, which serves as the basic processing unit for subsequent routing matching and database writes. This data batch is cached within the transaction context and is not immediately output or committed. This ensures that if an exception occurs in the subsequent processing logic, the batch can be fully rolled back, avoiding the inconsistency of "partial write success, partial failure."

[0052] By implementing steps 301 to 303, the system can build data batches with atomicity and consistency guarantees based on the transaction capabilities of the buffer channel, which not only achieves a smooth transition from streaming data to a batch processing structure, but also provides a structured and controllable data input source for subsequent route matching and database writing.

[0053] In one possible implementation, Figure 4 This is a fourth flow chart of the data direct write method provided by the present invention, such as Figure 4 As shown, in step 102, the target database table is determined based on the information contained in the data batch, which specifically includes the following steps: Step 401: Call the dynamic routing decision engine to match the message header information of each data in the data batch or the predefined fields in the message body. The dynamic routing decision engine is configured with at least one set of routing rules, each set of routing rules includes matching conditions and a database table corresponding to the matching conditions.

[0054] Step 402: When the data in the data batch meets the matching condition corresponding to a set of routing rules in the dynamic routing decision engine, the data is routed to the database table corresponding to the matching condition.

[0055] Step 403: When the data in the data batch meets the matching conditions of multiple sets of routing rules, the target database table is determined according to the preset priority.

[0056] In this embodiment, steps 401 through 403 determine the target database table for the extracted data batch. Specifically, dynamic routing rules are used to match each data item in the batch, providing an accurate table-level target for subsequent writes. This process is a core step in enabling concurrent writes to multiple tables and supporting complex business table sharding strategies, effectively improving write flexibility and system scalability.

[0057] In step 401, the system uses a pre-set dynamic routing decision engine to match the message header information or predefined fields in the message body of each data item in the data batch. The dynamic routing decision engine is rule-driven and maintains at least one set of routing rules. Each set of rules includes a matching condition and a corresponding database table identifier. The matching condition can be the value of an event header field (such as event_class and status) or the content of a field in the message body (such as device_id and event_type). It supports various matching methods, including regular expressions, ranges, and enumeration sets. During the batch traversal process, the system applies all routing rules to each data item to determine whether it meets a set of matching conditions.

[0058] In step 402, when a piece of data meets the matching criteria corresponding to a set of routing rules in the dynamic routing decision engine, the system routes the data to the database table corresponding to the matching criteria. This method of identifying the target table based on data content breaks through the limitations of traditional "fixed table" writing, enabling the system to process data for multiple businesses, multiple tenants, and multiple topics. For example, data with event_class equal to "user_behavior" will be routed to the user behavior log table, while data with event_class equal to "device_monitor" will be routed to the device monitoring table, achieving semantic-level traffic diversion.

[0059] To address real-world scenarios where the same piece of data might simultaneously meet multiple sets of routing rules, step 403 introduces a priority determination mechanism. Specifically, when a piece of data meets multiple matching criteria, the system sorts them according to the preconfigured priority attributes of the routing rules and selects the target database table with the highest priority as the final write destination. Priority is typically expressed as an integer, with lower values ​​indicating higher priority. This supports both static configuration and dynamic adjustment, ensuring a clear decision path when multiple matches occur.

[0060] By implementing steps 401 through 403, the system can dynamically partition data batches into tables at a granular level based on a flexible rule-based system, improving both the accuracy of data routing and the system's adaptability to complex business structures. Furthermore, the priority determination mechanism avoids the uncertainty associated with multiple matches, ensuring the uniqueness and controllability of each piece of data at the routing level, and providing a clear and defined target database table foundation for the construction and delivery of subsequent write requests.

[0061] In one possible implementation, Figure 5 This is the fifth flow chart of the data direct write method provided by the present invention, such as Figure 5 As shown, after step 403, the following steps are also included: Step 501: When the routing rules in the dynamic routing decision engine change, the routing rules are hot updated, wherein the dynamic routing decision engine or the streaming acquisition framework does not need to be restarted during the hot update.

[0062] In this embodiment, step 501 is used to perform a hot update on the routing rules when changes occur. This step is designed to address the issue of traditional routing configuration changes requiring a service restart for them to take effect, thereby ensuring the real-time adjustability of routing policies and the continuous availability of system services. By introducing a hot update mechanism, the system can dynamically adapt to changes in business logic, table structure adjustments, or tenant policy reorganizations without interrupting the data processing flow, thereby significantly improving the maintainability and flexibility of the data processing system.

[0063] In this embodiment, the system's built-in dynamic routing decision engine uses an external configuration-driven approach to maintain routing rules. Routing rules are stored in structured configuration files (such as JSON or YAML), which define matching conditions, target database table identifiers, and corresponding priority information. When business operations require the addition, modification, or deletion of a routing rule, system administrators can directly edit the configuration file, with the changes taking effect immediately through a hot update mechanism.

[0064] To achieve the above capabilities, the dynamic routing decision engine detects changes in configuration files in real time through a monitoring mechanism or a remote configuration push mechanism. Once a change event is detected, the system will immediately read the latest configuration content and atomically replace the existing routing rules to ensure that the routing rules are in a consistent state during the update process and do not affect the currently ongoing data batch routing calculations. In terms of specific implementation, version number control and double-buffer loading strategies can be used to load new rules in parallel while the old version of the rules is still in use. After the replacement is completed, it automatically switches to the new version to avoid the risk of temporary unavailability or missing rules during the update process.

[0065] Step 103: Based on the target database table, the data batch is constructed into a write request that complies with the target database batch write interface specification.

[0066] Figure 6 This is the sixth flow chart of the data direct write method provided by the present invention, such as Figure 6 As shown, step 103 specifically includes the following steps: Step 601: Select a serialization format corresponding to the data batch according to the batch write interface specification of the target database table.

[0067] In a possible implementation, the serialization format includes a CSV format and a CSV format with header information; wherein the CSV format uses a comma as a field separator and a line break as a record separator; the header information includes the column name and data type of the target database table.

[0068] In this embodiment, step 601 is used to select the serialization format corresponding to the data batch based on the batch write interface specification of the target database table. This step is designed to ensure that the data batch to be written is compatible with the batch write interface of the target database in terms of structure, field order, and data format. This avoids write failures, field misalignment, or data parsing errors caused by format mismatches, thereby improving the overall write success rate and stability.

[0069] In actual implementation, since the target database uses a batch-oriented write interface, it is usually required that the written data conform to a specific serialization format. To meet this requirement, before constructing the write request, the system first parses the field structure and data type of the table based on the information of the determined target database table, and then determines the serialization format that should be used for the current data batch. In this embodiment, the supported serialization formats include CSV format and CSV format with header information. The system will automatically select based on factors such as the complexity of the table structure, field stability, and whether explicit column name mapping is required.

[0070] When CSV is selected as the serialization format, the system concatenates the field values ​​of each data entry in the batch into a single line, following the preset field order. Fields are separated by commas, and records are separated by line breaks (\n), resulting in standard header-less CSV content. This format is suitable for scenarios where the field order is identical to the database table and the field meanings are clear, providing extremely high write efficiency.

[0071] If you select the CSV format with header information, the system appends a header line to the generated data. This header contains all the field names of the target database table and an optional data type description. This helps the database automatically map column names to fields when receiving data. This format is particularly suitable for scenarios where the order of fields may change or the database supports automatic field recognition, effectively improving write robustness and fault tolerance.

[0072] To ensure strict alignment between fields and the table structure, the system also performs field mapping and format verification during this step. By consulting the target table structure definition, the system automatically aligns data fields with database fields and formats special fields (such as dates and Boolean values) to ensure that the serialized data can be accurately parsed by the database. Furthermore, the system supports the completion or placeholder filling of null and missing fields to prevent write failures due to incomplete fields.

[0073] Step 602: Serialize the data batch according to the selected serialization format to obtain serialized batch data.

[0074] In this embodiment, step 602 is used to serialize the data batch according to the selected serialization format to obtain serialized batch data. This step is designed to convert the structured data batch into a standardized data string acceptable to the target database interface, thereby ensuring that the constructed write request has good parsability and data consistency, and avoiding problems such as write failures, field misalignment, or data type errors.

[0075] During implementation, the system first receives the serialization format determined in step 601, including CSV and CSV with header information. The system then initiates the serialization engine, sequentially transcoding each data record in the batch. To ensure that the field order is identical to the target database table, the system extracts the standard output order for each field based on the table structure configuration before serialization. This ensures that even if the key-value pair order in the original data varies, the field arrangement in serialization remains consistent.

[0076] When serializing in CSV format, the system extracts the field values ​​in each data entry as strings, using commas as delimiters between fields. Any special characters, such as commas, newlines, and double quotes, that may appear in the field values ​​are escaped. For example, field values ​​containing commas are automatically enclosed in double quotes to prevent them from being mistaken for field boundaries in the target database. Each data entry is serialized as a single line of text, and all data records are concatenated and separated by newlines (\n), resulting in a clearly structured and semantically consistent batch data text.

[0077] If you choose the CSV format with header information, the system adds a header line before the data above. This includes the column name (and optionally the field type) for each field, and the header fields are also separated by commas. This header line ensures automatic column name recognition and field mapping on the database side, and is suitable for applications with a large number of fields, where the order of fields may vary, or when multiple tables reuse the same data model. This format further enhances the self-describing ability and compatibility of write requests, especially when the target database has "Header-Aware" mode enabled, which directly eliminates the need for field alignment configuration.

[0078] Step 603: Construct a write request that complies with the target database batch write interface specification based on the serialized batch data. The write request includes a request header and a request body. The request header includes authentication information, the target database table name, the request type, and the data format information. The request body includes the serialized batch data.

[0079] In this embodiment, step 603 constructs a write request based on the serialized batch data that complies with the target database's batch write interface specification. This write request includes a request header and a request body. This step is designed to encapsulate the formatted batch data into a request message that can be directly parsed and received by the target database. This ensures the integrity, accuracy, and interface compatibility of the data written, thereby improving the success rate of database write operations and the system's throughput.

[0080] In practice, the system first receives the serialized batch data generated in step 602 as the core content for constructing the request body. To ensure that the write request has clear semantic boundaries and operational context during transmission and parsing, the system divides the entire write request into two components: the request header and the request body, in accordance with the target database's batch write interface specification.

[0081] The request header contains several key fields that instruct the target database how to handle the current write task. Specifically, they include: Authentication information: such as username, password, or access token, used to ensure the legal identity of the write request; Target database table name: identifies the database table corresponding to this write operation to ensure that the data is written to the correct location; Request type: usually set to "insert" or "batch-insert", indicating that this is a batch insert operation; Data format information: Indicates the serialization format (such as CSV or CSV-with-header) and its encoding method to facilitate format recognition and field parsing by the database.

[0082] The above request header fields are organized in a key-value pair format and carried in the HTTP / 2 protocol message header to ensure efficient transmission, high compression rate and flow control capabilities.

[0083] The request body directly carries the serialized batch data generated in step 602. When constructing the request body, the system ensures that the data structure is complete and strictly conforms to the format specified in the request header. For CSV formats with header information, the system includes both the header and the data rows in the request body. For standard CSV formats, only the data record rows are included, and field alignment relies on the format description in the request header.

[0084] To ensure the legitimacy of the overall request structure and parsing accuracy, the system performs an integrity check on the write request after construction. This includes verifying the completeness of header fields, the validity of the target table name, and the consistency of the data format declaration and content. If any issues are found, the request is terminated and an error log is recorded. The completed write request serves as input for the subsequent send operation in step 604, and the connection pool module is responsible for the actual transmission.

[0085] Step 104: Send the write request to the target database table through the connection pool of the preset transmission protocol.

[0086] In this embodiment, step 104 sends the write request to the target database table via a connection pool using a pre-defined transport protocol. This step is designed to achieve high-throughput, low-latency, and structured data writes, resolving issues such as multiple connection establishments, format mismatches, and data inconsistencies in traditional solutions. This allows for stable writes of large-scale streaming data to analytical database systems such as StarRocks.

[0087] In a possible implementation, the preset transmission protocol is the HTTP / 2 protocol, the connection pool is a connection pool constructed according to the HTTP / 2 protocol, the connection pool is preset with multiple concurrent connections, and adopts a connection multiplexing strategy.

[0088] In this embodiment, the transmission channel configuration for write requests is determined. Specifically, the transmission protocol is preset to HTTP / 2 and a connection pool based on this protocol is established. The connection pool is preset with multiple concurrent connections and adopts a connection reuse strategy. This step is designed to address the problems of low connection reuse rate, weak concurrency, and high write jitter in traditional write channels, thereby improving the network transmission efficiency and overall system throughput of data write requests in high-frequency batches.

[0089] During implementation, the system pre-sets the transport protocol to HTTP / 2. Compared to traditional HTTP / 1.1, HTTP / 2 supports multiplexing at the connection management level, allowing multiple requests and responses to be sent concurrently within a single TCP connection. This significantly reduces the latency and resource overhead associated with frequent connection establishment and release. Especially during batch data writes, HTTP / 2's flow control and header compression mechanisms effectively improve network utilization and ensure stable performance during high-concurrency writes.

[0090] The connection pool, built based on the HTTP / 2 protocol, serves as a key support component for write request transmission and is created during system initialization. The connection pool maintains a number of reusable persistent connections. The system's default connection pool supports at least 100 concurrent connections, and the actual size can be automatically expanded or manually configured and tuned based on business load. The connection pool's scheduling logic uses a connection reuse strategy. When idle connections exist, existing connections are prioritized for request transmission. A new connection establishment process is initiated only when no connections are available in the connection pool. This reuse strategy avoids the resource waste of repeatedly establishing connections for each request and improves the real-time nature of request issuance.

[0091] Further, an available connection according to the HTTP / 2 protocol is obtained from the connection pool, and a write request is sent to the target database through the available connection.

[0092] In this embodiment, the setting of this step is intended to achieve efficient delivery of the constructed write request at the network transport layer, fully utilize the concurrency and multiplexing capabilities of the connection pool, reduce connection establishment overhead, and improve the throughput and stability of batch data writing. It is particularly suitable for real-time warehousing scenarios of large-scale streaming data.

[0093] Specifically, after the system completes the write request construction in step 603, it calls the connection pool management module to initiate the request send operation. This connection pool is built based on the HTTP / 2 protocol and features multiplexing, connection sharing, and flow control support. The connection pool is configured during system initialization, with multiple concurrent connection channels preset (e.g., no fewer than 100). A connection status monitoring mechanism continuously tracks the availability and load of each connection.

[0094] Before executing a send operation, the system first retrieves an available HTTP / 2 connection from the connection pool. This connection must be idle or reusable, and its corresponding number of stream channels must not have reached the protocol limit. If multiple connections are available, the system will select the connection based on the least loaded connection first, to avoid overloading hot connections and affecting overall sending performance.

[0095] After acquiring a connection, the system immediately sends the completed write request through the HTTP / 2 connection channel. During the transmission process, the request header and request body are encapsulated as complete HTTP / 2 frames and sent frame by frame. The protocol layer's frame compression mechanism and stream priority control mechanism enable the concurrent transmission of multiple requests with extremely low latency. Especially in scenarios where multiple batches of data are sent continuously, the connection pool's reuse mechanism allows multiple batches to share the same TCP connection, avoiding resource jitter caused by repeated connection establishment and release, significantly improving transmission stability and overall bandwidth utilization.

[0096] In one possible implementation, Figure 7 This is the seventh flow chart of the data direct write method provided by the present invention, such as Figure 7 As shown, after step 104, the following steps are also included: Step 701: Obtain the response result of the target database.

[0097] Step 702: When the response result indicates that the write request fails to be sent, the write request is resent to the target database table according to the exponential backoff mechanism until the sending is successful or the number of resends reaches the sending number threshold.

[0098] Step 703: When the number of resends reaches the sending number threshold, a retry exception log is generated and the corresponding data batch is returned to the buffer channel.

[0099] In this embodiment, steps 701 through 703 process the target database's response after a write request is sent. If an exception occurs, retries are performed using an exponential backoff mechanism until the retry threshold is reached. If necessary, the data batch is rolled back to the buffer channel. This processing mechanism is designed to ensure that write operations are self-recovering in complex network environments or when the target database responds abnormally, effectively improving the system's fault tolerance and write success rate, and preventing data loss and write failures caused by network outages or high concurrency conflicts.

[0100] In step 701, after completing the write request (see step 604), the system will synchronously or asynchronously obtain a response from the target database. This response typically includes a status code, confirmation information, or an error description. When the database returns a status code indicating a successful write (e.g., HTTP status code 200 or internal status code "OK"), the system marks the current data batch as complete and releases the connection resources and transaction context used. However, if the received response indicates a write failure (e.g., due to a connection timeout, interface exception, or database throttling), the next step in the retry process is triggered.

[0101] In step 702, the system uses an exponential backoff mechanism to resend the write request to the target database table according to a preset strategy. Exponential backoff is a classic network conflict resolution strategy. Its basic principle is that after a write request fails, instead of retrying immediately, an exponentially increasing waiting time is set based on the number of failures n. The specific retry delay time calculation formula is: ; in, is the retry delay time, is the initial wait time (e.g., 100ms), and n is the current number of retries. Before each retry, the system enters a wait state. After the delay expires, it retrieves an available connection from the connection pool and attempts to send the original write request. This process minimizes the load on the target database and reduces the risk of a sudden burst of requests failing. The system also maintains a retry counter, which is cleared immediately after a successful request. If the maximum number of retries (e.g., 3) is reached without success, the system enters the exception handling phase.

[0102] To ensure data reliability and business continuity, in step 703, the system will trigger the fault handling process when the number of retransmissions reaches the transmission threshold. Specifically, the system first generates a retry exception log, recording the batch ID of the failed request, the reason for failure, the response status code, the retry process and other information for subsequent manual analysis or alarm system access; then, the system will return the current data batch to the buffer channel, that is, rewrite it to the Flume Channel, and mark it as data to be retried to ensure that the batch will not be lost or the processing link will not be interrupted due to errors. When the next round of transactions is started, the system will give priority to processing such rollback batches to achieve batch-level fault-tolerant reprocessing.

[0103] By implementing steps 701 to 703, the present invention establishes a comprehensive write failure handling mechanism that effectively balances system stability and write performance in high-concurrency, large-scale, real-time write scenarios, enhancing the adaptability of the data transmission link. This mechanism provides the system with robustness against occasional write anomalies, significantly reducing the risk of data loss due to network fluctuations or database jitter, and is an important supplementary means of ensuring accurate and efficient data writing.

[0104] Furthermore, after completing the write request and response processing (i.e., steps 701 to 703), this application also requires archiving the status of the current write operation and tracking its progress. This step aims to establish a complete write result observability mechanism, resolving the "write-only" or "no-notice" write failure issues in traditional data pipelines, thereby improving the system's transparency, monitorability, and traceability.

[0105] Specifically, after each data batch is written, the system records the write context information for the current batch in a dedicated tracking log or database table. This recorded information includes, but is not limited to: the unique identifier of the data batch, the name of the target database table, the routing match result, the write request build time, the actual write start and end times, the HTTP response status code, whether to retry, the final write result (success / failure), and, in the case of failure, the error cause and number of retries. This information is aggregated by the write scheduling module and can be reported in real time via the log system or external monitoring platforms (such as Prometheus and ELK) for exception alerts and operational status visualization.

[0106] When the system detects that a batch has failed to be written and the maximum number of retries has been triggered (see step 703), the system marks the batch's tracking record status as "failed pending" and adds its batch ID to the abnormal batch backtracking queue. This queue is regularly scanned by a dedicated background task thread. Using manual intervention strategies or pre-set retry strategies, the data content of the failed batch is reloaded and injected into the buffer channel again, ensuring that critical data is not permanently lost due to transient failures.

[0107] Reference Figure 8 , Figure 8 This is a schematic diagram of the structure of the data direct write system provided by the present invention, the system includes: A first processing module is used to write the data to be written into the buffer channel of the streaming acquisition framework; A second processing module is used to extract the data to be written from the buffer channel according to a preset batch size to form a data batch, and determine the target database table based on the information contained in the data batch; A third processing module is used to construct the data batch into a write request that complies with the target database batch write interface specification according to the target database table; The fourth processing module is configured to send the write request to the target database table through a connection pool of a preset transmission protocol.

[0108] In a possible implementation, the second processing module is further configured to: Execute the transaction corresponding to the buffered channel; Sequentially extract the data to be written in a quantity corresponding to the preset batch size from the buffer channel until the preset batch size is reached or the data in the buffer channel is empty; According to the transaction mechanism, the extracted data to be written is temporarily stored to form data batches.

[0109] In a possible implementation, the second processing module is further configured to: The dynamic routing decision engine is called to match the message header information or predefined fields in the message body of each data in the data batch. The dynamic routing decision engine is configured with at least one set of routing rules. Each set of routing rules includes matching conditions and a database table corresponding to the matching conditions. When the data in the data batch meets the matching conditions corresponding to a set of routing rules in the dynamic routing decision engine, the data is routed to the database table corresponding to the matching conditions; When the data in a data batch meets the matching conditions of multiple sets of routing rules, the target database table is determined according to the preset priority.

[0110] In a possible implementation, the second processing module is further configured to: When the routing rules in the dynamic routing decision engine change, the routing rules are hot updated, wherein the dynamic routing decision engine or the streaming collection framework does not need to be restarted during the hot update.

[0111] In a possible implementation, the third processing module is further configured to: Select the serialization format corresponding to the data batch according to the batch write interface specification of the target database table; Serialize the data batches according to the selected serialization format to obtain serialized batch data; A write request that complies with the target database batch write interface specification is constructed based on the serialized batch data. The write request includes a request header and a request body. The request header includes authentication information, the target database table name, the request type, and the data format information. The request body includes the serialized batch data.

[0112] In a possible implementation, the fourth processing module is further configured to: Get the response result of the target database; When the response result indicates that the write request fails to be sent, the write request is resent to the target database table according to the exponential backoff mechanism until it is sent successfully or the number of resends reaches the sending threshold; When the number of resends reaches the send threshold, a retry exception log is generated and the corresponding data batch is returned to the buffer channel.

[0113] In a possible implementation, the first processing module is further configured to: Receive raw streaming data from the message queue; The original streaming data is input into the custom interceptor for at least one of field verification, time format conversion, or label generation to obtain the data to be written.

[0114] It should be noted that the data direct write system provided by the present invention can execute the data direct write method of any of the above embodiments during specific operation, which will not be described in detail in this embodiment.

[0115] Figure 9 Schematic diagram of the structure of the electronic device provided by the present invention, such as Figure 9 As shown, the electronic device may include: a processor 910 (processor), a communication interface 920 (CommunicationsInterface), a memory 930 (memory) and a communication bus 940, wherein the processor 910, the communication interface 920, and the memory 930 communicate with each other via the communication bus 940. The processor 910 may call the logic instructions in the memory 930 to execute a data direct write method, which includes: writing the data to be written into the buffer channel of the streaming acquisition framework; extracting the data to be written from the buffer channel according to a preset batch size to form a data batch, and determining the target database table based on the information contained in the data batch; constructing the data batch into a write request that complies with the target database batch write interface specification based on the target database table; and sending the write request to the target database table through a connection pool of a preset transmission protocol.

[0116] Furthermore, the logic instructions in the aforementioned memory 930 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the portion that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product, stored in a storage medium, includes instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, a mobile hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0117] On the other hand, the present invention also provides a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by a computer, the computer can execute the data direct writing method provided by the above embodiments, which includes: writing the data to be written into the buffer channel of the streaming acquisition framework; extracting the data to be written from the buffer channel according to a preset batch size to form a data batch, and determining the target database table based on the information contained in the data batch; constructing the data batch into a write request that complies with the target database batch write interface specification based on the target database table; and sending the write request to the target database table through the connection pool of the preset transmission protocol.

[0118] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, it is implemented to execute the data direct writing method provided by the above-mentioned embodiments, the method comprising: writing the data to be written into the buffer channel of the streaming acquisition framework; extracting the data to be written from the buffer channel according to a preset batch size to form a data batch, and determining the target database table based on the information contained in the data batch; constructing the data batch into a write request that complies with the target database batch write interface specification based on the target database table; and sending the write request to the target database table through a connection pool of a preset transmission protocol.

[0119] The system embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units. That is, they may be located in one place or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of this embodiment. Persons of ordinary skill in the art will be able to understand and implement the present invention without inventive effort.

[0120] Through the description of the above embodiments, those skilled in the art will clearly understand that each embodiment can be implemented using software plus a necessary general-purpose hardware platform, or of course, hardware. Based on this understanding, the essence of the above technical solution, or the portion that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a magnetic disk, or an optical disk, and includes a number of instructions for causing a computer device (such as a personal computer, server, or network device) to execute the methods of each embodiment or certain portions of the embodiments.

[0121] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A data direct writing method, characterized in that: include: Write the data to be written into the buffer channel of the streaming acquisition framework; Extracting the data to be written from the buffer channel according to a preset batch size to form a data batch, and determining a target database table based on information contained in the data batch; According to the target database table, constructing the data batch into a write request that complies with the target database batch write interface specification; The write request is sent to the target database table through a connection pool with a preset transmission protocol.

2. The data direct write method according to claim 1, characterized in that: The buffer channel of the streaming acquisition framework is a buffer channel with a transaction mechanism; extracting the data to be written from the buffer channel according to a preset batch size to form a data batch includes: Executing a transaction corresponding to the buffer channel; Sequentially extracting a quantity of data to be written corresponding to the preset batch size from the buffer channel until the preset batch size is reached or the data in the buffer channel is empty; The data to be written is temporarily stored and extracted according to the transaction mechanism to form the data batch.

3. The data direct writing method according to claim 1, wherein: Determining the target database table based on the information included in the data batch includes: Invoking a dynamic routing decision engine to match predefined fields in message header information or message body of each data in the data batch, wherein the dynamic routing decision engine is configured with at least one set of routing rules, each set of routing rules including a matching condition and a database table corresponding to the matching condition; When the data in the data batch meets the matching condition corresponding to a set of routing rules in the dynamic routing decision engine, the data is routed to the database table corresponding to the matching condition; When the data in the data batch meets the matching conditions of multiple groups of routing rules, the target database table is determined according to a preset priority.

4. The data direct write method according to claim 3, characterized in that: Also includes: When the routing rules in the dynamic routing decision engine change, the routing rules are hot updated, wherein the dynamic routing decision engine or the streaming acquisition framework does not need to be restarted during the hot update.

5. The data direct writing method according to claim 1, wherein: The step of constructing the data batch into a write request that complies with a target database batch write interface specification according to the target database table includes: Selecting a serialization format corresponding to the data batch according to the batch write interface specification of the target database table; Serializing the data batch according to the selected serialization format to obtain serialized batch data; A write request that complies with the target database batch write interface specification is constructed based on the serialized batch data, wherein the write request includes a request header and a request body, wherein the request header includes authentication information, a target database table name, a request type, and data format information, and the request body includes the serialized batch data.

6. The data direct writing method according to claim 5, characterized in that: The serialization format includes CSV format and CSV format with header information; wherein, the CSV format uses commas as field separators and line breaks as record separators; the header information includes the column names and data types of the target database table.

7. The data direct writing method according to claim 1, characterized in that: After the connection pool of the preset transmission protocol sends the write request to the target database table, the method further includes: Obtaining a response result from the target database; When the response result indicates that the write request fails to be sent, resending the write request to the target database table according to an exponential backoff mechanism until the sending is successful or the number of resends reaches a sending number threshold; When the number of resends reaches the sending number threshold, a retry exception log is generated and the corresponding data batch is returned to the buffer channel.

8. The data direct writing method according to claim 1, characterized in that: Before writing the data to be written into the buffer channel of the streaming acquisition framework, the method further includes: Receive raw streaming data from the message queue; The original streaming data is input into a custom interceptor for at least one of field verification, time format conversion, or label generation to obtain the data to be written.

9. A data direct write system, characterized in that: include: A first processing module is used to write the data to be written into the buffer channel of the streaming acquisition framework; a second processing module, configured to extract the data to be written from the buffer channel according to a preset batch size to form a data batch, and determine a target database table based on information contained in the data batch; A third processing module is configured to construct the data batch into a write request that complies with a target database batch write interface specification according to the target database table; A fourth processing module is configured to send the write request to the target database table through a connection pool of a preset transmission protocol.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that: When the processor executes the computer program, the direct data writing method according to any one of claims 1 to 8 is implemented.

Images