A streaming routing file output method and device based on expression partitioning and dynamic schema, equipment and medium

Abstract

The application discloses a streaming routing file output method and device based on expression partition and dynamic Schema, equipment and medium, relates to the computer technical field, and includes: obtaining the output configuration of a streaming job, and parsing its directory expression into a directory strategy chain composed of constants, field values and mapping conversion strategies; for input records, based on the write data field, the object to be written is extracted, the target output directory is obtained by splicing according to the directory strategy chain, and the target Schema is determined according to the Schema strategy; the object to be written, the target Schema and the target file path are aggregated to the metadata cache with the target output directory as the key, and when the cumulative record number reaches the threshold or the checkpoint is triggered, they are converted into file bytes according to the target Schema and written into the distributed file system, and the cache is rebuilt after writing. Flexible and scalable dynamic routing file output can be realized in the streaming job, and memory control and abnormal isolation are considered.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and in particular to a method, apparatus, device, and medium for streaming routed file output based on expression partitioning and dynamic schema. Background Technology

[0002] With the rapid development of real-time computing technology, streaming data output scenarios are becoming increasingly complex. In data lake and lakeware warehouse integrated architectures, streaming jobs not only need to write data to fixed directories, but also frequently need to dynamically determine the output path based on business attributes. For example, in the scenario of IoT device data entering the lake, tens of thousands of sensors in a factory report data in real time. The data formats and access frequencies of different product lines, different workshops, and even different equipment models vary greatly. Data from high-value core equipment needs to be written to high-performance storage to support real-time alarms, while data from ordinary equipment needs to be written to low-cost storage for offline analysis. Therefore, streaming jobs need to dynamically generate output directories according to the hierarchy of "product line / equipment model / region / date" during writing, so as to automatically distribute the data according to equipment type and avoid all data being mixed in the same directory, which would lead to a full table scan during queries. For example, in the context of e-commerce omni-channel user behavior attribution, user behavior tracking data generated across applications, mini-programs, web pages, and offline stores is uniformly collected into the same streaming pipeline. To support refined operational analysis by channel and terminal, the streaming job needs to split the data and write it to different directories according to the hierarchy of "channel platform / application version / event type / date" during the writing process. This allows downstream applications to only read the corresponding subdirectories when querying specific behaviors under a certain terminal or version, without having to scan the entire dataset. It is evident that the output path in the above scenario includes both fixed constant fragments and variable fragments taken from record fields. Furthermore, it may be necessary to convert field values ​​into corresponding directory fragments based on business semantics. Moreover, the same job often needs to output data to multiple tables and multiple partitions simultaneously. As a mainstream real-time computing engine, Flink's native FileSink component provides basic file writing and file visibility after checkpointing. However, when faced with the aforementioned dynamic routing output requirements, users usually need to extend BucketAssigner and RollingPolicy themselves. However, such extensions are often limited to fixed formats and limited partitioning rules, making it difficult to uniformly manage dynamic directory expressions, multi-schema switching, and the partition commit status of newly added tables at runtime.

[0003] Meanwhile, streaming input data typically contains multiple event types, each with different data structures. If the entire Sink uses only a single fixed schema, field semantics will be lost or a large number of redundant fields will be created. Conversely, building a separate Sink node for each event type will lead to job topology expansion, increased resource overhead, and more complex configuration and maintenance. Furthermore, in actual production environments, the checkpoint interval for streaming jobs is often set based on overall latency and fault tolerance requirements, and may not match the data backlog rate. When relying solely on checkpoint trigger files for writing, the amount of data to be written aggregated by path within the Sink component may increase dramatically during sudden surges in traffic or long checkpoint intervals, significantly increasing the risk of memory overflow. For abnormal data, without a unified degradation handling mechanism, a single record parsing failure or write anomaly may cause the entire job to fail or pollute the normal output directory, affecting the stability of the data link. In Hive multi-table dynamic routing output scenarios, the target table may appear dynamically during job execution. If partition commit triggers, policy chains, and task tracking states cannot be maintained independently for each table, it is easy to cause partition omissions or duplicate commits. In Hadoop clusters with Kerberos authentication enabled, long-running jobs also face the problem of write failures caused by expired authentication tickets.

[0004] As can be seen from the above, how to achieve flexible, secure, and scalable dynamic routing file output in streaming jobs, while taking into account memory control, exception isolation, and long-term operational stability, is an urgent problem to be solved. Summary of the Invention

[0005] In view of this, the purpose of this invention is to provide a method, apparatus, device, and medium for outputting streaming routing files based on expression partitioning and dynamic schema, enabling flexible, secure, and scalable dynamic routing file output in streaming operations, while also ensuring controllable memory, anomaly isolation, and long-term operational stability. The specific solution is as follows: Firstly, this application provides a method for outputting streaming route files based on expression partitioning and dynamic schema, including: Obtain the output configuration of the streaming job, and scan the directory expression in the output configuration character by character to parse the directory expression into a directory strategy chain consisting of constant strategy, field value strategy and mapping conversion strategy; For the input records received by the streaming job, the corresponding objects to be written are extracted from the input records based on the write data field in the output configuration. The input records are then retrieved and concatenated according to the order of the directory policy chain to obtain the target output directory corresponding to the input records. The target schema corresponding to the objects to be written is determined based on the schema policy in the output configuration. Using the target output directory as the key, the object to be written, the target schema, and the target file path generated based on the target output directory are aggregated into the metadata cache. When the cumulative number of records processed by the streaming job reaches a preset threshold or the streaming job triggers a checkpoint, the object to be written in the metadata cache is converted into target format file bytes according to the corresponding target schema so as to be written to the corresponding target file path in the distributed file system. After the writing is completed, the metadata cache is rebuilt.

[0006] Optionally, the step of scanning the directory expression in the output configuration character by character to parse the directory expression into a directory strategy chain consisting of constant strategy, field value strategy, and mapping conversion strategy includes: The directory expression in the output configuration is scanned character by character to parse the ordinary text in the directory expression into a constant strategy carrying corresponding constant fragments, to parse the content in the directory expression surrounded by preset field identifiers into a field value strategy carrying corresponding value fields, and to parse the content in the directory expression surrounded by preset function identifiers into a mapping conversion strategy carrying corresponding mapping conversion functions and corresponding input parameter fields. The constant strategy, the field value strategy, and the mapping conversion strategy are organized in the order of their appearance in the directory expression to obtain the corresponding directory strategy chain.

[0007] Optionally, the step of retrieving and concatenating the values ​​of the input records according to the order of the directory strategy chain to obtain the target output directory corresponding to the input records includes: The constant strategy, the field value retrieval strategy, and the mapping conversion strategy are executed sequentially according to the order of the directory strategy chain to obtain the corresponding directory fragments; The obtained directory fragments are concatenated according to the order of the directory strategy chain to obtain the target output directory corresponding to the input record; Specifically, when executing the constant strategy, the constant fragment carried by the constant strategy is used as the corresponding directory fragment; when executing the field value retrieval strategy, the field value in the input record corresponding to the value field is used as the corresponding directory fragment; when executing the mapping conversion strategy, the mapping conversion function is called with the field value in the input record corresponding to the input parameter field as the input parameter, and the mapping result obtained from the call is used as the corresponding directory fragment.

[0008] Optionally, determining the target schema corresponding to the object to be written based on the schema strategy in the output configuration includes: If the Schema strategy in the output configuration is a routing Schema strategy, then the data definition class name corresponding to the input record is obtained by concatenating the type identifier field in the input record, or the event code field, event type, and package name in the output configuration. The fields contained in the data definition class corresponding to the data definition class name and the parent class of the data definition class are read by reflection. After filtering the preset exclusion fields in the read fields, the corresponding Avro Schema is generated as the target Schema. The target Schema is cached with the data definition class name as the key so that the cached target Schema can be reused when the data definition class name is obtained again. If the schema strategy in the output configuration is a custom schema strategy, then the unified Avro schema in the output configuration will be used as the target schema; The preset exclusion fields include fields with modifiers such as static, transient, final, or local, as well as preset non-business fields.

[0009] Optionally, the method for outputting streaming routing files based on expression partitioning and dynamic schema further includes: If the extraction of the object to be written fails, the data definition class corresponding to the data definition class name does not exist, the type of the fields contained in the data definition class is not supported, or the target file path is written, then the input record or the object to be written is converted into an error record containing the error field. Using the default error directory in the output configuration as the key, the error records and the degradation schema containing only the error fields are aggregated into the error metadata cache. This is so that when the cumulative number of records processed by the streaming job reaches the preset threshold or the streaming job triggers a checkpoint, the error records are converted into target format file bytes according to the degradation schema and written to the file path in the distributed file system corresponding to the default error directory.

[0010] Optionally, when the cumulative number of records processed by the streaming job reaches a preset threshold or the streaming job triggers a checkpoint, the objects to be written in the metadata cache are converted into target format file bytes according to the corresponding target schema so as to be written to the corresponding target file path in the distributed file system, and the metadata cache is rebuilt after the writing is completed, including: If the cumulative number of records processed by the streaming job reaches a preset threshold, a write operation is actively triggered to convert the objects to be written in the metadata cache into target format file bytes according to the corresponding target schema and write them to the corresponding target file path in the distributed file system. If the streaming job triggers a checkpoint, when the output configuration indicates that the preset authentication mechanism is enabled, it is determined whether the time elapsed since the last authentication renewal exceeds the authentication renewal period in the output configuration. If it does, authentication renewal is performed based on the preset key table file, and the write operation is triggered after the authentication renewal. The filename of the file obtained through the write operation is generated based on the timestamp of the checkpoint and the subtask number of the streaming job. After the write operation is completed, regardless of whether the write operation was successful or not, the metadata cache and the error metadata cache are cleared and rebuilt.

[0011] Optionally, the method for outputting streaming routing files based on expression partitioning and dynamic schema further includes: If the target output directory is the partition directory corresponding to the Hive routing table, then during the partition submitter initialization, the set of existing routing table names is restored from the operator state of the streaming job. If the routing table name corresponding to the file obtained through the write operation does not belong to the set of existing routing table names, then an independent partition commit trigger, partition commit policy and task tracker are created for the Hive routing table corresponding to the routing table name, and the routing table name is added to the set of existing routing table names. Upon receiving the partition commit information, the partition commit operation is executed asynchronously based on the partition commit trigger and partition commit policy of the routing table name corresponding to the partition commit information. When the streaming job triggers a checkpoint, the state of the set of existing routing table names and the state of the partition commit trigger are saved to the operator state.

[0012] Secondly, this application provides a streaming route file output device based on expression partitioning and dynamic schema, comprising: The expression parsing module is used to obtain the output configuration of the streaming job and scan the directory expression in the output configuration character by character to parse the directory expression into a directory strategy chain consisting of constant strategy, field value strategy and mapping conversion strategy; The recording routing module is used to extract the corresponding object to be written from the input record received by the streaming job based on the write data field in the output configuration, extract the value of the input record and concatenate it according to the order of the directory policy chain to obtain the target output directory corresponding to the input record, and determine the target schema corresponding to the object to be written based on the schema policy in the output configuration. The cache rebuilding module is used to aggregate the object to be written, the target schema, and the target file path generated based on the target output directory into the metadata cache, using the target output directory as the key. When the cumulative number of records processed by the streaming job reaches a preset threshold or the streaming job triggers a checkpoint, the object to be written in the metadata cache is converted into target format file bytes according to the corresponding target schema so as to be written to the corresponding target file path in the distributed file system. After the writing is completed, the metadata cache is rebuilt.

[0013] Thirdly, this application provides an electronic device, comprising: Memory, used to store computer programs; A processor is configured to execute the computer program to implement the aforementioned method for outputting streaming routed files based on expression partitioning and dynamic schema.

[0014] Fourthly, this application provides a computer-readable storage medium for storing a computer program, wherein the computer program, when executed by a processor, implements the aforementioned method for streaming route file output based on expression partitioning and dynamic schema.

[0015] This application provides a streaming route file output method based on expression partitioning and dynamic schema, comprising: obtaining the output configuration of a streaming job, and scanning the directory expression in the output configuration character by character to parse the directory expression into a directory policy chain composed of constant policy, field value retrieval policy and mapping conversion policy; for the input record received by the streaming job, extracting the corresponding object to be written from the input record based on the write data field in the output configuration, retrieving and concatenating the values ​​of the input record according to the order of the directory policy chain to obtain the target output directory corresponding to the input record, and determining the target schema corresponding to the object to be written based on the schema policy in the output configuration; using the target output directory as the key, aggregating the object to be written, the target schema and the target file path generated based on the target output directory into a metadata cache, and when the cumulative number of records processed by the streaming job reaches a preset number threshold or the streaming job triggers a checkpoint, converting the object to be written in the metadata cache into target format file bytes according to the corresponding target schema so as to write to the corresponding target file path in the distributed file system, and rebuilding the metadata cache after writing is completed.

[0016] As can be seen from the above, this application generates the target output directory by parsing the directory expression into three strategies: constants, field values, and asset conversion, and combining them in sequence. This allows the output path to be expressed by a mixture of constants, upstream record fields, and business asset mapping functions, significantly improving the expressive power of routing rules compared to a fixed-path scheme that only partitions by bucket. By maintaining metadata caching by target output directory within the same sink and dynamically generating and caching schemas by event type, data from different paths and different data patterns can be aggregated and written out separately within the same job, avoiding topology bloat and configuration redundancy caused by creating a separate sink for each event. By using a dual-trigger disk write-to-database approach with quantity thresholds and checkpoints, the visibility of files during the checkpoint phase is guaranteed while effectively suppressing the continuous growth of aggregated data in memory. Thus, flexible, secure, and scalable dynamic routing file output can be achieved in streaming jobs, while also ensuring controllable memory, exception isolation, and long-term operational stability. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0018] Figure 1This is a flowchart of a streaming routing file output method based on expression partitioning and dynamic schema disclosed in this invention; Figure 2 This is a schematic diagram of a streaming routing file output device based on expression partitioning and dynamic schema disclosed in this invention; Figure 3 This is a structural diagram of an electronic device disclosed in this invention. Detailed Implementation

[0019] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0020] With the rapid development of real-time computing technology, streaming data output scenarios are becoming increasingly complex. In data lake and lakeware warehouse architectures, streaming jobs not only need to write data to fixed directories, but also frequently need to dynamically determine the output path based on business attributes. For example, data from different business lines or different event types need to be written to their respective partition directories, and even within the same job, data needs to be output to multiple tables and multiple partitions simultaneously. As a mainstream real-time computing engine, Flink's native FileSink component provides basic file writing and file visibility after checkpointing. However, when facing the above-mentioned dynamic routing output requirements, users usually need to extend BucketAssigner and RollingPolicy themselves. However, such extensions are often limited to fixed formats and limited partitioning rules, making it difficult to uniformly manage dynamic directory expressions, multi-schema switching, and the partition commit status of newly added tables at runtime. At the same time, streaming input data usually contains multiple event types, and the data structures corresponding to different types are different. If the entire Sink uses only a single fixed schema, field semantics will be lost or a large number of redundant fields will be generated; if a separate Sink node is built for each event type, it will lead to job topology expansion, increased resource overhead, and complex configuration and maintenance. Furthermore, in real-world production environments, the checkpoint interval for streaming jobs is often set based on overall latency and fault tolerance requirements, and may not match the data backlog rate. When relying solely on checkpoint-triggered file writing, the amount of data to be written aggregated by path within the Sink component can increase dramatically during sudden surges in traffic or long checkpoint intervals, significantly increasing the risk of memory overflow. For abnormal data, without a unified degradation handling mechanism, a single record parsing failure or write anomaly can lead to the failure of the entire job or contaminate the normal output directory, affecting the stability of the data chain. In Hive multi-table dynamic routing output scenarios, the target table may dynamically appear during job execution. If partition commit triggers, policy chains, and task tracking states cannot be maintained independently for each table, it can easily lead to missed or duplicate partition commits. In Hadoop clusters with Kerberos authentication enabled, long-running jobs also face the problem of write failures due to expired authentication tickets. To address these issues, this application provides a streaming routing file output scheme based on expression partitioning and dynamic schema, which enables flexible, secure, and scalable dynamic routing file output in streaming jobs, while also ensuring memory control, anomaly isolation, and long-term operational stability.

[0021] See Figure 1 As shown in the figure, this application discloses a method for outputting streaming routing files based on expression partitioning and dynamic schema, including: Step S11: Obtain the output configuration of the streaming job, and scan the directory expression in the output configuration character by character to parse the directory expression into a directory strategy chain consisting of constant strategy, field value strategy and mapping conversion strategy.

[0022] The solution provided in this application can be generally composed of a routing sink component, a directory expression parser, a dynamic schema generator, an in-memory metadata cache, a resource write operator, an error data degradation channel, and a Hive dynamic partition committer. Specifically, the routing sink component is responsible for record access, cache aggregation, dual-trigger disk persistence, and authentication renewal; the directory expression parser and dynamic directory / bucket allocation unit are responsible for converting directory expressions into executable policy chains and calculating the target output path per record at runtime; the dynamic schema generator is responsible for generating and caching the data schema required by the target format according to event type; the error data degradation channel is responsible for uniformly writing various abnormal records to the default error directory; and the Hive dynamic partition committer is responsible for multi-table partition commits at runtime. All of the above components work collaboratively within the same streaming job to achieve dynamic selection of directories, data schemas, and commit strategies per record, and control memory usage and file visibility through a dual-trigger approach of quantity thresholds and checkpoints.

[0023] In this embodiment, when the streaming job starts and the routing Sink component completes initialization, the component reads all the output configurations required for this output at once. These output configurations include, but are not limited to, directory expressions, default error directories, write data fields, schema policies, event code fields, event types, package names, preset quantity thresholds, and authentication renewal periods. Specifically, the directory expression describes the composition rules of the target output directory; the default error directory is used to handle various exception records; the write data field indicates which field of the input record to extract the actual business data to be written to disk; the schema policy indicates whether a dynamically generated or uniformly fixed data pattern is used; the event code field, event type, and package name are used together to locate the data definition class when dynamically generating the data pattern; the preset quantity threshold controls the timing of proactive disk writing; and the authentication renewal period controls the refresh frequency of authentication tickets under the controlled cluster. These configuration items are parsed once during the job startup phase and reside within the Sink component, and are not parsed repeatedly during runtime, thus avoiding the performance overhead caused by repeatedly parsing configurations record by record.

[0024] Furthermore, the step of scanning the directory expression in the output configuration character by character to parse the directory expression into a directory strategy chain consisting of constant strategies, field value strategies, and mapping conversion strategies may include: scanning the directory expression in the output configuration character by character to parse the ordinary text in the directory expression into a constant strategy carrying corresponding constant fragments, parse the content in the directory expression surrounded by preset field identifiers into a field value strategy carrying corresponding value fields, and parse the content in the directory expression surrounded by preset function identifiers into a mapping conversion strategy carrying corresponding mapping conversion functions and corresponding input parameter fields; organizing the constant strategies, field value strategies, and mapping conversion strategies in the order of their appearance in the directory expression to obtain the corresponding directory strategy chain. In other words, the directory expression parser scans sequentially starting from the first character of the expression. When it encounters plain text not surrounded by any identifiers, it encapsulates the text as is into a constant strategy, which carries the corresponding constant fragment. When it encounters content surrounded by preset field identifiers, it identifies the content within the identifiers as value fields and encapsulates it into a field value strategy. When it encounters content surrounded by preset function identifiers, it identifies the first term within the identifier as a mapping transformation function, and the remaining terms as the input parameter fields of the function, and encapsulates them into a mapping transformation strategy. After scanning, the parser organizes the three types of strategies into an ordered directory strategy chain according to the order in which they appear in the directory expression.

[0025] It is worth mentioning that the mapping conversion function is used to handle scenarios beyond constants and field values ​​that require business semantic conversion to obtain directory fragments, such as mapping the cluster name and event code to obtain the corresponding package name or directory name. It is understood that the specific implementation of the mapping conversion function, besides using a business asset conversion function, can also be replaced by dictionary mapping, user-defined functions, rule engines, or remote metadata service queries, as long as it can return a definite directory fragment based on the input parameter field values; no specific limitations are made here. This embodiment externalizes the path concatenation logic originally coupled in the code into a configurable expression by parsing the directory expression into an ordered policy chain composed of three types of policies in one go. This allows the output path to be expressed by a mixture of constants, upstream record fields, and business mapping functions, significantly improving the expressive power of routing rules while reducing the parsing overhead of the expression.

[0026] Step S12: For the input record received by the streaming job, extract the corresponding object to be written from the input record based on the write data field in the output configuration, extract the value of the input record and concatenate it according to the order of the directory policy chain to obtain the target output directory corresponding to the input record, and determine the target schema corresponding to the object to be written based on the schema policy in the output configuration.

[0027] In this embodiment, after each input record arrives at the routing Sink component, the Sink first extracts the actual object to be written to disk from the input record based on the write data field in the output configuration. The object to be written can be a JSON object or a Map object. Subsequently, the Sink executes the aforementioned directory policy chain with the input record as the context, thereby obtaining the target output directory to which the record belongs. Specifically, the step of retrieving and concatenating values ​​from the input record according to the order of the directory strategy chain to obtain the target output directory corresponding to the input record may include: sequentially executing the constant strategy, the field value retrieval strategy, and the mapping transformation strategy according to the order of the directory strategy chain to obtain corresponding directory fragments; concatenating the obtained directory fragments according to the order of the directory strategy chain to obtain the target output directory corresponding to the input record; wherein, when executing the constant strategy, the constant fragment carried by the constant strategy is used as the corresponding directory fragment; when executing the field value retrieval strategy, the field value in the input record corresponding to the value field is used as the corresponding directory fragment; when executing the mapping transformation strategy, the mapping transformation function is called with the field value in the input record corresponding to the input parameter field as the input parameter, and the mapping result obtained from the call is used as the corresponding directory fragment. In other words, each policy in the policy chain produces a directory fragment at runtime. Constant policies directly return the constant fragment they carry, field-retrieving policies read the value of a specified field from the current input record, and mapping / transformation policies call the mapping / transformation function with the input parameter field values ​​from the current input record and return the mapping result. The Sink concatenates the directory fragments end-to-end according to the original order of the policy chain to obtain the target output directory corresponding to that record. Since the execution of the policy chain is at the granularity of a single record, the same Sink can produce different target output directories for different records at the same time, thereby achieving dynamic routing by record.

[0028] Furthermore, the Sink also needs to determine the target schema corresponding to the object to be written. Specifically, determining the target schema corresponding to the object to be written based on the schema strategy in the output configuration may include: if the schema strategy in the output configuration is a routing schema strategy, then concatenating the type identifier field in the input record, or the event code field, event type, and package name in the output configuration, to obtain the data definition class name corresponding to the input record; reading the fields contained in the data definition class corresponding to the data definition class name and the parent class of the data definition class through reflection; filtering the preset exclusion fields in the read fields to generate the corresponding AvroSchema as the target schema; and caching the target schema with the data definition class name as the key so that the cached target schema can be reused when the data definition class name is obtained again; if the schema strategy in the output configuration is a custom schema strategy, then the unified Avro Schema in the output configuration is used as the target schema; wherein, the preset exclusion fields include fields with modifiers of static, transient, final, or local, as well as preset non-business fields. In other words, under the routing schema strategy, the Sink prioritizes reading the type identifier field (e.g., the @type field) from the input record to directly obtain the data definition class name. If the record does not have a type identifier field, it concatenates the package name from the output configuration as a prefix, the value corresponding to the event code field as the body, and the event type as a suffix to obtain the corresponding data definition class name. After obtaining the class name, the Sink loads the data definition class via reflection and reads all fields declared by the class and its parent classes along its inheritance chain. The read fields are then filtered, removing fields with modifiers such as static, transient, final, or local, as well as pre-defined non-business fields such as tracing identifiers and cache mappings. Only fields with business semantics are retained, and the corresponding Avro schema is generated as the target schema according to the field type mapping relationship. To avoid the performance overhead caused by repeatedly triggering reflection for the same type of record, the Sink caches the generated target schema in the schema cache using the data definition class name as the key. The cached result is reused when the same class name is obtained again. Under the custom schema strategy, the Sink no longer performs class name inference and reflection, but directly adopts the unified Avro schema given in the output configuration as the target schema to adapt to scenarios where the upstream data structure is completely consistent or where the user has already completed schema governance. It is understood that the target schema can be obtained not only through class reflection, but also through schema registry centers, metadata centers, or interface queries, etc., without being specifically limited here.

[0029] Furthermore, to prevent a single abnormal record from blocking the entire data link, this embodiment also sets up a unified error data degradation channel. Specifically, the streaming routing file output method based on expression partitioning and dynamic schema may further include: if the extraction of the object to be written fails, the data definition class corresponding to the data definition class name does not exist, the type of the fields contained in the data definition class is not supported, or the target file path writing fails, then the input record or the object to be written is converted into an error record containing error fields; using the default error directory in the output configuration as the key, the error record and the degradation schema containing only the error fields are aggregated into the error metadata cache, so that when the cumulative number of records processed by the streaming job reaches the preset number threshold or the streaming job triggers a checkpoint, the error record is converted into target format file bytes according to the degradation schema and written to the file path corresponding to the default error directory in the distributed file system. In other words, when situations such as single data parsing failure, non-existent data definition class, unsupported field type, or target path writing failure occur, the Sink does not interrupt the job. Instead, it serializes the original input record or the extracted object to be written as a whole and places it in an error field, forming an error record containing only that error field. This error record is then aggregated into an independent error metadata cache using the default error directory as the key. The data schema corresponding to this error metadata cache is a downgraded schema containing only the error field, so regardless of how abnormal the structure of the original record is, it can be serialized and written to disk normally. It is understood that, in addition to being written to the default error directory, the error record can also be sent to downstream systems such as message queues, alarm systems, or audit tables for subsequent anomaly analysis and data supplementation; no specific limitations are made here. This embodiment completes path calculation, data schema selection, and anomaly degradation within the same Sink, enabling data of different event types to be correctly distinguished and protected within a single job topology. This avoids the topology bloat and configuration redundancy caused by creating a separate Sink for each event, and ensures that normal data output is not contaminated or blocked by individual abnormal records.

[0030] Step S13: Using the target output directory as the key, aggregate the object to be written, the target schema, and the target file path generated based on the target output directory into the metadata cache. When the cumulative number of records processed by the streaming job reaches a preset threshold or the streaming job triggers a checkpoint, convert the object to be written in the metadata cache into target format file bytes according to the corresponding target schema so as to write it into the corresponding target file path in the distributed file system. After writing is completed, rebuild the metadata cache.

[0031] In this embodiment, the Sink maintains an in-memory metadata cache using the target output directory as the key. Each value in the cache is a file metadata object, which contains the target file path generated based on the target output directory, a data list consisting of several objects to be written, and a class name or schema identifier used to identify the data pattern. Accordingly, records falling under the same target output directory are aggregated into the same batch of file content, while records from different target output directories and different data patterns are aggregated separately without interference, thus enabling the batch writing of multi-path, multi-pattern data to disk in a single write operation.

[0032] Furthermore, the step of converting the objects to be written in the metadata cache into target format file bytes according to the corresponding target schema when the cumulative number of records processed by the streaming job reaches a preset threshold or the streaming job triggers a checkpoint, so as to write them to the corresponding target file path in the distributed file system, and rebuilding the metadata cache after the write operation is completed, may include: if the cumulative number of records processed by the streaming job reaches a preset threshold, actively triggering a write operation to convert the objects to be written in the metadata cache into target format file bytes according to the corresponding target schema and write them to the corresponding target file path in the distributed file system; if the streaming job triggers a checkpoint, when the output configuration indicates that a preset authentication mechanism is enabled, determining whether the time elapsed since the last authentication renewal exceeds the authentication renewal period in the output configuration, if it exceeds, performing authentication renewal based on a preset key table file, and triggering the write operation after authentication renewal, and generating the filename of the file obtained through the write operation based on the timestamp of the checkpoint and the subtask number of the streaming job; after the write operation is completed, regardless of whether the write operation is successful, clearing and rebuilding the metadata cache and the error metadata cache. In other words, this embodiment employs a dual-trigger disk write mechanism involving both quantity thresholds and checkpoints: On one hand, the Sink counts the cumulative number of processed records, and once a preset quantity threshold is reached, it proactively performs a write operation, converting the data in each path cache into target format file bytes according to their respective target schemas and writing them to the distributed file system; on the other hand, when a streaming job triggers a checkpoint, a write operation is also performed, and a filename is generated by combining the timestamp of the checkpoint with the current subtask number. This ensures the uniqueness of the filename under the same parallelism and makes the written file visible externally after the checkpoint is completed. Specifically, in scenarios where the output configuration indicates the activation of a preset authentication mechanism, the Sink will first determine in the checkpoint process whether the time elapsed since the last authentication renewal exceeds the configured authentication renewal period. If it does, authentication renewal is performed based on the preset key table file (e.g., calling the corresponding ticket check and re-login interface), and then the write operation is performed, thereby preventing long-running jobs from failing to write due to expired authentication tickets. After the write operation is completed, regardless of success or failure, the Sink will clear and rebuild the metadata cache and error metadata cache, releasing memory in a timely manner and preparing for the aggregation of the next batch of data. It is understood that, in addition to columnar formats such as Parquet, the target format can also be replaced with Avro, ORC, JSON, or other columnar or row-based file formats; the distributed file system can be a distributed file storage system or an object storage system; the preset quantity threshold can also be extended to a byte size threshold, a time threshold, a path-level threshold, or a mixture of the above thresholds, and no specific limitation is made here.

[0033] Furthermore, for scenarios where the target output directory points to a Hive routing table partition, this embodiment also provides a runtime-expandable multi-table partition submission mechanism. Specifically, the streaming route file output method based on expression partitioning and dynamic schema may further include: if the target output directory is a partition directory corresponding to a Hive routing table, then during partition submitter initialization, the set of existing routing table names is restored from the operator state of the streaming job; if the routing table name corresponding to the file obtained through the write operation does not belong to the set of existing routing table names, then an independent partition submission trigger, partition submission policy, and task tracker are created for the Hive routing table corresponding to the routing table name, and the routing table name is added to the set of existing routing table names; after receiving partition submission information, the partition submission operation is asynchronously executed based on the partition submission trigger and partition submission policy of the routing table name corresponding to the partition submission information, and when the streaming job triggers a checkpoint, the state of the set of existing routing table names and the state of the partition submission trigger are saved to the operator state. In other words, during the initialization phase, the partition committer first recovers the set of route table names that the job has previously encountered from the operator state, thus ensuring the continuation of the original commit context after fault recovery or job restart. During operation, when it is detected that the route table to which a certain written file belongs is not yet in the set, the partition committer creates an independent set of partition commit triggers, partition commit policies, and task trackers for that table as needed, and registers the table name into the set, so that the target table list does not need to be fully pre-configured before job startup. When a partition commit message is received, the partition committer selects the trigger and policy specific to that table based on the route table name corresponding to the message and executes the partition commit asynchronously, thus preventing blocking when multiple tables coexist. At each checkpoint, the partition committer saves the set of already encountered route table names and the state of each trigger to the operator state, ensuring that the partition commit progress can be persisted along with the job state. It can be understood that the partition commit policy can be a successful file commit policy, a metadata service commit policy, a custom commit policy, or a policy chain composed of multiple policies mentioned above, without specific limitations here. This embodiment effectively avoids the problems of missing and duplicate partition commits caused by adding new tables at runtime by using table-isolated commit contexts and stateful table name sets.

[0034] In one specific implementation, the aforementioned scenario of IoT device data entering the lake is used as an example. When the job starts, the output configuration read by the routing Sink component contains the directory expression "oss: / / iot-lake / #productLine# / #deviceType# / ${getStorageTierdeviceType} / dt=#collectDate# / ", the data field to be written is payload, the schema strategy is the routing schema strategy, and the default error directory is "oss: / / iot-lake / _error / ". After scanning character by character, the directory expression parser parses ordinary text such as "oss: / / iot-lake / " and " / dt=" into constant strategies, parses productLine, deviceType, and collectDate (enclosed by field identifiers) into field value strategies, and parses content enclosed by function identifiers into mapping strategies using getStorageTier as the mapping transformation function and deviceType as the input parameter field, organizing them into a directory strategy chain in the order of appearance. getStorageTier is used to query device file assets based on device model and return the storage level corresponding to that model. High-value core devices return "hot," and ordinary devices return "cold." The directories corresponding to these two types are configured with different storage media and lifecycle strategies in object storage, thus achieving hot and cold separation. When a temperature sensor report arrives, the Sink extracts the object to be written based on the write data field and executes the strategy chain sequentially to obtain the target output directory. Simultaneously, based on the record's type identifier field, it obtains the data definition class name, and through reflection and field filtering, generates an Avro Schema containing only business fields as the target Schema, which is then cached by class name. A vibration sensor report is routed to the target output directory, whose target schema contains fields completely different from the former. It is evident that the same sink handles routing and schema determination for two types of records with different paths and data structures, eliminating the need for a separate sink node for each type of device. If a record fails to process due to missing fields or a non-existent data definition class, it is downgraded to an error record and written to the default error directory, without affecting the output of normal data. These two types of records are aggregated in the metadata cache using their respective target output directories as keys. When the cumulative number of processed records reaches a preset threshold or a job triggers a checkpoint, they are converted to target format file bytes according to their respective target schemas and written out. After writing, the cache is cleared and rebuilt. It is understood that this scenario is only one specific application of this application. For example, in an e-commerce full-domain user behavior attribution scenario, only the directory expression and other configuration items need to be adjusted to allow the same sink to split and write data according to channel platform, application version, and event type; the remaining steps are the same and will not be elaborated here.

[0035] As can be seen from the above, this application embodiment parses the directory expression character by character into constant strategies, field value strategies, and mapping transformation strategies, and organizes them into an ordered strategy chain. This allows the output path to be expressed by a mixture of constants, upstream record fields, and business mapping functions, significantly improving the expressive power of routing rules compared to a fixed path scheme that only partitions by bucket. By maintaining metadata caching within the same sink using the target output directory as the key, and dynamically locating the data definition class based on the type identifier field or event code, event type, and package name, and generating and caching the target schema through reflection and field filtering, data from different paths and different data modes can be aggregated and written out separately within the same job, reducing the topology bloat and configuration maintenance costs caused by multi-sink splitting. Through quantity The threshold and checkpoint dual-trigger disk persistence mechanism effectively suppresses the continuous growth of aggregated data in memory while ensuring file visibility during the checkpoint phase, reducing the risk of memory overflow under sudden traffic and long checkpoint cycles. A unified error data degradation channel writes various abnormal records to the default error directory using a degradation schema, preventing a single abnormal record from blocking normal data output or polluting the normal directory. By creating independent commit triggers, commit policies, and task trackers for newly added routing tables at runtime, and incorporating table name sets and trigger states into operator states, runtime scalability and fault recovery for multi-table partitioned commits are achieved. Embedding authentication renewal into the checkpoint process ensures the write stability of long-running jobs in a controlled cluster environment. This enables flexible, secure, and scalable dynamic routing file output in streaming jobs, while also ensuring memory control, anomaly isolation, and long-term operational stability.

[0036] See Figure 2 As shown in the figure, this application discloses a streaming routing file output device based on expression partitioning and dynamic schema, including: The expression parsing module 11 is used to obtain the output configuration of the streaming job and scan the directory expression in the output configuration character by character to parse the directory expression into a directory strategy chain consisting of constant strategy, field value strategy and mapping conversion strategy; The routing module 12 is used to extract the corresponding object to be written from the input record based on the write data field in the output configuration for the input record received by the streaming job, extract the value of the input record and concatenate it according to the order of the directory policy chain to obtain the target output directory corresponding to the input record, and determine the target schema corresponding to the object to be written based on the schema policy in the output configuration. The cache reconstruction module 13 is used to aggregate the object to be written, the target schema, and the target file path generated based on the target output directory into the metadata cache using the target output directory as the key. When the cumulative number of records processed by the streaming job reaches a preset threshold or the streaming job triggers a checkpoint, the object to be written in the metadata cache is converted into target format file bytes according to the corresponding target schema so as to be written to the corresponding target file path in the distributed file system. After the writing is completed, the metadata cache is reconstructed.

[0037] In some specific embodiments, the expression parsing module 11 may specifically include: The strategy parsing unit is used to scan the directory expression in the output configuration character by character, so as to parse the ordinary text in the directory expression into a constant strategy carrying the corresponding constant fragment, parse the content in the directory expression surrounded by a preset field identifier into a field value strategy carrying the corresponding value field, and parse the content in the directory expression surrounded by a preset function identifier into a mapping conversion strategy carrying the corresponding mapping conversion function and the corresponding input parameter field. The strategy chain organization unit is used to organize the constant strategy, the field value strategy, and the mapping transformation strategy in the order of their appearance in the directory expression to obtain the corresponding directory strategy chain.

[0038] In some specific embodiments, the routing recording module 12 may specifically include: The directory fragment generation unit is used to execute the constant strategy, the field value retrieval strategy, and the mapping conversion strategy sequentially according to the directory strategy chain to obtain corresponding directory fragments; wherein, when executing the constant strategy, the constant fragment carried by the constant strategy is used as the corresponding directory fragment; when executing the field value retrieval strategy, the field value in the input record corresponding to the value field is used as the corresponding directory fragment; when executing the mapping conversion strategy, the mapping conversion function is called with the field value in the input record corresponding to the input parameter field as the input parameter, and the mapping result obtained from the call is used as the corresponding directory fragment; The directory splicing submodule is used to splice the obtained directory fragments according to the order of the directory strategy chain to obtain the target output directory corresponding to the input record; The first schema determination unit is configured to, if the schema strategy in the output configuration is a routing schema strategy, concatenate the type identifier field in the input record, or the event code field, event type, and package name in the output configuration to obtain the data definition class name corresponding to the input record, read the fields contained in the data definition class corresponding to the data definition class name and the parent class of the data definition class through reflection, filter the preset exclusion fields in the read fields to generate the corresponding Avro schema as the target schema, and cache the target schema with the data definition class name as the key so that the cached target schema can be reused when the data definition class name is obtained again; The second schema determination unit is used to take the unified Avro schema in the output configuration as the target schema if the schema strategy in the output configuration is a custom schema strategy. The preset exclusion fields include fields with modifiers such as static, transient, final, or local, as well as preset non-business fields.

[0039] In some specific embodiments, the cache reconstruction module 13 may specifically include: The threshold triggering unit is used to actively trigger a write operation if the cumulative number of records processed by the streaming job reaches a preset threshold, so as to convert the objects to be written in the metadata cache into target format file bytes according to the corresponding target schema and write them to the corresponding target file path in the distributed file system. The checkpoint triggering unit is used to determine whether the time elapsed since the last authentication renewal exceeds the authentication renewal period in the output configuration when the output configuration indicates that the preset authentication mechanism is enabled, if the streaming job triggers a checkpoint, and if so, to perform authentication renewal based on the preset key table file, and trigger the write operation after the authentication renewal, and to generate the filename of the file obtained by the write operation based on the timestamp of the checkpoint and the subtask number of the streaming job. The cache cleanup unit is used to clear and rebuild the metadata cache and the error metadata cache after the write operation is completed, regardless of whether the write operation was successful.

[0040] In some specific embodiments, the streaming route file output device based on expression partitioning and dynamic schema may further include: An error record generation unit is used to convert the input record or the object to be written into an error record containing error fields if the extraction of the object to be written fails, the data definition class corresponding to the data definition class name does not exist, the type of the fields contained in the data definition class is not supported, or the target file path is written fails. The error data degradation unit is used to aggregate the error records and the degradation schema containing only the error fields into the error metadata cache, using the default error directory in the output configuration as the key. This is so that when the cumulative number of records processed by the streaming job reaches the preset threshold or the streaming job triggers a checkpoint, the error records are converted into target format file bytes according to the degradation schema and written to the file path in the distributed file system corresponding to the default error directory. The state recovery unit is used to recover the set of existing route table names from the operator state of the streaming job when the partition submitter is initialized if the target output directory is the partition directory corresponding to the Hive routing table. The context creation unit is used to create an independent partition commit trigger, partition commit policy and task tracker for the Hive route table corresponding to the route table name if the route table name corresponding to the file obtained by the write operation does not belong to the set of existing route table names, and to add the route table name to the set of existing route table names. The partition submission unit is used to asynchronously execute a partition submission operation based on the partition submission trigger and partition submission policy of the routing table name corresponding to the partition submission information after receiving the partition submission information, and to save the set of existing routing table names and the state of the partition submission trigger to the operator state when the streaming job triggers a checkpoint.

[0041] Furthermore, embodiments of this application also disclose an electronic device, Figure 3 This is a structural diagram of an electronic device 20 according to an exemplary embodiment. The content of the diagram should not be construed as limiting the scope of this application. The electronic device 20 may specifically include: at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input / output interface 25, and a communication bus 26. The memory 22 stores a computer program, which is loaded and executed by the processor 21 to implement the relevant steps in the expression-partitioned and dynamic schema-based streaming routing file output method disclosed in any of the foregoing embodiments. Furthermore, the electronic device 20 in this embodiment may specifically be a computer.

[0042] In this embodiment, the power supply 23 is used to provide operating voltage for each hardware device on the electronic device 20; the communication interface 24 can create a data transmission channel between the electronic device 20 and external devices, and the communication protocol it follows can be any communication protocol applicable to the technical solution of this application, and is not specifically limited here; the input / output interface 25 is used to acquire external input data or output data to the outside world, and its specific interface type can be selected according to specific application needs, and is not specifically limited here.

[0043] In addition, the memory 22, as a carrier for resource storage, can be a read-only memory, random access memory, disk or optical disk, etc. The resources stored thereon can include operating system 221, computer program 222, etc., and the storage method can be temporary storage or permanent storage.

[0044] The operating system 221 is used to manage and control the various hardware devices on the electronic device 20 and the computer program 222, which may be Windows Server, Netware, Unix, Linux, etc. In addition to including a computer program capable of performing the expression-partitioned and dynamic schema-based streaming route file output method executed by the electronic device 20 as disclosed in any of the foregoing embodiments, the computer program 222 may further include computer programs capable of performing other specific tasks.

[0045] Furthermore, this application also discloses a computer-readable storage medium for storing a computer program; wherein, when the computer program is executed by a processor, it implements the aforementioned disclosed method for streaming route file output based on expression partitioning and dynamic schema. Specific steps of this method can be found in the corresponding content disclosed in the foregoing embodiments, and will not be repeated here.

[0046] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to in the method section.

[0047] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0048] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented directly by hardware, a software module executed by a processor, or a combination of both. The software module can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.

[0049] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0050] The technical solutions provided in this application have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A method for streaming routed file output based on expression partitioning and dynamic schema, characterized in that, include: Obtain the output configuration of the streaming job, and scan the directory expression in the output configuration character by character to parse the directory expression into a directory strategy chain consisting of constant strategy, field value strategy and mapping conversion strategy; For the input records received by the streaming job, the corresponding objects to be written are extracted from the input records based on the write data field in the output configuration. The input records are then retrieved and concatenated according to the order of the directory policy chain to obtain the target output directory corresponding to the input records. The target schema corresponding to the objects to be written is determined based on the schema policy in the output configuration. Using the target output directory as the key, the object to be written, the target schema, and the target file path generated based on the target output directory are aggregated into the metadata cache. When the cumulative number of records processed by the streaming job reaches a preset threshold or the streaming job triggers a checkpoint, the object to be written in the metadata cache is converted into target format file bytes according to the corresponding target schema so as to be written to the corresponding target file path in the distributed file system. After the writing is completed, the metadata cache is rebuilt.

2. The streaming route file output method based on expression partitioning and dynamic schema according to claim 1, characterized in that, The step of scanning the directory expression in the output configuration character by character to parse the directory expression into a directory strategy chain consisting of constant strategy, field value strategy, and mapping conversion strategy includes: The directory expression in the output configuration is scanned character by character to parse the ordinary text in the directory expression into a constant strategy carrying corresponding constant fragments, to parse the content in the directory expression surrounded by preset field identifiers into a field value strategy carrying corresponding value fields, and to parse the content in the directory expression surrounded by preset function identifiers into a mapping conversion strategy carrying corresponding mapping conversion functions and corresponding input parameter fields. The constant strategy, the field value strategy, and the mapping conversion strategy are organized in the order of their appearance in the directory expression to obtain the corresponding directory strategy chain.

3. The streaming route file output method based on expression partitioning and dynamic schema according to claim 2, characterized in that, The step of retrieving and concatenating values ​​from the input records according to the order of the directory strategy chain to obtain the target output directory corresponding to the input records includes: The constant strategy, the field value retrieval strategy, and the mapping conversion strategy are executed sequentially according to the order of the directory strategy chain to obtain the corresponding directory fragments; The obtained directory fragments are concatenated according to the order of the directory strategy chain to obtain the target output directory corresponding to the input record; Specifically, when executing the constant strategy, the constant fragment carried by the constant strategy is used as the corresponding directory fragment; when executing the field value retrieval strategy, the field value in the input record corresponding to the value field is used as the corresponding directory fragment; when executing the mapping conversion strategy, the mapping conversion function is called with the field value in the input record corresponding to the input parameter field as the input parameter, and the mapping result obtained from the call is used as the corresponding directory fragment.

4. The streaming route file output method based on expression partitioning and dynamic schema according to claim 1, characterized in that, The step of determining the target schema corresponding to the object to be written based on the schema strategy in the output configuration includes: If the Schema strategy in the output configuration is a routing Schema strategy, then the data definition class name corresponding to the input record is obtained by concatenating the type identifier field in the input record, or the event code field, event type, and package name in the output configuration. The fields contained in the data definition class corresponding to the data definition class name and the parent class of the data definition class are read by reflection. After filtering the preset exclusion fields in the read fields, the corresponding Avro Schema is generated as the target Schema. The target Schema is cached with the data definition class name as the key so that the cached target Schema can be reused when the data definition class name is obtained again. If the schema strategy in the output configuration is a custom schema strategy, then the unified Avro schema in the output configuration will be used as the target schema; The preset exclusion fields include fields with modifiers such as static, transient, final, or local, as well as preset non-business fields.

5. The streaming route file output method based on expression partitioning and dynamic schema according to claim 4, characterized in that, Also includes: If the extraction of the object to be written fails, the data definition class corresponding to the data definition class name does not exist, the type of the fields contained in the data definition class is not supported, or the target file path is written, then the input record or the object to be written is converted into an error record containing the error field. Using the default error directory in the output configuration as the key, the error records and the degradation schema containing only the error fields are aggregated into the error metadata cache. This is so that when the cumulative number of records processed by the streaming job reaches the preset threshold or the streaming job triggers a checkpoint, the error records are converted into target format file bytes according to the degradation schema and written to the file path in the distributed file system corresponding to the default error directory.

6. The streaming route file output method based on expression partitioning and dynamic schema according to claim 5, characterized in that, When the cumulative number of records processed by the streaming job reaches a preset threshold or the streaming job triggers a checkpoint, the objects to be written in the metadata cache are converted into target format file bytes according to the corresponding target schema so as to be written to the corresponding target file path in the distributed file system, and the metadata cache is rebuilt after the writing is completed, including: If the cumulative number of records processed by the streaming job reaches a preset threshold, a write operation is actively triggered to convert the objects to be written in the metadata cache into target format file bytes according to the corresponding target schema and write them to the corresponding target file path in the distributed file system. If the streaming job triggers a checkpoint, when the output configuration indicates that the preset authentication mechanism is enabled, it is determined whether the time elapsed since the last authentication renewal exceeds the authentication renewal period in the output configuration. If it does, authentication renewal is performed based on the preset key table file, and the write operation is triggered after the authentication renewal. The filename of the file obtained through the write operation is generated based on the timestamp of the checkpoint and the subtask number of the streaming job. After the write operation is completed, regardless of whether the write operation was successful or not, the metadata cache and the error metadata cache are cleared and rebuilt.

7. The streaming route file output method based on expression partitioning and dynamic schema according to claim 6, characterized in that, Also includes: If the target output directory is the partition directory corresponding to the Hive routing table, then during the partition submitter initialization, the set of existing routing table names is restored from the operator state of the streaming job. If the routing table name corresponding to the file obtained through the write operation does not belong to the set of existing routing table names, then an independent partition commit trigger, partition commit policy and task tracker are created for the Hive routing table corresponding to the routing table name, and the routing table name is added to the set of existing routing table names. Upon receiving the partition commit information, the partition commit operation is executed asynchronously based on the partition commit trigger and partition commit policy of the routing table name corresponding to the partition commit information. When the streaming job triggers a checkpoint, the state of the set of existing routing table names and the state of the partition commit trigger are saved to the operator state.

8. A streaming route file output device based on expression partitioning and dynamic schema, characterized in that, include: The expression parsing module is used to obtain the output configuration of the streaming job and scan the directory expression in the output configuration character by character to parse the directory expression into a directory strategy chain consisting of constant strategy, field value strategy and mapping conversion strategy; The recording routing module is used to extract the corresponding object to be written from the input record received by the streaming job based on the write data field in the output configuration, extract the value of the input record and concatenate it according to the order of the directory policy chain to obtain the target output directory corresponding to the input record, and determine the target schema corresponding to the object to be written based on the schema policy in the output configuration. The cache rebuilding module is used to aggregate the object to be written, the target schema, and the target file path generated based on the target output directory into the metadata cache, using the target output directory as the key. When the cumulative number of records processed by the streaming job reaches a preset threshold or the streaming job triggers a checkpoint, the object to be written in the metadata cache is converted into target format file bytes according to the corresponding target schema so as to be written to the corresponding target file path in the distributed file system. After the writing is completed, the metadata cache is rebuilt.

9. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor for executing the computer program to implement the streaming route file output method based on expression partitioning and dynamic schema as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, Used to store computer programs, wherein the computer programs, when executed by a processor, implement the streaming route file output method based on expression partitioning and dynamic schema as described in any one of claims 1 to 7.

Images