Write method, write device, processor and stream processing platform of topic

Abstract

This application provides a method, device, processor, and stream processing platform for writing topics. The method includes: acquiring incoming data from the stream processing platform to obtain target data; generating multiple records based on the target data, each record including a partition field, the partition field representing the correspondence between the record and a topic on the stream processing platform; invoking the current transaction and writing the multiple records to their corresponding topics according to the partition fields, with each record corresponding to a topic one-to-one. In this method, since the partition field can represent the correspondence between the records and the topics on the stream processing platform, multiple topic writing can be completed using only one connector. Each connector only needs to create one node, which significantly reduces resource consumption compared to the prior art where one node is created for each topic, thus solving the problem of excessive resource consumption during multi-topic writing in the prior art.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of topic writing, in particular, to a topic writing method, a writing device, a computer readable storage medium, a processor and a stream processing platform. BACKGROUND

[0002] In Kafka, if multiple topic records are to be written at the same time, these records can be submitted through the same transaction. The Flink native Kafka connector will use Kafka transactions to ensure single record precise once writing only in the Semantic.EXACTLY_ONCE mode, but does not support writing multiple topics at the same time. Currently, to achieve Flink multi-topic writing, multiple sinks can be added, one sink corresponding to one Kafka connector, and the method steps are as follows:

[0003] (1) One topic corresponds to the creation of one Kafka connector FlinkKafkaProducer;

[0004] (2) Add the created FlinkKafkaProducer to the task topology one by one through the addSink() method of DataStream.

[0005] The multi-sink solution can send a record to multiple topics, but this solution has many disadvantages:

[0006] (1) Cannot guarantee strong consistency between topics. Multi-topic writing is not in the same transaction, and cannot guarantee that writing is successful or failed at the same time, so it cannot guarantee strong consistency between topics;

[0007] (2) Reduces task processing performance. If there are thousands of topics, then thousands of sinks need to be created. For Flink, adding a node also means that Flink needs to allocate corresponding resources to run the task corresponding to this node, and at the same time, data needs to be routed to the sink, and network transmission is also needed in the middle. Therefore, creating thousands of sinks requires a large amount of resources, which will greatly reduce the task processing performance.

[0008] The multi-sink solution can send a record to multiple topics, but this solution has many disadvantages:

[0009] (1) Cannot guarantee strong consistency between topics. Multi-topic writing is not in the same transaction, and cannot guarantee that writing is successful or failed at the same time, so it cannot guarantee strong consistency between topics;

[0010] (2) Reduce the task processing performance. If it is thousands of topics, it needs to create thousands of sinks. For Flink, adding a node also means that Flink needs to allocate corresponding resources to run the task corresponding to the node, and at the same time, the data is routed to the sink, and the network transmission is also needed in the middle. Therefore, creating thousands of sinks needs to occupy a large amount of resources, which will greatly reduce the task processing performance.

[0011] The above information disclosed in the background section is only used to enhance the understanding of the background of the technology described herein, and therefore, the background may contain some information that is not considered prior art by those skilled in the art in the country. SUMMARY

[0012] The main purpose of the present application is to provide a topic writing method, writing device, computer readable storage medium, processor and stream processing platform, to solve the problem of excessive resource occupation in the prior art.

[0013] According to an aspect of an embodiment of the present application, a topic writing method is provided, comprising: obtaining incoming data of a stream processing platform to obtain target data; generating a plurality of records according to the target data, the records comprising a partition field, the partition field being used to represent the correspondence between the records and the topics of the stream processing platform; calling a current transaction, and writing a plurality of records to the corresponding topics according to the partition field, the records and the topics corresponding one by one.

[0014] Optionally, before calling the current transaction and writing a plurality of records to the corresponding topics according to the partition field, the method further comprises: querying the state of the current transaction; in the case that the current transaction is in a to-be-submitted state, adding the current transaction to a to-be-submitted transaction set, and generating a new current transaction.

[0015] Optionally, after calling the current transaction and writing a plurality of records to the corresponding topics according to the partition field, the method further comprises: adding the current transaction to a to-be-submitted transaction set; and submitting the transactions of the to-be-submitted transaction set in the order of addition.

[0016] Optionally, generating a plurality of records according to the target data comprises: obtaining a corresponding task configuration file according to the task code of the target data, the configuration mode of the task configuration file comprising an output field determination mode and a partition mode; and processing the target data according to the output field determination mode and the partition mode to generate a plurality of records.

[0017] Optionally, the method for determining the output fields includes outputting all fields and outputting a specified subset of fields.

[0018] Optionally, the partitioning method includes a custom partitioning method and a default partitioning method.

[0019] According to another aspect of the present invention, a topic writing device is also provided, comprising: an acquisition unit for acquiring incoming data from a stream processing platform to obtain target data; a generation unit for generating multiple records based on the target data, the records including a partition field, the partition field being used to characterize the correspondence between the records and the topics of the stream processing platform; and a writing unit for invoking the current transaction and writing the multiple records to the corresponding topics according to the partition field, the records corresponding one-to-one with the topics.

[0020] According to another aspect of the present invention, a computer-readable storage medium is also provided, the computer-readable storage medium including a stored program, wherein the program executes any one of the methods described.

[0021] According to another aspect of the present invention, a processor is also provided, the processor being configured to run a program, wherein the program, when running, executes any one of the methods described.

[0022] According to one aspect of the present invention, a stream processing platform is also provided, including a connector, one or more processors, a memory, a display device, and one or more programs, wherein the connector is used to write records to a topic, the one or more programs are stored in the memory and configured to be executed by the one or more processors, and the one or more programs include methods for performing any one of the methods described.

[0023] In this embodiment of the invention, the topic writing method first obtains the incoming data from the stream processing platform to get the target data; then, multiple records are generated based on the target data, each record including a partition field, which characterizes the correspondence between the record and the topic of the stream processing platform; finally, the current transaction is invoked, and the multiple records are written to the corresponding topic according to the partition field, with each record corresponding to a topic one-to-one. This method can write multiple records to the corresponding topic simply by invoking the current transaction. Since the partition field characterizes the correspondence between the record and the topic of the stream processing platform, multiple topic writing can be completed through a single connector. Each connector only needs to create one node, which significantly reduces resource consumption compared to the prior art where one node is created for each topic, thus solving the problem of excessive resource consumption for multiple topic writing in the prior art. Attached Figure Description

[0024] The accompanying drawings, which form part of this application, are used to provide a further understanding of this application. The illustrative embodiments and descriptions of this application are used to explain this application and do not constitute an undue limitation of this application. In the drawings:

[0025] Figure 1 A schematic diagram of a topic writing method according to an embodiment of this application is shown;

[0026] Figure 2 A schematic diagram of a topic writing apparatus according to an embodiment of this application is shown;

[0027] Figure 3 A schematic diagram of the processing logic of a connector according to an embodiment of this application is shown. Detailed Implementation

[0028] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. This application will now be described in detail with reference to the accompanying drawings and embodiments.

[0029] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0030] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate for the embodiments of this application described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0031] For ease of description, the following explains some of the nouns or terms used in the embodiments of this application:

[0032] Flink: Apache Flink is a framework and distributed processing engine for stateful computation on unbounded and bounded streams of data.

[0033] Kafka: Apache Kafka is a distributed stream processing platform.

[0034] topic: In Kafka, a topic is a category or subscription source name to which records are published.

[0035] FlinkKafkaProducer: Flink's native Kafka connector, used for writing data to a Kafka topic.

[0036] Checkpoint: A state-tolerance mechanism in Flink. It persists the state of operators, forming checkpoints for fault recovery.

[0037] As mentioned in the background section, existing multi-topic writing methods consume excessive resources. To address this issue, in a typical embodiment of this application, a topic writing method, writing device, computer-readable storage medium, processor, and stream processing platform are provided.

[0038] According to an embodiment of this application, a method for writing a topic is provided.

[0039] Figure 1 This is a flowchart of a topic writing method according to an embodiment of this application. Figure 1 As shown, the method includes the following steps:

[0040] Step S101: Obtain the incoming data from the stream processing platform to get the target data;

[0041] Step S102: Generate multiple records based on the target data. Each record includes a partition field, which is used to characterize the correspondence between the record and the topic of the stream processing platform.

[0042] Step S103: Invoke the current transaction and write multiple records to the corresponding topics according to the partition field. Each record corresponds to a topic.

[0043] In the aforementioned topic writing method, firstly, the incoming data from the stream processing platform is obtained to acquire the target data; then, multiple records are generated based on the target data, each record including a partition field. This partition field represents the correspondence between the record and the topic on the stream processing platform; finally, the current transaction is invoked, and the multiple records are written to the corresponding topic based on the partition field. Each record corresponds one-to-one with its respective topic. This method can write multiple records to the corresponding topic simply by invoking the current transaction. Since the partition field represents the correspondence between the record and the topic on the stream processing platform, multiple topic writing can be completed using only one connector. Each connector only needs to create one node, which significantly reduces resource consumption compared to existing technologies that create one node per topic. This solves the problem of excessive resource consumption for multi-topic writing in existing technologies.

[0044] In practical applications, multiple topic names can be configured under a Kafka node. For example, tb-dpdtl-curt-acc-topic and dpst-open-clos-acct-book-info-topic can both be topic names.

[0045] It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases the steps shown or described may be executed in a different order than that shown here.

[0046] In one embodiment of this application, before invoking the current transaction and writing multiple records to the corresponding topic according to the partition field, the method further includes: querying the status of the current transaction; if the current transaction is in a pending commit state, adding the current transaction to the pending commit transaction set and generating a new current transaction; subsequently, by invoking the current transaction and writing multiple records to the corresponding topic according to the partition field in the records, the method further realizes writing multiple topics through a single connector, further solves the problem of excessive resource consumption for writing multiple topics in the prior art, and further ensures strong consistency of data between topics.

[0047] Specifically, in practical applications, when Flink initiates a checkpoint, it calls the connector's snapshotState() method to take a snapshot; it calls the preCommit() method to add the current transaction to the set of transactions to be committed, and at the same time calls the beginTransactionInternal() method to generate a new current transaction.

[0048] In one specific embodiment of this application, when the current transaction is in a failed commit state, a transaction rollback will be performed, without saving the operations performed on the transaction, and restoring the behavior to the previous correct state. For example, in a transfer scenario, if an error occurs during the transfer process, i.e., money has been deducted from the requester's bank account, but the money in the recipient's bank account has not increased, an error will occur. Therefore, in this scenario, a transaction rollback is required, i.e., the operation of deducting money from the requester's bank account is not saved, thus returning to the state before the transaction was executed, i.e., the money in the requester's bank account was not successfully deducted, and the money in the recipient's bank account was not increased.

[0049] In another embodiment of this application, after invoking the current transaction and writing multiple records to the corresponding topic according to the partition field, the method further includes: adding the current transaction to the set of transactions to be committed; and committing the transactions in the set of transactions to be committed in the order of addition. In this embodiment, the transactions in the set of transactions to be committed are committed in the order in which the current transaction is added to the set of transactions to be committed, thus ensuring that it is possible to know in a relatively timely manner whether the transaction has been successfully executed.

[0050] In one specific embodiment of this application, after the checkpoint is completed, Flink will call back the connector's notifyCheckpointComplete() method and call the commit() method to commit the transactions to be committed one by one.

[0051] To generate multiple records more simply and efficiently, in another embodiment of this application, generating multiple records based on the target data includes: obtaining a corresponding task configuration file based on the task code of the target data, wherein the configuration method of the task configuration file includes an output field determination method and a partitioning method; and processing the target data based on the output field determination method and the partitioning method to generate multiple records.

[0052] In one specific embodiment of this application, when a record arrives, the connector's invoke() method is called; in the invoke() method, multiple records are generated, each corresponding to a different topic, and the current transaction is used to send the record.

[0053] In practical applications, the above task configuration file can be a YAML task configuration file.

[0054] In another embodiment of this application, the above-mentioned output field determination method includes outputting all fields and outputting a specified portion of the fields. This ensures that the target data can be processed more flexibly according to the output field determination method, and further ensures that multiple records are obtained more efficiently.

[0055] Specifically, the all_fields node can be used to indicate whether all fields are output. When all_fields is true, all fields are output; when all_fields is false, a specified subset of fields is output. Of course, when all_fields is false, the optional_fields node can be configured to specify which fields are to be output.

[0056] To ensure that the target data can be processed more flexibly according to the partitioning method to obtain multiple records, in one embodiment of this application, the partitioning method includes a custom partitioning method and a default partitioning method.

[0057] Specifically, the partition_field node can be used to configure the partitioning method, and the partition_class node can be used to configure a custom partitioning method. Of course, you can also choose to configure either the partition_field node or the partition_class node to implement custom partitioning. When neither the partition_field node nor the partition_class node is configured, the default partitioning method is round-robin.

[0058] This application also provides a topic writing device. It should be noted that the topic writing device of this application can be used to execute the topic writing method provided in this application. The topic writing device provided in this application is described below.

[0059] Figure 2 This is a schematic diagram of a topic writing device according to an embodiment of this application. Figure 2 As shown, the device includes:

[0060] The acquisition unit 10 is used to acquire the incoming data from the stream processing platform and obtain the target data;

[0061] The generation unit 20 is used to generate multiple records based on the target data. The records include a partition field, which is used to characterize the correspondence between the records and the topics of the stream processing platform.

[0062] The write unit 30 is used to invoke the current transaction and write multiple records to the corresponding topics according to the partition field. Each record corresponds to a topic.

[0063] In the aforementioned topic writing device, the acquisition unit acquires incoming data from the stream processing platform to obtain target data; the generation unit generates multiple records based on the target data, each record including a partition field that represents the correspondence between the record and the topic of the stream processing platform; the writing unit invokes the current transaction and writes the multiple records to the corresponding topic based on the partition field, with each record corresponding to a topic one-to-one. This device can write multiple records to the corresponding topic simply by invoking the current transaction. Since the partition field represents the correspondence between the record and the topic of the stream processing platform, multiple topic writing can be completed through a single connector, and each connector only needs to create one node. Compared to existing technologies where one node is created for each topic, this significantly reduces resource consumption and solves the problem of excessive resource consumption during multi-topic writing in existing technologies.

[0064] In practical applications, multiple topic names can be configured under a Kafka node. For example, tb-dpdtl-curt-acc-topic and dpst-open-clos-acct-book-info-topic can both be topic names.

[0065] In one embodiment of this application, the above-mentioned device further includes a query unit and a first addition unit. The query unit is used to query the status of the current transaction before calling the current transaction and writing multiple records to the corresponding topic according to the partition field. The first addition unit is used to add the current transaction to the set of transactions to be committed when the current transaction is in a pending commit state, and generate a new current transaction. Subsequently, by calling the current transaction, multiple records are written to the corresponding topic according to the partition field in the records. This further realizes the writing of multiple topics through a single connector, further solves the problem of excessive resource consumption for writing multiple topics in the prior art, and further ensures strong consistency of data between topics.

[0066] Specifically, in practical applications, when Flink initiates a checkpoint, it calls the connector's snapshotState() method to take a snapshot; it calls the preCommit() method to add the current transaction to the set of transactions to be committed, and at the same time calls the beginTransactionInternal() method to generate a new current transaction.

[0067] In one specific embodiment of this application, when the current transaction is in a failed commit state, a transaction rollback will be performed, without saving the operations performed on the transaction, and restoring the behavior to the previous correct state. For example, in a transfer scenario, if an error occurs during the transfer process, i.e., money has been deducted from the requester's bank account, but the money in the recipient's bank account has not increased, an error will occur. Therefore, in this scenario, a transaction rollback is required, i.e., the operation of deducting money from the requester's bank account is not saved, thus returning to the state before the transaction was executed, i.e., the money in the requester's bank account was not successfully deducted, and the money in the recipient's bank account was not increased.

[0068] In another embodiment of this application, the above-mentioned apparatus further includes a second adding unit and a commit unit. The second adding unit is used to add the current transaction to the set of transactions to be committed after invoking the current transaction and writing multiple records to the corresponding topic according to the partition field. The commit unit is used to commit the transactions in the set of transactions to be committed in the order they were added. In this embodiment, the transactions in the set of transactions to be committed are committed in the order in which the current transaction was added to the set of transactions to be committed, thus ensuring that it is possible to know relatively quickly whether the transaction has been successfully executed.

[0069] In one specific embodiment of this application, after the checkpoint is completed, Flink will call back the connector's notifyCheckpointComplete() method and call the commit() method to commit the transactions to be committed one by one.

[0070] To generate multiple records more simply and efficiently, in another embodiment of this application, the generation unit includes an acquisition module and a processing module. The acquisition module is used to acquire a corresponding task configuration file based on the task code of the target data. The configuration method of the task configuration file includes an output field determination method and a partitioning method. The processing module is used to process the target data according to the output field determination method and the partitioning method to generate multiple records.

[0071] In one specific embodiment of this application, when a record arrives, the connector's invoke() method is called; in the invoke() method, multiple records are generated, each corresponding to a different topic, and the current transaction is used to send the record.

[0072] In practical applications, the above task configuration file can be a YAML task configuration file.

[0073] In another embodiment of this application, the above-mentioned output field determination method includes outputting all fields and outputting a specified portion of the fields. This ensures that the target data can be processed more flexibly according to the output field determination method, and further ensures that multiple records are obtained more efficiently.

[0074] Specifically, the all_fields node can be used to indicate whether all fields are output. When all_fields is true, all fields are output; when all_fields is false, a specified subset of fields is output. Of course, when all_fields is false, the optional_fields node can be configured to specify which fields are to be output.

[0075] To ensure that the target data can be processed more flexibly according to the partitioning method to obtain multiple records, in one embodiment of this application, the partitioning method includes a custom partitioning method and a default partitioning method.

[0076] Specifically, the partition_field node can be used to configure the partitioning method, and the partition_class node can be used to configure a custom partitioning method. Of course, you can also choose to configure either the partition_field node or the partition_class node to implement custom partitioning. When neither the partition_field node nor the partition_class node is configured, the default partitioning method is round-robin.

[0077] The above-mentioned topic writing device includes a processor and a memory. The above-mentioned acquisition unit, generation unit and writing unit are all stored in the memory as program units. The processor executes the above-mentioned program units stored in the memory to realize the corresponding functions.

[0078] The processor contains a kernel, which retrieves the corresponding program units from memory. One or more kernels can be configured, and adjusting kernel parameters can address the problem of excessive resource consumption from multi-topic writes in existing technologies.

[0079] The memory may include non-permanent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM, and the memory includes at least one memory chip.

[0080] This invention provides a computer-readable storage medium storing a program that, when executed by a processor, implements the above-described topic writing method.

[0081] This invention provides a processor for running a program, wherein the program executes the writing method of the topic.

[0082] In a typical embodiment of this application, a stream processing platform is also provided, which includes a connector, one or more processors, a memory, a display device, and one or more programs, wherein the connector is used to write records to a topic, the one or more programs are stored in the memory and configured to be executed by the one or more processors, and the one or more programs include methods for performing any of the above-described methods.

[0083] The aforementioned stream processing platform can execute any of the methods described above. In the topic writing method, firstly, the incoming data from the stream processing platform is obtained to acquire the target data; then, multiple records are generated based on the target data. These records include a partition field, which characterizes the correspondence between the records and the topics of the stream processing platform; finally, the current transaction is invoked, and the multiple records are written to the corresponding topics based on the partition field. Each record corresponds one-to-one with its respective topic. This method can write multiple records to the corresponding topics simply by invoking the current transaction. Since the partition field characterizes the correspondence between the records and the topics of the stream processing platform, multiple topic writing can be completed using only one connector. Each connector only needs to create one node, which significantly reduces resource consumption compared to existing technologies that create one node per topic. This solves the problem of excessive resource consumption for multi-topic writing in existing technologies.

[0084] To enable those skilled in the art to more clearly understand the technical solution of this application, the following description will be provided in conjunction with specific embodiments:

[0085] Example

[0086] This application supports writing to multiple topics through a custom Kafka connector, which can be named MultiTopicFlinkKafkaProducer. The roles of the various classes / interfaces are as follows: CheckpointedFunction interface: This interface is used to perform checkpoint operations. The initializeState() and snapshotState() methods are used to initialize the state and take a snapshot of the state, respectively. CheckpointListener interface: This interface is used to listen for checkpoint activities. notifyCheckpointComplete() and notifyCheckpointAborted() are called when the checkpoint is successful and when it is canceled, respectively. RichSinkFunction interface: This interface is a rich function interface for sinks, providing lifecycle methods. The open() and close() methods are called when the operator is created and destroyed, respectively. TwoPhaseCommitSinkFunction interface: This abstract class encapsulates the logic template for two-phase commit. Subclasses can implement their own transaction commit logic by implementing the beginTransaction(), preCommit(), and commit() methods.

[0087] like Figure 3As shown, the connector MultiTopicFlinkKafkaProducer inherits the TwoPhaseCommitSinkFunction interface. Its processing logic is as follows: First, it performs checkpoint processing. When Flink initiates a checkpoint, it calls the connector's snapshotState() method to take a snapshot. In the preCommit() method, the current transaction is added to the set of transactions to be committed, and the beginTransactionInternal() method is called to generate a new current transaction. Second, it processes single records, i.e., the record entry processing logic. When a record enters, the Kafka connector's invoke() method is called. In the invoke() method, multiple records are generated, each corresponding to a different topic, and the current transaction is used to send the records. Specifically, when creating a record, the task code is read from the input data, and the corresponding YAML task configuration file is read based on the task code. The task configuration file defines the output field determination method and partitioning method, and the corresponding record is generated according to the configuration. Finally, a callback occurs after checkpoint completion. When checkpointing is complete, Flink calls back the connector's notifyCheckpointComplete() method; the commit() method is then called to commit the transactions to be committed one by one.

[0088] This invention provides a device including a processor, a memory, and a program stored in the memory and executable on the processor. When the processor executes the program, it performs at least the following steps:

[0089] Step S101: Obtain the incoming data from the stream processing platform to get the target data;

[0090] Step S102: Generate multiple records based on the target data. Each record includes a partition field, which is used to characterize the correspondence between the record and the topic of the stream processing platform.

[0091] Step S103: Invoke the current transaction and write multiple records to the corresponding topics according to the partition field. Each record corresponds to a topic.

[0092] The devices mentioned in this article can be servers, PCs, tablets, mobile phones, etc.

[0093] This application also provides a computer program product, which, when executed on a data processing device, is suitable for executing an initialization program having at least the following method steps:

[0094] Step S101: Obtain the incoming data from the stream processing platform to get the target data;

[0095] Step S102: Generate multiple records based on the target data. Each record includes a partition field, which is used to characterize the correspondence between the record and the topic of the stream processing platform.

[0096] Step S103: Invoke the current transaction and write multiple records to the corresponding topics according to the partition field. Each record corresponds to a topic.

[0097] In the above embodiments of the present invention, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0098] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units described above can be a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces, or indirect coupling or communication connection between units or modules, and may be electrical or other forms.

[0099] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0100] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0101] If the aforementioned integrated units are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a computer-readable storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned computer-readable storage medium includes various media capable of storing program code, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.

[0102] As can be seen from the above description, the embodiments of this application achieve the following technical effects:

[0103] 1) In the topic writing method of this application, firstly, the incoming data from the stream processing platform is obtained to obtain the target data; then, multiple records are generated based on the target data, and the records include a partition field, which is used to characterize the correspondence between the records and the topics of the stream processing platform; finally, the current transaction is invoked, and the multiple records are written to the corresponding topics according to the partition field, with each record corresponding to a topic one-to-one. This method can write multiple records to the corresponding topics simply by invoking the current transaction. Since the partition field can characterize the correspondence between the records and the topics of the stream processing platform, multiple topic writing can be completed through a single connector. A single connector only needs to create one node, which greatly reduces resource consumption compared to the existing technology of creating one node per topic, thus solving the problem of excessive resource consumption for multiple topic writing in the existing technology.

[0104] 2) In the topic writing device of this application, the acquisition unit is used to acquire incoming data from the stream processing platform to obtain target data; the generation unit is used to generate multiple records based on the target data, the records including a partition field, the partition field being used to characterize the correspondence between the records and the topics of the stream processing platform; the writing unit is used to call the current transaction and write the multiple records to the corresponding topics according to the partition field, the records corresponding one-to-one with the topics. This device can write multiple records to the corresponding topics simply by calling the current transaction. Since the partition field can characterize the correspondence between the records and the topics of the stream processing platform, multiple topic writing can be completed through only one connector, and one connector only needs to create one node. Compared with the prior art, which creates one node per topic, this greatly reduces resource consumption and solves the problem of excessive resource consumption for multiple topic writing in the prior art.

[0105] 3) The stream processing platform of this application can execute any of the above methods. In the topic writing method, firstly, the incoming data from the stream processing platform is obtained to get the target data; then, multiple records are generated based on the target data. The records include a partition field, which is used to characterize the correspondence between the records and the topics of the stream processing platform; finally, the current transaction is called, and the multiple records are written to the corresponding topics according to the partition field. The records correspond one-to-one with the topics. This method can write multiple records to the corresponding topics simply by calling the current transaction. Since the partition field can characterize the correspondence between the records and the topics of the stream processing platform, multiple topic writing can be completed through a single connector. A single connector only needs to create one node. Compared with the prior art, which creates one node per topic, this greatly reduces resource consumption and solves the problem of excessive resource consumption for multiple topic writing in the prior art.

[0106] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A method for writing a topic, characterized in that, include: Obtain the incoming data from the stream processing platform to get the target data; Multiple records are generated based on the target data. Each record includes a partition field, which is used to characterize the correspondence between the record and the topic of the stream processing platform. Invoke the current transaction and write multiple records to the corresponding topic according to the partition field, wherein each record corresponds one-to-one with the topic; Generate multiple records based on the target data, including: obtaining the corresponding task configuration file based on the task code of the target data, wherein the configuration method of the task configuration file includes output field determination method and partitioning method; The target data is processed according to the output field determination method and the partitioning method to generate multiple records.

2. The method according to claim 1, characterized in that, Before invoking the current transaction and writing the multiple records to the corresponding topic according to the partition field, the method further includes: Query the status of the current transaction; If the current transaction is in a pending commit state, add the current transaction to the pending commit transaction set and generate a new current transaction.

3. The method according to claim 1, characterized in that, After invoking the current transaction and writing multiple records to the corresponding topic according to the partition field, the method further includes: Add the current transaction to the set of transactions to be committed; The transactions in the set of transactions to be committed are committed in the order they were added.

4. The method according to claim 1, characterized in that, The output field determination methods include outputting all fields and outputting a specified subset of fields.

5. The method according to claim 1, characterized in that, The partitioning methods include custom partitioning and default partitioning.

6. A topic writing device, characterized in that, include: The acquisition unit is used to acquire incoming data from the stream processing platform and obtain the target data. A generation unit is used to generate multiple records based on the target data. Each record includes a partition field, which is used to characterize the correspondence between the record and the topic of the stream processing platform. The write unit is used to invoke the current transaction and write multiple records to the corresponding topic according to the partition field, wherein each record corresponds one-to-one with the topic; The generation unit includes an acquisition module and a processing module. The acquisition module is used to acquire a corresponding task configuration file based on the task code of the target data. The configuration method of the task configuration file includes an output field determination method and a partitioning method. The processing module is used to process the target data according to the output field determination method and the partitioning method to generate multiple records.

7. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored program, wherein the program performs the method according to any one of claims 1 to 5.

8. A processor, characterized in that, The processor is used to run a program, wherein the program executes the method according to any one of claims 1 to 5 when it runs.

9. A stream processing platform, characterized in that, The device includes a connector, one or more processors, a memory, a display device, and one or more programs, wherein the connector is used to write records to a topic, the one or more programs are stored in the memory and configured to be executed by the one or more processors, and the one or more programs include methods for performing any one of claims 1 to 5.

Images