Data processing method and device, equipment, storage medium and program product

Abstract

The invention discloses a data processing method and device, equipment, a storage medium and a program product, and the method comprises the steps: determining a first message theme corresponding to each production application according to a mapping relation between the production applications and message themes; if the first message themes corresponding to the production applications are the same, taking the names of the production applications as first prefixes of the first message themes to obtain second message themes; determining a third message theme corresponding to a consumption application from the second message themes according to a mapping relation between the consumption application and the first message theme and a mapping relation between the consumption application and the production application; and sending the third message theme. According to the method, only one production application is used for data production under the same topic, data under the topic cannot be mixed, data consumption errors cannot occur when a subsequent consumption application consumes the topic, and it is guaranteed that data processing under the topic is correct.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a data processing method, device, equipment, storage medium and program product. Background Art

[0002] The message topics in the Kafka cluster should normally correspond to production applications and consumer applications. In some scenarios, there will be some different production applications and consumer applications that need to produce and consume the same topic at the same time. Since the same topic does not distinguish data from different production applications, but directly arranges the data in the order in which it is produced, different production applications will produce data on the same topic in order. When different consumer applications consume the topic later, data processing errors will occur due to data mixing. Summary of the invention

[0003] The embodiment of the present application provides a data processing method, device, equipment, storage medium and program product, which obtains the second message topic by taking the name of the production application as the first prefix of the first message topic corresponding to the production application, and then determines the third message topic uniquely corresponding to the consumption application from multiple second message topics according to the mapping relationship between the consumption application and the first message topic and the mapping relationship between the consumption application and the production application. There is only one production application producing data under the same topic, so the data under the topic will not be mixed, and there will be no data consumption errors when the subsequent consumption application consumes the topic, which is conducive to ensuring the correct processing of data under the topic.

[0004] In a first aspect, an embodiment of the present application provides a data processing method, which is applied to a server, comprising:

[0005] Determine, according to the mapping relationship between production applications and message topics, a first message topic corresponding to each of the production applications;

[0006] If the first message topics corresponding to the production applications are the same, the name of the production application is used as the first prefix of the first message topic to obtain the second message topic;

[0007] Determine, from the second message topic, a third message topic corresponding to the consuming application according to a mapping relationship between the consuming application and the first message topic, and a mapping relationship between the consuming application and the producing application;

[0008] The third message topic is sent.

[0009] In a second aspect, an embodiment of the present application provides a data processing method, which is applied to a client of a consumer application, including:

[0010] According to the mapping relationship between the consumer application and the production application, and the mapping relationship between the message topic and the production application, determine the fourth message topic corresponding to the consumer application from the received message topic, where the prefix of the fourth message topic is used to indicate the production application name corresponding to the consumer application;

[0011] Determine a message node of the fourth message topic according to a prefix of the fourth message topic;

[0012] The message node of the fourth message topic consumes the messages in the fourth message topic.

[0013] In a third aspect, an embodiment of the present application provides a data processing device, including:

[0014] A first determining unit, configured to determine a first message topic corresponding to each production application according to a mapping relationship between the production application and the message topic;

[0015] an acquiring unit, configured to obtain a second message topic by using the name of the production application as a first prefix of the first message topic if the first message topics corresponding to the production applications are the same;

[0016] A second determining unit, configured to determine a third message topic corresponding to the consuming application from the second message topic according to a mapping relationship between the consuming application and the first message topic and a mapping relationship between the consuming application and the producing application;

[0017] A sending unit is used to send the third message topic.

[0018] In a fourth aspect, an embodiment of the present application provides a data processing device, including:

[0019] A first determining unit is used to determine a fourth message topic corresponding to the consumer application from the received message topic according to a mapping relationship between the consumer application and the production application, and a mapping relationship between the message topic and the production application, wherein the prefix of the fourth message topic is used to indicate a production application name corresponding to the consumer application;

[0020] A second determining unit, configured to determine a message node of the fourth message topic according to a prefix of the fourth message topic;

[0021] A processing unit is used to consume the messages in the fourth message topic at the message node of the fourth message topic.

[0022] In a fifth aspect, an embodiment of the present application provides a data processing device, the device comprising:

[0023] A memory, a processor, and a data processing program stored in the memory and executable on the processor, wherein the data processing program is configured to implement part or all of the steps described in any method of the first aspect or the second aspect.

[0024] In a sixth aspect, an embodiment of the present application provides a computer-readable storage medium having executable program code stored thereon, and when the executable program code is executed by a processor, some or all of the steps described in any method in the first aspect or the second aspect are implemented.

[0025] In a seventh aspect, an embodiment of the present application provides a computer program product, wherein the computer program product includes a computer program, and the computer program is operable to cause a computer to execute some or all of the steps described in any method in the first aspect or the second aspect of the embodiment of the present application. The computer program product can be a software installation package.

[0026] By implementing the embodiment of the present application, the server first determines the first message topic corresponding to each production application according to the mapping relationship between the production application and the message topic; then, if the first message topics corresponding to the production applications are the same, the name of the production application is used as the first prefix of the first message topic to obtain the second message topic; then, according to the mapping relationship between the consumption application and the first message topic, and the mapping relationship between the consumption application and the production application, the third message topic corresponding to the consumption application is determined from the second message topic; finally, the third message topic is sent. The second message topic is obtained by using the name of the production application as the first prefix of the first message topic corresponding to the production application, and then, according to the mapping relationship between the consumption application and the first message topic and the mapping relationship between the consumption application and the production application, the third message topic uniquely corresponding to the consumption application is determined from multiple second message topics. There is only one production application under the same topic to produce data, the data under the topic will not be mixed, and there will be no data consumption errors when the subsequent consumption application consumes the topic, which is conducive to ensuring the correct processing of data under the topic. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the background technology, the drawings required for use in the embodiments of the present application or the background technology will be described below.

[0028] Figure 1 It is a schematic diagram of the architecture of a data processing system provided in an embodiment of the present application;

[0029] Figure 2 is a flow chart of a data processing method provided in an embodiment of the present application;

[0030] Figure 3This is a schematic diagram of a message topic in a Kafka cluster provided in an embodiment of the present application;

[0031] Figure 4 is a flow chart of another data processing method provided in an embodiment of the present application;

[0032] Figure 5 is a structural schematic diagram of a data processing device provided in an embodiment of the present application;

[0033] Figure 6 is a structural schematic diagram of another data processing device provided in an embodiment of the present application;

[0034] Figure 7 It is a structural diagram of a data processing device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0035] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. According to the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of this application.

[0036] The terms "first", "second", "third", etc. in the specification, claims and drawings of this application are used to distinguish different objects, rather than to describe a specific order. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but optionally includes steps or units that are not listed, or optionally includes other steps or units inherent to these processes, methods, products or devices.

[0037] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0038] The key concepts and features involved in the embodiments of the present application are first explained below.

[0039] (1) Kafka: Kafka is an open source stream processing platform developed by the Apache Software Foundation and written in Scala and Java. Kafka is a high-throughput distributed publish-subscribe messaging system that can process all action stream data of consumer applications on a website. Kafka is based on the Pub / Sub model, supports efficient transmission and storage of messages in a distributed environment, provides reliable messaging and persistence, and also supports scenarios such as stream processing and real-time processing.

[0040] (2) Topic: In Kafka, topic is a logical concept for storing messages, which can be understood as a collection of messages. Each message sent to the Kafka cluster comes with a category, indicating which topic the message is to be sent to. In terms of storage, messages from different topics are stored separately. Each topic can have multiple production applications sending messages to it, and multiple consumer applications can consume messages from the same topic.

[0041] (3) Production application: A production application is an application that sends messages to a Kafka cluster. The production application writes the message to one or more partitions of a topic, and then the Kafka cluster stores the message in the message queue of the partition. Production applications can send messages to Kafka in an asynchronous or synchronous manner. The asynchronous method means that the production application can send messages to multiple topics at the same time without having to wait for confirmation that the message has been successfully sent. The synchronous method means that the production application must wait for Kafka to confirm that the message has been successfully sent before sending the next message.

[0042] (4) Consumer Application: A consumer application is an application that reads messages from a Kafka queue. A consumer application can read messages from one or more partitions of one or more topics. A consumer application can read messages in order or by timestamp or offset. A consumer application can process messages in different ways, such as writing messages to external storage or performing real-time processing on messages. The client of a consumer application subscribes to a topic from Kafka and pulls messages from the topic.

[0043] The message topics in the Kafka cluster should normally correspond to production applications and consumer applications. In some scenarios, there will be some different production applications and consumer applications that need to produce and consume the same topic at the same time. Since the same topic does not distinguish data from different production applications, but directly arranges the data in the order in which it is produced, different production applications will produce data on the same topic in order. When different consumer applications consume the topic later, data processing errors will occur due to data mixing.

[0044] In order to solve the problem of data mixing when different production applications produce the same topic in time sequence, the existing technology adopts the following methods:

[0045] Method 1: Before sending a message to a topic, the production application adds a field in the message header that can identify its own application. For example, the application name. When reading a message, the consumer application first parses the application identifier in the message header, and then decides whether to process the message based on the identifier. In this way, even if different production applications send messages to the same topic, the consumer application can distinguish the source of the message and process it in a targeted manner.

[0046] Method 2: Add an intermediate component between the production application and the Kafka cluster for data preprocessing. The intermediate component receives data from different production applications, then classifies and preprocesses the data, and finally sends the data to the Kafka topic. For example, the intermediate component can add additional identifiers to the data based on the source application of the data, so that the data entering the topic can be distinguished by the consuming application.

[0047] It can be seen that although the above two methods can allow consumer applications to correctly consume data produced by different production applications in the topic, they both require a large amount of computing resources to perform data preprocessing operations, such as adding message headers to the data produced by all production applications, and adding intermediate components to preprocess the data produced by all production applications.

[0048] In response to the above problems, the embodiments of the present application provide a data processing method, device, equipment, storage medium and program product. The server first determines the first message topic corresponding to each production application according to the mapping relationship between the production application and the message topic; then, if the first message topics corresponding to the production applications are the same, the name of the production application is used as the first prefix of the first message topic to obtain the second message topic; then, according to the mapping relationship between the consumption application and the first message topic, and the mapping relationship between the consumption application and the production application, the third message topic corresponding to the consumption application is determined from the second message topic; finally, the third message topic is sent. The second message topic is obtained by using the name of the production application as the first prefix of the first message topic corresponding to the production application, and then the third message topic uniquely corresponding to the consumption application is determined from multiple second message topics according to the mapping relationship between the consumption application and the first message topic and the mapping relationship between the consumption application and the production application. There is only one production application under the same topic to produce data, the data under the topic will not be mixed, and there will be no data consumption errors when the subsequent consumption application consumes the topic, which is conducive to ensuring the correct processing of data under the topic.

[0049] The data processing method, apparatus, device, storage medium and program product provided in the embodiments of the present application can be applied to Figure 1 In the data processing system shown, see Figure 1 , Figure 1 1 is a schematic diagram of the architecture of a data processing system provided by an embodiment of the present application. The data processing system 100 includes a client 101 and a server 102. The client 101 can communicate with the server 102 through a network. The client 101 refers to a device used by a user, such as a smart phone, a computer, etc., wherein the client 101 includes a client 101 for a production application and a client 101 for a consumption application. In this solution, the client 101 is mainly responsible for the production and consumption of data. Specifically, the client 101 for the production application is responsible for producing data and sending it to the server 102 for processing. The client 101 for the consumption application is responsible for determining a message node from the server 102 and consuming data at the message node. The user can interact with the data processing system 100 through the client 101, set the mapping relationship between the production application and the first message topic, the mapping relationship between the production application and the consumption application, and the mapping relationship between the consumption application and the first message topic, etc.; the server 102 refers to a remote computer used to process a large number of computing tasks and store data. In this scheme, the server 102 uses the name of the production application as the first prefix of the first message topic corresponding to the production application to obtain the second message topic, and then determines the third message topic uniquely corresponding to the consumer application from multiple second message topics based on the mapping relationship between the consumer application and the first message topic and the mapping relationship between the consumer application and the production application.

[0050] Based on this, the present application provides a data processing method, apparatus, device, storage medium and program product, and the present application is described in detail below in conjunction with the accompanying drawings.

[0051] See also Figure 2 , Figure 2 is a flow chart of a data processing method provided in an embodiment of the present application, such as Figure 2 As shown, the method comprises the following steps:

[0052] S201, determining a first message topic corresponding to each production application according to a mapping relationship between production applications and message topics.

[0053] Among them, the execution subject of this method can be Figure 1 Server 102 of data processing system 100.

[0054] Among them, the server 102 can be a server in a kafka cluster, and the message subject can be a topic in a kafka cluster. Specifically, the kafka cluster is a distributed message system composed of multiple kafka servers. There are generally multiple topics in the kafka cluster, and each topic can have one or more partitions. The parallel processing and load balancing of data are realized through partitions. For example, topicA has 3 partitions (partition0, partition1, partition2). When the production application that needs to be mapped to topicA sends a message to topicA, the message will be distributed to different partitions according to certain strategies (such as polling, hash calculation based on the message key, etc.).

[0055] Among them, each topic is a message collection. The production application sends the generated messages to a specific topic, and the consumer application subscribes to and receives messages from a specific topic. The mapping relationship between the production application and the topic can be one-to-one, one-to-many, many-to-one or many-to-many, and there is no restriction here.

[0056] For example, there are production application 1 and production application 2. When the mapping relationship between production application and topic is one-to-one, production application 1 is mapped to topicA in the kafka cluster, and production application 2 is mapped to topicB in the kafka cluster; when the mapping relationship between production application and topic is one-to-many, production application 1 is mapped to topicA and topicB; when the mapping relationship between production application and topic is many-to-one, both production application 1 and production application 2 are mapped to topicA; when the mapping relationship between production application and topic is many-to-many, production application 1 is mapped to topicA and topicB, and production application 2 is mapped to topicA and topicB.

[0057] S202: If the first message topics corresponding to the production applications are the same, the name of the production application is used as a first prefix of the first message topic to obtain a second message topic.

[0058] Among them, if the same first message topic exists in one or more first message topics corresponding to multiple production applications, the second message topic is obtained by using the name of the production application as the first prefix of the first message topic. Specifically, when the mapping relationship between the production application and the first message topic is many-to-one, all production applications correspond to the same first message topic, and the first message topics corresponding to the production applications are the same; when the mapping relationship between the production application and the first message topic is many-to-many, the same first message topic exists in multiple first message topics corresponding to multiple production applications, and the first message topics corresponding to the production applications are the same.

[0059] Among them, it is possible to determine whether the first message topics corresponding to multiple production applications are the same by comparing whether the characters of the first message topics corresponding to multiple production applications are exactly the same. For example, the characters of the first message topic (topicA) corresponding to production application 1 and the second message topic (topicA) corresponding to production application 2 are exactly the same. At this time, the first message topics corresponding to production application 1 and production application 2 are the same.

[0060] Exemplarily, when the mapping relationship between the production application and the first message topic is many-to-one, production application 1, production application 2, and production application 3 all correspond to the first message topic A, and the first message topics corresponding to the production applications are the same; when the mapping relationship between the production application and the first consumer topic is many-to-many, production application 1 corresponds to the first message topic A and the first message topic B, production application 2 corresponds to the first message topic A and the first message topic C, and production application 3 corresponds to the first message topic A and the first message topic D. At this time, the same first message topic A exists among the multiple first message topics corresponding to production application 1, production application 2, and production application 3, and the first message topics corresponding to the production applications are the same.

[0061] In a possible implementation, the name of the production application is used as the first prefix of the first message subject to obtain the second message subject, including:

[0062] The second prefix of the first message topic is updated to the name of the production application as the first prefix of the first message topic, to obtain the second message topic, where the second prefix is ​​the preset prefix of the first message topic.

[0063] Among them, in the Kafka cluster, the topic has a certain naming convention. For example, the topic used to store user login logs can be named "user_login_logs", and the preset prefix can be set or changed by the user on the server 102. For example, the preset prefix can be "default". At this time, the preset prefix is ​​added in front of the topic used to store user login logs, and the first message subject obtained is "default.user_login_logs", or "default-user_login_logs".

[0064] When adding a prefix to the message subject, the connection character between the prefix and the original message subject can be ".", "-" or "_", and the connection character can also be other special characters such as space, and there is no restriction on the connection character.

[0065] When setting or changing a prefix for a topic, a topic with a specified prefix can be generated through encoding encapsulation. The encoding method can be Java, Python, JavaScript, etc.

[0066] Taking Java as an example for specific explanation, when using Java to add a prefix to a topic, you need to first create a tool class, define a constant in the tool class to represent the prefix to be added, and then create a static method to add a prefix to the passed in topic. For example, in the tool class Topicprefix, define a constant as default, and then create a static method in the Topicprefix class to add the prefix default to the passed in topic.

[0067] Among them, when determining that the first message topic is the message topic corresponding to the production application, the prefix string in the first message topic is replaced with the name of the production application, and all characters except the prefix in the first message topic are not changed. For example, the name of the production application is app1, and the first message topic corresponding to app1 is default.topicA. At this time, the prefix string of the first message topic is replaced with app1, and the second message topic is app1.topicA.

[0068] In another example, the first message subject does not have a prefix, and the name of the production application is used as the first prefix of the first message subject to obtain the second message subject, including:

[0069] The name of the production application is used as the first prefix of the first message topic to obtain the second message topic.

[0070] Among them, the connecting character between the first prefix and the first message topic can be a special character, space or other character, and there is no limitation here. For example, the first message topic corresponding to the production application app1 is topicA. At this time, the name of the production application is used as the first prefix of the first message topic, and the connecting character between the first prefix and the original first message topic is ".". At this time, the second message topic is "app1.topicA".

[0071] In another example, when determining that the first message subject is the message subject corresponding to the production application, the prefix string in the first message subject is not replaced, but a new first prefix is ​​directly added in front of the preset prefix of the first message subject, and the first prefix is ​​the name of the production application.

[0072] The first prefix and the preset prefix can be connected by special characters or spaces, or they can be directly connected to the preset prefix. For example, the name of the production application is app1, and the first message topic corresponding to app1 is default.topicA. At this time, the prefix string of the first message topic is not replaced, but the first prefix (app1) is added in front of the preset prefix of the first message topic. The first prefix and the preset prefix are connected by the special character ".", and the second message topic is app1.default.topicA.

[0073] It can be seen that in this example, by using the name of the production application as the prefix of the message topic corresponding to the production application, the second message topic is obtained, so that the unique production application corresponding to the message topic can be determined according to the prefix of each message topic, which is conducive to avoiding the situation where multiple production applications produce messages for a topic at the same time and cause data mixing.

[0074] In a possible implementation, the method further includes:

[0075] If the first message topic corresponding to the production application is different, the second prefix of the first message topic is updated to the name of the production application as the first prefix of the first message topic to obtain the second message topic, and the second prefix is ​​the preset prefix of the first message topic.

[0076] When adding a prefix to the message subject, the connection character between the prefix and the original message subject can be ".", "-" or "_", and the connection character can also be other special characters such as space, and there is no restriction on the connection character.

[0077] When setting or changing a prefix for a topic, a topic with a specified prefix can be generated through coding encapsulation, and the coding method can be Java, Python, JavaScript, etc. Taking Java as an example for specific explanation, when adding a prefix to a topic using Java, you need to first create a tool class, define a constant in the tool class to represent the prefix to be added, and then create a static method to add a prefix to the incoming topic. For example, in the tool class Topicprefix, define a constant as default, and then create a static method in the Topicprefix class to add the prefix default to the incoming topic.

[0078] When determining that the first message subject is the message subject corresponding to the production application, the prefix character string in the first message subject is replaced with the name of the production application, without changing all characters except the prefix in the first message subject.

[0079] For example, see Figure 3 , Figure 3 is a schematic diagram of a message topic in a Kafka cluster provided in an embodiment of the present application, such as Figure 3 As shown, the preset prefix is ​​Def, and there are three first message topics in the kafka cluster, namely Def.topicA, Def.topicB, and Def.topicC. At this time, the production applications include app1, app2, and app3. The first message topics corresponding to app1 include Def.topicA and Def.topicC, the first message topics corresponding to app2 include Def.topicA and Def.topicB, and the first message topic corresponding to app3 is Def.topicC. At this time, the prefix strings in the first message topics are replaced with the names of the corresponding production applications to obtain multiple second message topics. By classifying the strings in the message topics except the prefix, the message topics with the same strings except the prefix are constructed into a list, and the three message topic lists are {Def.topicA, app1.topicA, app2.topicA}, {Def.topicB, app2.topicB}, and {Def.topicC, app1.topicC, app3.topicC}.

[0080] In another example, the first message subject does not have a prefix, and the name of the production application is used as the first prefix of the first message subject to obtain the second message subject, including:

[0081] The name of the production application is used as the first prefix of the first message topic to obtain the second message topic.

[0082] Among them, the connecting character between the first prefix and the first message topic can be a special character, space or other character, and there is no limitation here. For example, the first message topic corresponding to the production application app1 is topicA. At this time, the name of the production application is used as the first prefix of the first message topic, and the connecting character between the first prefix and the original first message topic is ".". At this time, the second message topic is "app1.topicA".

[0083] In another example, when determining that the first message subject is the message subject corresponding to the production application, the prefix string in the first message subject is not replaced, but a new first prefix is ​​directly added in front of the preset prefix of the first message subject, and the first prefix is ​​the name of the production application.

[0084] The first prefix and the preset prefix can be connected by special characters or spaces, and the first prefix can also be directly connected to the preset prefix. For example, the names of the production applications are app1 and app2, respectively, the first message topic corresponding to app1 is default.topicA, and the first message topic corresponding to app2 is default.topicB. At this time, the prefix string of default.topicA is not replaced, but the first prefix (app1) is added in front of the preset prefix of default.topicA, and the first prefix and the preset prefix are connected by the special character ".", so that the second message topic is app1.default.topicA, and the first prefix (app2) is added in front of the preset prefix of default.topicB, and the first prefix and the preset prefix are connected by the special character ".", so that the second message topic is app2.default.topicA.

[0085] It can be seen that in this example, by using the name of the production application as the prefix of the message topic corresponding to the production application, the second message topic is obtained, so that the unique production application corresponding to the message topic can be determined according to the prefix of each message topic, which is conducive to avoiding the situation where multiple production applications produce messages for a topic at the same time and cause data mixing.

[0086] S203: Determine a third message topic corresponding to the consuming application from the second message topic according to the mapping relationship between the consuming application and the first message topic and the mapping relationship between the consuming application and the producing application.

[0087] Among them, each topic is a message collection. The production application sends the generated messages to a specific topic, and the consumer application subscribes to and receives messages from the specific topic. The mapping relationship between the consumer application and the first message topic can be one-to-one, many-to-one, one-to-many or many-to-many. The mapping relationship between the consumer application and the production application can be one-to-one, many-to-one, one-to-many or many-to-many. There is no restriction on the mapping relationship between the consumer application and the first message topic, and the mapping relationship between the consumer application and the production application.

[0088] Specifically, the first message topic corresponding to the consumer application is determined according to the mapping relationship between the consumer application and the first message topic, and the production application corresponding to the consumer application is determined according to the mapping relationship between the consumer application and the production application. At this time, the third message topic corresponding to the consumer application in the second message topic is obtained by updating the second prefix of the first message topic corresponding to the consumer application to the name of the production application corresponding to the consumer application; or, after determining the first message topic corresponding to the consumer application and the production application corresponding to the consumer application, the third message topic corresponding to the consumer application in the second message topic is obtained by using the production application corresponding to the consumer application as the prefix of the first message topic corresponding to the consumer application.

[0089] Exemplarily, the first message topic corresponding to the consumer application app3 is default.topicA, and the production application corresponding to the consumer application app3 is app1. At this time, by updating the prefix of default.topicA to app1, the third message topic app1.topicA corresponding to the consumer application app3 is obtained; or, the first message topic corresponding to the consumer application app3 is default.topicA, and the production application corresponding to the consumer application app3 is app1. At this time, by using app1 as the prefix of default.topicA, the third message topic app1.default.topicA corresponding to the consumer application app3 is obtained.

[0090] In a possible implementation, the method further includes:

[0091] If the third message topic corresponding to the consuming application is not determined from the second message topic based on the mapping relationship between the consuming application and the first message topic and the mapping relationship between the consuming application and the producing application, the message topic corresponding to the consuming application is determined to be the first message topic.

[0092] Among them, if the first message topic corresponding to the consumer application cannot be determined based on the mapping relationship between the consumer application and the first message topic, and / or the production application corresponding to the consumer application cannot be determined based on the mapping relationship between the consumer application and the production application, then the third message topic corresponding to the consumer application cannot be determined from the second message topic.

[0093] Specifically, if after determining the first message topic corresponding to the consumer application based on the mapping relationship between the consumer application and the first message topic, the production application corresponding to the consumer application cannot be determined based on the mapping relationship between the consumer application and the production application, then the message topic corresponding to the consumer application is determined to be the first message topic, and if the first message topic has a preset prefix, the preset prefix of the first message topic will be retained; if the first message topic does not have a prefix, the first message topic will not be processed. For example, based on the mapping relationship between the consumer application and the first message topic, it is determined that the consumer application app3 corresponds to default.topicA. At this time, based on the mapping relationship between the consumer application and the production application, the production application corresponding to the consumer application cannot be determined, then the preset prefix of the first message topic is retained, and the message topic corresponding to the consumer application is determined to be default.topicA.

[0094] It can be seen that in this example, when the production application corresponding to the consumer application is not specified, the first message topic corresponding to the consumer application is directly determined as the message topic that needs to be consumed, ensuring that all consumer applications can consume normally. By directly specifying the consumer application to consume from the first message topic, there is no need to pay attention to the specific production application, and the relevant configuration work can be omitted.

[0095] S204, sending a third message topic.

[0096] After the server sends the corresponding third message topic to the consumer application, the consumer application can obtain the message in the corresponding third message topic by sending a request to the Kafka cluster. The message will be transferred from the third message topic stored in the Kafka cluster to the memory of the consumer application for processing.

[0097] Specifically, the consumer application obtains messages from the Kafka cluster through polling operations. Polling operations refer to the consumer application sending a request to the Kafka cluster to ask whether there are new messages available for consumption. The Kafka cluster will take out the message from the Topic partition corresponding to the consumer application according to the request of the consumer application and send the message to the consumer application. The message is returned to the consumer application in the form of ConsumerRecords, and the consumer application then processes these messages.

[0098] It can be seen that in this example, the second message topic is obtained by using the name of the production application as the first prefix of the first message topic corresponding to the production application, and then the third message topic uniquely corresponding to the consumption application is determined from multiple second message topics based on the mapping relationship between the consumption application and the first message topic and the mapping relationship between the consumption application and the production application. There is only one production application producing data under the same topic, the data under the topic will not be mixed, and there will be no data consumption errors when the subsequent consumption application consumes the topic, which is conducive to ensuring the correct processing of data under the topic.

[0099] See also Figure 4 , Figure 4 is a flow chart of another data processing method provided in an embodiment of the present application. Figure 4 As shown, the method comprises the following steps:

[0100] S401, according to the mapping relationship between the consumer application and the production application, and the mapping relationship between the message topic and the production application, determine the fourth message topic corresponding to the consumer application from the received message topic, and the prefix of the fourth message topic is used to indicate the production application name corresponding to the consumer application.

[0101] The execution subject of this method may be a client of a consumption application.

[0102] S402: Determine a message node of the fourth message topic according to the prefix of the fourth message topic.

[0103] There are multiple server nodes (brokers) in the kafka cluster, and messages are stored in partitions of multiple server nodes. A topic can have one or more partitions. After determining the fourth message topic corresponding to the consumer application, the partition where the message to be consumed by the consumer application is located can be determined based on the prefix of the fourth consumer topic or the fourth consumer topic. For example, based on the prefix "app1" of "app1.topicA", the message node is determined to be partition 1 of server node 1.

[0104] S403: Consume the messages in the fourth message topic at the message node of the fourth message topic.

[0105] The consumer application obtains messages from the assigned partitions through polling operations. Each partition has its own offset to record the reading position of the message. The consumer application ensures the integrity of the message based on the offset and ensures that the message is read in order.

[0106] Among them, it should be noted that Figure 4 The specific implementation of each operation in the method embodiment shown can be found in the above Figure 2The description in the method embodiment shown will not be repeated in detail here.

[0107] See also Figure 5 , Figure 5 is a structural diagram of a data processing device provided in an embodiment of the present application, such as Figure 5 As shown, the data processing device 500 includes:

[0108] A first determining unit 501 is used to determine a first message topic corresponding to each production application according to a mapping relationship between the production application and the message topic;

[0109] The acquisition unit 502 is configured to obtain a second message topic by using the name of the production application as a first prefix of the first message topic if the first message topics corresponding to the production applications are the same;

[0110] A second determining unit 503 is configured to determine a third message topic corresponding to the consuming application from the second message topic according to a mapping relationship between the consuming application and the first message topic and a mapping relationship between the consuming application and the producing application;

[0111] The sending unit 504 is configured to send the third message topic.

[0112] In one possible implementation, in terms of using the name of the production application as the first prefix of the first message topic to obtain the second message topic, the acquisition unit 502 is specifically used to: update the second prefix of the first message topic to the name of the production application as the first prefix of the first message topic, to obtain the second message topic, wherein the second prefix is ​​a preset prefix of the first message topic.

[0113] In one possible implementation, the acquisition unit 502 is also used to: if the first message topic corresponding to the production application is different, update the second prefix of the first message topic to the name of the production application as the first prefix of the first message topic, and obtain the second message topic, wherein the second prefix is ​​a preset prefix of the first message topic.

[0114] In a possible implementation, the second determination unit 503 is also used to: if the third message topic corresponding to the consumer application is not determined from the second message topic based on the mapping relationship between the consumer application and the first message topic, and the mapping relationship between the consumer application and the production application, then determine that the message topic corresponding to the consumer application is the first message topic.

[0115] In a possible implementation manner, the second determining unit 503 is further configured to: reserve a second prefix of the first message subject.

[0116] It is worth noting that the specific functional implementation of the data processing device 500 is shown in the above Figure 2 The description of the data processing method shown in the figure, for example, the first determination unit 501 is used to implement the relevant content of executing S201, the acquisition unit 502 is used to implement the relevant content of executing S202, the second determination unit 503 is used to implement the relevant content of executing S203, and the sending unit 504 is used to implement the relevant content of executing S204. Each unit or module in the data processing device 500 can be respectively or completely combined into one or more other units or modules to constitute, or one (some) of the units or modules can also be further divided into multiple smaller units or modules in function to constitute, which can achieve the same operation without affecting the realization of the technical effect of the embodiment of the present invention. The above-mentioned units or modules are divided according to logical functions. In practical applications, the function of one unit (or module) is implemented by multiple units (or modules), or the function of multiple units (or modules) is implemented by one unit (or module).

[0117] See also Figure 6 , Figure 6 is a schematic diagram of the structure of another data processing device provided in an embodiment of the present application, such as Figure 6 As shown, the data processing device 600 includes:

[0118] The first determining unit 601 is used to determine a fourth message topic corresponding to the consumer application from the received message topic according to the mapping relationship between the consumer application and the production application, and the mapping relationship between the message topic and the production application, wherein the prefix of the fourth message topic is used to indicate the production application name corresponding to the consumer application;

[0119] A second determining unit 602 is used to determine a message node of the fourth message topic according to a prefix of the fourth message topic;

[0120] The processing unit 603 is configured to consume the messages in the fourth message topic at the message node of the fourth message topic.

[0121] It is worth noting that the specific functional implementation of the data processing device 600 is shown in the above Figure 4The description of the data processing method shown in FIG. 6 is that, for example, the first determining unit 601 is used to implement the relevant content of executing S401, the second determining unit 602 is used to implement the relevant content of executing S402, and the processing unit 603 is used to implement the relevant content of executing S403. Each unit or module in the data processing device 600 can be separately or completely combined into one or more other units or modules to constitute, or one (some) of the units or modules can also be further divided into multiple smaller units or modules in function to constitute, which can achieve the same operation without affecting the realization of the technical effects of the embodiments of the present invention. The above-mentioned units or modules are divided according to logical functions. In practical applications, the function of one unit (or module) is implemented by multiple units (or modules), or the function of multiple units (or modules) is implemented by one unit (or module).

[0122] According to the description of the above method embodiment and related device embodiment, please refer to Figure 7 , Figure 7 is a structural diagram of a data processing device provided in an embodiment of the present application, Figure 7 The data processing device 700 shown includes a processor 701 , a memory 702 , a communication interface 703 and a bus 704 . The processor 701 , the memory 702 , and the communication interface 703 are connected to each other through the bus 704 .

[0123] Optionally, the memory 702 is a ROM, a static storage device, a dynamic storage device or a RAM.

[0124] The memory 702 can store executable program codes. When the executable program codes stored in the memory 702 are executed by the processor 701, the processor 701 and the communication interface 703 are used to execute Figure 2 or Figure 4 The various steps of the data processing method of the illustrated embodiment.

[0125] The processor 701 adopts a general-purpose CPU, a microprocessor, an application-specific integrated circuit ASIC, a GPU or one or more integrated circuits to execute relevant programs to perform the data processing method of the method embodiment of the present application.

[0126] The processor 701 can also be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the data processing method of the present application can be completed by the hardware integrated logic circuit in the processor 701 or the instruction in the form of software. Optionally, the processor 701 is a general-purpose processor, DSP, ASIC, FPGA or other programmable logic device, discrete gate or transistor logic device, discrete hardware component. The processor can implement or execute the methods, steps and logic block diagrams disclosed in the embodiments of the present application. The general-purpose processor is a microprocessor or the processor is any conventional processor, etc. The steps of the method disclosed in the embodiment of the present application can be directly embodied as a hardware decoding processor to be executed, or a combination of hardware and software modules in the decoding processor to be executed. The optional software module is located in a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register or other mature storage medium in the art. The storage medium is located in the memory 702, and the processor 701 reads the information in the memory 702, and combines its hardware to complete the functions required to be executed by the modules included in a data processing device 500 or a data processing device 600 of the embodiment of the present application, or executes the method of the present application Figure 2 or Figure 4 The data processing method of the illustrated embodiment.

[0127] The communication interface 703 uses a transceiver or other related device such as, but not limited to, a transceiver.

[0128] The bus 704 may include a path for transmitting information between various components of the data processing device 700 (eg, the memory 702 , the processor 701 , the communication interface 703 ).

[0129] It should be noted that although Figure 7 The data processing device 700 shown only shows a memory, a processor, and a communication interface. However, in the specific implementation process, those skilled in the art should understand that the data processing device 700 also includes other devices necessary for normal operation. At the same time, according to specific needs, those skilled in the art should understand that the data processing device 700 may also include hardware devices for implementing other additional functions. In addition, those skilled in the art should understand that the data processing device 700 may also include only the devices necessary for implementing the embodiments of the present application, and does not necessarily include Figure 7 All devices shown in .

[0130] An embodiment of the present application provides a computer-readable storage medium, in which a computer program for electronic data exchange is stored. The computer program includes execution instructions, and the execution instructions are used to execute part or all of the steps of any one of the data processing methods recorded in the above-mentioned data processing method embodiments. The above-mentioned computer includes an electronic terminal device.

[0131] An embodiment of the present application provides a computer program product, wherein the computer program product includes a computer program, and the computer program can be operated to enable a computer to perform part or all of the steps of any data processing method recorded in the above method embodiments. The computer program product can be a software installation package.

[0132] It should be noted that, for the sake of simplicity, the embodiments of any of the aforementioned data processing methods are described as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the described order of actions, because according to the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by the present application.

[0133] The above is a detailed introduction to the embodiments of the present application. Specific examples are used herein to illustrate the principles and implementation methods of a data processing method, device, equipment, storage medium and program product of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea; at the same time, for a person skilled in the art, based on the idea of ​​a data processing method, device, equipment, storage medium and program product of the present application, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

[0134] The present application is described with reference to the flowcharts and / or block diagrams of the methods, hardware products, and computer program products of the embodiments of the present application. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of the processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0135] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1The memory may include: a flash disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0136] Although the present application is described herein in conjunction with various embodiments, in the process of implementing the claimed application, those skilled in the art may understand and implement other variations of the disclosed embodiments by viewing the drawings, the disclosure, and the appended claims. In the claims, the word "comprising" does not exclude other components or steps, and "a" or "one" does not exclude multiple situations. Certain measures are recorded in different dependent claims, but this does not mean that these measures cannot be combined to produce good results.

[0137] A person skilled in the art can understand that all or part of the steps in the various methods of the method embodiments of any of the above-mentioned data processing methods can be completed by instructing related hardware through a program, and the program can be stored in a computer-readable memory, and the memory can include: a flash drive, a read-only memory (English: Read-Only Memory, abbreviated as: ROM), a random access memory (English: Random Access Memory, abbreviated as: RAM), a magnetic disk or an optical disk, etc.

[0138] It can be understood that any product that is controlled or configured to execute the processing method of the flowchart described in an embodiment of a data processing method of the present application, such as the device and computer program product of the above flowchart, falls within the scope of the related products described in the present application.

[0139] Obviously, those skilled in the art can make various changes and modifications to the data processing method, device, equipment, storage medium and program product provided by the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A data processing method, characterized in that: Applied to a server, the method comprises: Determine, according to the mapping relationship between production applications and message topics, a first message topic corresponding to each of the production applications; If the first message topics corresponding to the production applications are the same, the name of the production application is used as the first prefix of the first message topic to obtain the second message topic; Determine, from the second message topic, a third message topic corresponding to the consuming application according to a mapping relationship between the consuming application and the first message topic, and a mapping relationship between the consuming application and the producing application; The third message topic is sent.

2. The method according to claim 1, characterized in that The step of using the name of the production application as a first prefix of the first message subject to obtain the second message subject includes: The second prefix of the first message topic is updated to the name of the production application as the first prefix of the first message topic, to obtain a second message topic, where the second prefix is ​​a preset prefix of the first message topic.

3. The method according to claim 1, characterized in that The method further comprises: If the first message topic corresponding to the production application is different, the second prefix of the first message topic is updated to the name of the production application as the first prefix of the first message topic to obtain the second message topic, and the second prefix is ​​the preset prefix of the first message topic.

4. The method according to any one of claims 1 to 3, characterized in that: The method further comprises: If the third message topic corresponding to the consumer application is not determined from the second message topic based on the mapping relationship between the consumer application and the first message topic, and the mapping relationship between the consumer application and the production application, then the message topic corresponding to the consumer application is determined to be the first message topic.

5. The method according to claim 4, characterized in that The method further comprises: A second prefix of the first message subject is reserved.

6. A data processing method, characterized in that: Applied to a client of a consumer application, the method comprises: According to the mapping relationship between the consumer application and the production application, and the mapping relationship between the message topic and the production application, determine the fourth message topic corresponding to the consumer application from the received message topic, where the prefix of the fourth message topic is used to indicate the production application name corresponding to the consumer application; Determine a message node of the fourth message topic according to a prefix of the fourth message topic; The message node of the fourth message topic consumes the messages in the fourth message topic.

7. A data processing device, characterized in that: The device comprises: A first determining unit, configured to determine a first message topic corresponding to each production application according to a mapping relationship between the production application and the message topic; an acquiring unit, configured to obtain a second message topic by using the name of the production application as a first prefix of the first message topic if the first message topics corresponding to the production applications are the same; A second determining unit, configured to determine a third message topic corresponding to the consuming application from the second message topic according to a mapping relationship between the consuming application and the first message topic and a mapping relationship between the consuming application and the producing application; A sending unit is used to send the third message topic.

8. A data processing device, characterized in that: The device comprises: A first determining unit is used to determine a fourth message topic corresponding to the consumer application from the received message topic according to a mapping relationship between the consumer application and the production application, and a mapping relationship between the message topic and the production application, wherein the prefix of the fourth message topic is used to indicate a production application name corresponding to the consumer application; A second determining unit, configured to determine a message node of the fourth message topic according to a prefix of the fourth message topic; A processing unit is used to consume the messages in the fourth message topic at the message node of the fourth message topic.

9. A data processing device, characterized in that: The device comprises: A memory, a processor, and an executable program code stored in the memory and executable on the processor, wherein the processor executes the steps of the data processing method according to any one of claims 1 to 6 when executing the executable program code.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores executable program code, wherein the executable program code includes execution instructions, and the execution instructions are used to execute the steps of the data processing method according to any one of claims 1 to 6.

11. A computer program product, characterized in that The computer program product comprises a computer program, and the computer program is used to make a computer execute the steps of the data processing method according to any one of claims 1 to 6.

Images