A messaging system service apparatus and data processing device

Abstract

The application discloses a message system service device and data processing equipment, and relates to the technical field of communication. A data production processor sends service data issued by a data production device to a data storage module through an interface module, and a data consumption processor acquires the service data from the data storage module and issues the service data to a data consumption device through the interface module. The data production device and the data consumption device do not need to be respectively provided with a processor provided with a client of a message system, so that the coupling between each other can be reduced; the data production device and the data consumption device are respectively responsible for issuing and receiving service data, so that the work burden of maintenance personnel of the data production device and the data consumption device is not increased. In addition, the data consumption device cannot actively acquire service data from the message system service device, and can only passively receive the service data issued by the data production device and stored in the message system service device, so that the safety of the data production device is improved.

Description

Technical Field

[0001] The present invention relates to the field of communication technology, and in particular to a message system service device and data processing equipment. Background Art

[0002] Kafka is a commonly used distributed publish-subscribe messaging system. The messaging system acts as a medium between data producers and data consumers. Data producers generate and send business data to the messaging system, while data consumers subscribe to and retrieve business data from the messaging system to perform operations such as analyzing and processing the business data. In traditional designs, data producers and consumers require additional processors equipped with client-side messaging systems before using the messaging system for data processing. This allows them to store and consume data properly using the messaging system. This results in a strong coupling between the data producers, consumers, and messaging system. Furthermore, since the messaging system client is difficult to use, it increases the workload for maintenance personnel on the data producers and consumers. Furthermore, in traditional designs, data consumers can actively access all business data stored in the messaging system by the data producers, posing security risks to the data producers. Summary of the Invention

[0003] The purpose of the present invention is to provide a message system service device and data processing equipment, which can reduce the coupling between data production devices, data consumption devices and message systems, will not increase the difficulty of the maintenance personnel of data production devices and data consumption devices, and can ensure the safety of data production devices.

[0004] To solve the above technical problems, the present invention provides a message system service device, comprising:

[0005] Interface module;

[0006] A data production processor, configured to receive the business data sent by the data production device through the interface module, and send the business data to the data storage module;

[0007] a data consumption processor, configured to obtain the service data from the data storage module and send the service data to the data consumption device through the interface module so that the data consumption device can analyze and process the service data;

[0008] The data storage module is used to store the business data.

[0009] Preferably, the data storage module is further configured to generate a storage success signal after detecting that the business data is successfully stored, and send the storage success signal to the data production processor;

[0010] The data production processor is further configured to send a wake-up instruction to the data consumption device upon receiving the storage success signal.

[0011] The data consumption processor is specifically configured to obtain the service data from the data storage module upon receiving the wake-up instruction, and to distribute the service data to the data consumption device through the interface module.

[0012] Preferably, the data storage module is further configured to generate a storage failure signal upon detecting the storage failure of the service data, and to send the storage failure signal to the data production processor.

[0013] The data production processor is further configured to resend the service data to the data storage module upon receiving the storage failure signal.

[0014] Preferably, the data production processor is further configured to determine the data type of the service data, to determine a designated storage area corresponding to the data type in the data storage module, and to inform the data consumption processor of the designated storage area so that the data consumption processor obtains the service data from the designated storage area.

[0015] The data storage module is specifically configured to store the service data in the designated storage area.

[0016] Preferably, the data storage module is a kafka server.

[0017] Preferably, the data production processor is specifically configured to receive classified service data distributed by the data production device through the interface module, and to send the classified service data to the data storage module, wherein the classified service data is obtained by classifying the service data according to the user type of the data consumption device by the data production device.

[0018] The data storage module is specifically configured to store the classified service data.

[0019] The data consumption processor is specifically configured to obtain the classified service data from the data storage module, and to distribute the classified service data to the data consumption device through an interface corresponding to the classified service data in the interface module.

[0020] Preferably, the data storage module is specifically configured to store the classified service data in a classified service storage area, wherein the classified service storage area corresponds one-to-one to the user type of the data consumption device corresponding to the classified service data.

[0021] To solve the above technical problems, the application further provides a data processing device, comprising the message system service device, and further comprising a data production device and a data consumption device;

[0022] The data production device is configured to issue service data to a data production processor in the message system service device.

[0023] The data consumption device is configured to analyze and process the service data after receiving the service data issued by a data consumption processor in the message system service device.

[0024] Preferably, the data production device is specifically configured to issue the service data to the data production processor according to a data type agreed with the data consumption device in advance.

[0025] Preferably, the data production device is further configured to classify the service data according to a user type of the data consumption device and generate classified service data.

[0026] In summary, the application provides a message system service device and a data processing device, comprising an interface module, a data production processor, a data consumption processor and a data storage module. The data production processor sends service data issued by a data production device to the data storage module through the interface module, and the data consumption processor acquires the service data from the data storage module and issues the service data to a data consumption device through the interface module. As can be seen, the data production device and the data consumption device do not need to be respectively provided with a processor of a client installed with a message system, and the coupling between them can be reduced. The data production device and the data consumption device are responsible for issuing service data and receiving service data respectively, and the work burden of maintenance personnel of the data production device and the data consumption device will not be increased. Moreover, the data consumption device cannot actively acquire service data from the message system service device, but only passively receives the service data issued by the data production device and stored in the message system service device, thereby improving the security of the data production device. BRIEF DESCRIPTION OF DRAWINGS

[0027] In order to more clearly illustrate the technical solutions in the embodiments of the application, the following will briefly introduce the drawings needed to be used in the prior art and embodiments. Obviously, the drawings in the following description are only some embodiments of the application, and for those skilled in the art, other drawings can also be obtained without creative labor based on these drawings.

[0028] Figure 1 A structural schematic diagram of the message system service device provided by the application;

[0029] Figure 2A schematic diagram of a docking mode of a data processing device provided by the present invention. DETAILED DESCRIPTION

[0030] The core of the present invention is to provide a message system service device and data processing equipment, which can reduce the coupling between data production devices, data consumption devices and message systems, will not increase the difficulty of the maintenance personnel of data production devices and data consumption devices, and can ensure the safety of data production devices.

[0031] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present invention.

[0032] Please refer to Figure 1 , Figure 1 This is a structural diagram of a message system service device provided by the present invention, which includes:

[0033] Interface module 1;

[0034] The data production processor 2 is used to receive the business data sent by the data production device through the interface module 1 and send the business data to the data storage module 4;

[0035] The data consumption processor 3 is used to obtain the service data from the data storage module 4 and send the service data to the data consumption device through the interface module 1 so that the data consumption device can analyze and process the service data;

[0036] The data storage module 4 is used to store business data.

[0037] In a conventional design, before the data production device and the data consumption device use the message system, a client of the message system needs to be additionally introduced into the respective systems, which leads to a relatively strong coupling between the data production device, the data consumption device and the message system, and requires the maintenance personnel of the data production device and the data consumption device to understand certain basic knowledge of the message system, thereby increasing the additional work burden of the maintenance personnel. Moreover, when a special message system service party is responsible for managing the message system, the data production device and the data consumption device need to be directly connected with the message system service party, thereby causing the message system service party to have to understand relevant business knowledge, and increasing the communication cost between the data production device, the data consumption device and the message system service party. In addition, since the data consumption device can actively acquire any business data from the message system, and the data production device cannot control which data can be consumed by the data consumption device, the data production device has certain security problems.

[0038] Therefore, the present application provides a message system service device, which comprises an interface module 1, a data production processor 2, a data consumption processor 3 and a data storage module 4. In the present application, the data production processor 2 and the data consumption processor 3 are integrated in the message system service device, the message system service device is responsible for the operation and maintenance of business data, and provides services for the data production device and the data consumption device in the form of an interface, so that the client of the message system does not need to be introduced into the system of the data production device and the system of the data consumption device, thereby reducing the coupling between the data production device, the data consumption device and the message system service party.

[0039] Specifically, after the data production device generates business data, the data production device delivers the business data to the data production processor 2 in the message system service device through the interface module 1. It needs to be noted that the data production device can select specific business data to be delivered, and does not need to deliver all business data to the data production processor 2 through the interface module 1. For example, when there are multiple data consumption devices and each data consumption device wants to consume different business data, the data production device can select the business data corresponding to the target data consumption device to deliver to the data production processor 2.

[0040] After receiving the business data, the data production processor 2 sends the business data to the data storage module 4, and the data storage module 4 stores the business data after receiving the business data. The data storage module 4 can be a data storage in a server or a database, etc., which is not particularly limited in the present application. Moreover, in the case of multiple data consumption devices as mentioned above, the data storage can divide a unique storage area for each data consumption device, so as to better manage and deliver the business data.

[0041] The data consumption processor 3 retrieves service data from the data storage module 4 and distributes the service data to the data consumption device through the interface module 1, so that the data consumption device can consume the service data, such as performing operations such as analyzing and processing the service data. It should be noted that when there are multiple data consumption devices, the data production processor 2 will notify the data consumption processor 3 in advance to obtain service data from the designated storage area in the data storage module 4 to avoid errors in the service data distribution.

[0042] From the perspective of data production devices, data consumption devices, and the messaging system service provider, data production devices are solely responsible for producing and distributing business data, while data consumption devices only need to passively receive and subsequently consume business data. Therefore, the operations and maintenance personnel of both data production and consumption devices do not need to have any knowledge of the messaging system, reducing their workload. Furthermore, since data consumption devices can only passively receive data distributed by data production devices via the messaging system service provider, data production devices can choose which data to consume, thus ensuring the security of data production devices.

[0043] In summary, the present invention provides a message system service device, including an interface module 1, a data production processor 2, a data consumption processor 3 and a data storage module 4. The data production processor 2 sends the business data issued by the data production device to the data storage module 4 through the interface module 1, and the data consumption processor 3 obtains the business data from the data storage module 4 and sends the business data to the data consumption device through the interface module 1. It can be seen that the data production device and the data consumption device do not need to be respectively equipped with a processor of a client installed with the message system, which can reduce the coupling between each other; the data production device and the data consumption device are respectively responsible for issuing business data and receiving business data, which will not increase the workload of the maintenance personnel of the data production device and the data consumption device. In addition, the data consumption device cannot actively obtain business data from the message system service device, but can only passively receive the business data issued by the data production device and stored in the message system service device, thereby improving the security of the data production device.

[0044] Based on the above embodiment:

[0045] As a preferred embodiment, the data storage module 4 is further configured to generate a storage success signal after detecting that the business data is successfully stored, and send the storage success signal to the data production processor 2;

[0046] The data production processor 2 is further configured to send a wake-up instruction to the data consumption device upon receiving a storage success signal;

[0047] The data consumption processor 3 is specifically configured to obtain service data from the data storage module 4 upon receiving a wake-up instruction, and send the service data to the data consumption device through the interface module 1 .

[0048] Taking into account that unexpected situations may occur during the process of storing business data in the data storage module 4, resulting in failure of business data storage, which may in turn affect the consumption of business data by the data consumption device, therefore, in this embodiment, the data storage module 4 can also detect whether the business data is stored successfully, and generate a storage success signal after detecting that the business data is stored successfully. In this embodiment, the storage success signal will be sent to the data production processor 2. After receiving the storage success signal, the data production processor 2 sends a wake-up instruction to the data consumer, and then the data consumption processor 3 can obtain the business data from the data storage module 4, which further ensures the security of the entire process of processing business data. After the data consumption device obtains the business data, it sends the business data to the data consumption device through the interface module 1 so that the data consumption device can consume the business data.

[0049] It should be noted that in this embodiment, the data storage module 4 can also directly send a storage success signal to the data consumption device after detecting that the business data is successfully stored. After receiving the storage success signal, the data consumption device obtains the business data from the data storage module 4 and sends it to the data consumption device through the interface module 1.

[0050] In summary, in this embodiment, the data consumption processor 3 can obtain business data only after the data storage module 4 detects that the data is stored successfully, further ensuring the security of business data processing by the business system composed of the data production device, data consumption device and message system service party.

[0051] As a preferred embodiment, the data storage module 4 is further configured to generate a storage failure signal after detecting a failure in storing the service data, and send the storage failure signal to the data production processor 2;

[0052] The data production processor 2 is further configured to resend the service data to the data storage module 4 after receiving a storage failure signal.

[0053] Considering that the data storage module 4 may fail to store business data, which may affect the data consumption device's consumption of business data, in this embodiment, the data storage module 4 also generates a storage failure signal when it detects a data storage failure. At the same time, to ensure that the data consumption device can consume business data normally, the data storage module 4 will send the storage failure signal to the data production processor 2. When the data production processor 2 receives the storage failure signal, it will resend the business data to the data storage module 4 so that the data storage module 4 can store the business data again.

[0054] In summary, in this embodiment, after the data storage module 4 detects a data storage failure, the data production processor 2 can still send business data to the data storage module 4 again to ensure stable operation of the business system.

[0055] As a preferred embodiment, the data production processor 2 is further configured to determine the data type of the business data, determine a designated storage area corresponding to the data type in the data storage module 4, and inform the data consumption processor 3 of the designated storage area so that the data consumption processor 3 obtains the business data from the designated storage area;

[0056] The data storage module 4 is specifically used to store business data in a designated storage area.

[0057] Before the data production device produces data, the data production device and the data consumption device will agree in advance on the data type of the business data. For example, different data types correspond to different data consumption devices. Therefore, the data production processor 2 will also identify the data type of the business data after the data production produces the business data. In order to achieve the purpose of providing business data of different data types to different data consumption devices, the data storage module 4 also divides different storage areas for business data of different data types. After determining the data type of the business data, the data production processor 2 determines the designated storage area corresponding to the data type in the data storage module 4 and notifies the data consumption processor 3 of the designated storage area. The data consumption processor 3 then obtains the business data from the designated storage area and sends the business data to the data consumption device.

[0058] In this embodiment, different types of business data are stored in different storage areas, which further optimizes the business data processing process of the business system.

[0059] As a preferred embodiment, the data storage module 4 is a Kafka server.

[0060] In this embodiment, the data storage module 4 is a Kafka server, and the business data can be stored in the data storage of the Kafka server. The Kafka messaging system has the advantages of high throughput, low latency, and high reliability, which further optimizes the process of processing business data using the messaging system service device.

[0061] It should also be noted that business data can be stored in a specified topic or a specified partition in the Kafka server, and this application does not impose any special restrictions on this.

[0062] As a preferred embodiment, the data production processor 2 is specifically configured to receive the classified service data issued by the data production device through the interface module 1, and send the classified service data to the data storage module 4, wherein the classified service data is data obtained by classifying service data by the data production device according to the user type of the data consumption device.

[0063] The data storage module 4 is specifically configured to store the classified service data.

[0064] The data consumption processor 3 is specifically configured to obtain the classified service data from the data storage module 4, and issue the classified service data to the data consumption device through the interface corresponding to the classified service data in the interface module 1.

[0065] To enhance the security of the data production device, the service data issued by the data production device can be classified service data, wherein the classified service data is data obtained by classifying service data by the data production device according to the user type of the data consumption device. By specifying different data consumption devices to consume different types of classified service data, the data production device can control the risk, avoiding the risk that the data consumption device can obtain any service data from the message system service in the traditional design scheme.

[0066] Specifically, after the data production device produces the classified service data, the data production processor 2 receives the classified service data through the interface module 1 and issues the classified service data to the data storage module 4, and the data storage module 4 stores the classified service data. It should be noted that the data storage module 4 can store different types of classified service data in different storage areas, for example, when the data storage module 4 is a kafka server, the kafka server can store different types of classified service data in different topics.

[0067] In addition, the data production device and the data consumption device can pre-arrange the transmission type and the receiving format of the classified service data, and bind different types of classified service data with different interfaces in the interface module 1, and different interfaces correspond to different data consumption devices, so that the data consumption processor 3 issues the classified service data to the data consumption device through the interface corresponding to the classified service data in the interface module 1 after obtaining the classified service data.

[0068] In summary, in the embodiment, the service data is classified according to the user type of the data consumption device, and the classified service data is issued to the data consumer through the interface corresponding to the classified service data, thereby ensuring the security of the data production device.

[0069] As a preferred embodiment, the data storage module 4 is specifically configured to store the classified service data in the classified service storage area, wherein the classified service storage area corresponds one-to-one to the user type of the data consumption device corresponding to the classified service data.

[0070] Taking into account that different data consumption devices correspond to different classified business data, in order to facilitate the data consumption processor 3 to obtain the classified business data corresponding to the data consumption device, in this embodiment, the data storage module 4 is divided into multiple areas, and different classified business data are stored in different classified business storage areas. The classified business storage areas correspond one-to-one to the user types of the data consumption devices corresponding to the classified business data, ensuring the convenience of processing business data using the message system service device.

[0071] The present application also provides a data processing device, comprising the above-mentioned message system service device, and also comprising a data production device and a data consumption device;

[0072] The data production device is used to send business data to the data production processor 2 in the message system service device;

[0073] The data consumption device is used to analyze and process the service data after receiving the service data sent by the data consumption processor 3 in the message system service device.

[0074] The present invention provides a data processing device, including a message system service device, a data production device, and a data consumption device, wherein the message system service device includes an interface module 1, a data production processor 2, a data consumption processor 3, and a data storage module 4. The data production processor 2 sends the business data issued by the data production device to the data storage module 4 through the interface module 1, and the data consumption processor 3 obtains the business data from the data storage module 4 and sends the business data to the data consumption device through the interface module 1. It can be seen that the data production device and the data consumption device do not need to be respectively equipped with a processor with a client of the message system installed, which can reduce the coupling between them; the data production device and the data consumption device are respectively responsible for issuing business data and receiving business data, which does not increase the workload of the maintenance personnel of the data production device and the data consumption device. In addition, the data consumption device cannot actively obtain business data from the message system service device, but can only passively receive the business data stored in the message system service device and issued by the data production device, thereby improving the security of the data production device.

[0075] For the relevant introduction of a data processing device provided by the present invention, please refer to the embodiment of the above-mentioned message system service device, and this application will not go into details here.

[0076] Based on the above embodiment:

[0077] As a preferred embodiment, the data production device is specifically used to send the business data to the data production processor 2 according to the data type pre-agreed with the data consumption device.

[0078] In this embodiment, the data production device will generate business data according to the data type agreed in advance with the data consumption device, so that the data production processor 2 will store the business data in the designated storage area after determining the data type of the business data, further optimizing the business system's processing of the business data.

[0079] As a preferred embodiment, the data production device is further configured to classify the service data according to the user type of the data consumption device and generate classified service data.

[0080] In this embodiment, the data production device categorizes business data according to the user type of the data consumption device and generates classified business data. After the data production device generates the classified business data, the data production processor 2 receives the classified business data through the interface module 1 and sends the classified business data to the data storage module 4, which stores the classified business data. By designating different data consumption devices to consume different types of classified business data, the data production device can control risks, avoiding the risk of data consumption devices being able to obtain any business data from the message system service provider, as occurs in traditional designs.

[0081] Please refer to Figure 2 , Figure 2 A schematic diagram of a docking mode of a data processing device provided by the present invention. The process of processing business data by the data processing device composed of a data production device, a data consumption device and a message system service device can be roughly as follows: the data production device and the data consumption device pre-agree on the data type required by the data consumption device, and then the data consumption device develops a corresponding interface according to the agreed data type and provides the interface to the data production device. In order to meet the needs of the message system service device to send business data to the data consumption device, the data production device informs the message system service device of the interface and the identification method of the data type. After completing the above preparation process, after producing the business data, the data production device can send the data to the message system service device through the developed interface according to the pre-agreed data type. The message system service device then uses the interface to send the business data to the data consumption device, so that the data consumption device can consume the business data, completing the whole process of business data processing.

[0082] In summary, the data processing device provided by this application avoids the need for data production and consumption devices to incorporate the client side of the message system service device. This reduces coupling between data processing devices, lowers maintenance costs for data processing devices, avoids the risk of data leakage within the data storage module 4 of the message system service device, reduces communication costs for all parties involved, and lowers the technical requirements for maintenance personnel of data production and consumption devices. Furthermore, data consumption devices no longer interact with the message system service device; the data consumed by the data consumption devices is entirely determined by the data production device, thereby improving the security of the data production device.

[0083] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0084] It should also be noted that, in this specification, the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, article, or apparatus comprising a series of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or apparatus. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or apparatus comprising the element.

[0085] The above description of the disclosed embodiments is intended to enable one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention is not limited to the embodiments shown herein but is intended to conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A message system service device, characterized in that: include: Interface module; A data production processor, configured to receive the business data sent by the data production device through the interface module, and send the business data to the data storage module; a data consumption processor, configured to obtain the service data from the data storage module and send the service data to the data consumption device through the interface module so that the data consumption device can analyze and process the service data; The data storage module is used to store the business data; The data storage module is further configured to generate a storage success signal after detecting that the business data is successfully stored, and send the storage success signal to the data production processor; The data production processor is further configured to send a wake-up instruction to the data consumption processor upon receiving the storage success signal; The data consumption processor is specifically configured to obtain the service data from the data storage module upon receiving the wake-up instruction, and send the service data to the data consumption device through the interface module.

2. The message system service device according to claim 1, wherein: The data storage module is further configured to generate a storage failure signal after detecting that the business data storage fails, and send the storage failure signal to the data production processor; The data production processor is further configured to resend the service data to the data storage module after receiving the storage failure signal.

3. The message system service device according to claim 1, wherein: The data production processor is further configured to determine a data type of the business data, determine a designated storage area corresponding to the data type in the data storage module, and inform the data consumption processor of the designated storage area so that the data consumption processor obtains the business data from the designated storage area; The data storage module is specifically configured to store the business data in the designated storage area.

4. The message system service device according to claim 1, wherein: The data storage module is a Kafka server.

5. The message system service device according to any one of claims 1 to 4, characterized in that: The data production processor is specifically configured to receive, through the interface module, the classified business data issued by the data production device, and send the classified business data to the data storage module, wherein the classified business data is data obtained after the data production device classifies the business data according to the user type of the data consumption device; The data storage module is specifically used to store the classified business data; The data consumption processor is specifically configured to obtain the classified service data from the data storage module, and send the classified service data to the data consumption device through an interface corresponding to the classified service data in the interface module.

6. The message system service device according to claim 5, wherein: The data storage module is specifically configured to store the classified service data in a classified service storage area, wherein the classified service storage area corresponds one-to-one to the user type of the data consumption device corresponding to the classified service data.

7. A data processing device, characterized in that: The message system service device according to any one of claims 1 to 6 further comprises a data production device and a data consumption device; The data production device is used to send business data to the data production processor in the message system service device; The data consumption device is used to analyze and process the service data after receiving the service data sent by the data consumption processor in the message system service device.

8. The data processing device according to claim 7, characterized in that The data production device is specifically used to send the business data to the data production processor according to the data type pre-agreed with the data consumption device.

9. The data processing device according to claim 7, wherein The data production device is further configured to classify the business data according to the user type of the data consumption device and generate classified business data.

Images