A data push method and device

Abstract

The present invention discloses a data push method and device, and relates to the field of computer technology. A specific implementation of the method includes: the back-end network communication middleware receives the change data, parses the attribute information of the change data, and obtains the change protocol; uses the business processing logic corresponding to the change protocol to perform business processing on the change data to obtain the processing result; determines the client corresponding to the change protocol, and pushes the processing result to the client. In this implementation, the entire data flow is from bottom to top, layer by layer, and the Netty Server performs business logic processing on the change data, and the final processing result is actively pushed to the client, which solves the data delay problem caused by the existing data pulling. The overall architecture is simple, suitable for all scenarios that need to push data, and has universal applicability.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a data pushing method and device. Background Art

[0002] The current common real-time data push method monitors database data, flushes the changed data to the Redis cluster, and finally uses ajax or websocket to pull the changed data on the front end to display it to the user. Some scenarios have urgent requirements for real-time data, which led to the LMAX platform framework, which has excellent real-time data processing performance.

[0003] In the process of implementing the present invention, the inventors found that the prior art has at least the following problems: data is obtained by pulling, and the pulling method is generally ajax or websocket based on a fixed frequency to refresh the page, so the push has a certain delay and is not suitable for scenarios that are sensitive to data real-time requirements. The LMAX platform framework is only designed for specific scenarios and the overall architecture is complex and not universal. Summary of the invention

[0004] In view of this, an embodiment of the present invention provides a data push method and device, which can at least solve the problem that data push in the prior art has a time delay or lacks universality.

[0005] To achieve the above object, according to one aspect of an embodiment of the present invention, a data push method is provided, comprising:

[0006] The back-end network communication middleware receives the change data, parses the attribute information of the change data, and obtains the change agreement;

[0007] Using the business processing logic corresponding to the change agreement, the change data is subjected to business processing to obtain a processing result;

[0008] A client corresponding to the changed protocol is determined, and the processing result is pushed to the client.

[0009] Optionally, before the back-end network communication middleware receives the change data, the method further includes:

[0010] In response to the operation of writing data into the database, real-time monitoring of the data in the database is triggered, so that when changes to the data are detected, the changed data is sent to the back-end network communication middleware.

[0011] Optionally, before responding to the operation of writing data into the database, the method further includes:

[0012] By calling the interface service, the data corresponding to each interface is obtained and written into the database.

[0013] Optionally, performing business processing on the changed data by using business processing logic corresponding to the changed protocol includes:

[0014] Sending the changed data and the business processing logic to a local lock-free queue;

[0015] The local lock-free queue performs business processing on the changed data using the business processing logic, obtains the processing result and sends it to the back-end communication middleware.

[0016] To achieve the above object, according to another aspect of an embodiment of the present invention, a data push device is provided, wherein the backend network communication middleware includes a parsing module, a processing module and a push module, wherein:

[0017] A parsing module, used for receiving the change data, parsing the attribute information of the change data, and obtaining the change agreement;

[0018] A processing module, used to perform business processing on the change data using the business processing logic corresponding to the change agreement to obtain a processing result;

[0019] The push module is used to determine the client corresponding to the changed protocol and push the processing result to the client.

[0020] Optionally, a monitoring module is also included, which is used to:

[0021] In response to the operation of writing data into the database, real-time monitoring of the data in the database is triggered, so that when changes to the data are detected, the changed data is sent to the back-end network communication middleware.

[0022] Optionally, the monitoring module is used to obtain data corresponding to each interface by calling an interface service, and write the data into a database.

[0023] Optionally, the processing module is used to: send the change data and the business processing logic to a local lock-free queue;

[0024] The local lock-free queue performs business processing on the changed data using the business processing logic, obtains the processing result and sends it to the back-end communication middleware.

[0025] To achieve the above objective, according to another aspect of an embodiment of the present invention, a data push electronic device is provided.

[0026] The electronic device of an embodiment of the present invention includes: one or more processors; a storage device for storing one or more programs, when the one or more programs are executed by the one or more processors, the one or more processors implement any of the above-mentioned data push methods.

[0027] To achieve the above objective, according to another aspect of an embodiment of the present invention, a computer-readable medium is provided, on which a computer program is stored, and when the program is executed by a processor, any of the above-mentioned data push methods is implemented.

[0028] According to the solution provided by the present invention, an embodiment of the above invention has the following advantages or beneficial effects: through the data perception function of canal, the perceived change data is actively pushed to the Netty Server, and the NettyServer pushes the change data to the Disruptor component for business logic processing, and the final processing result is actively pushed to the client, thereby reducing latency, and the overall architecture is simple, suitable for all scenarios that require pushing data, and has universal applicability.

[0029] The further effects of the above-mentioned non-conventional optional manner will be described below in conjunction with the specific implementation manner. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] The accompanying drawings are used to better understand the present invention and do not constitute an improper limitation of the present invention.

[0031] Figure 1 This is a schematic diagram of the main flow of a data push method according to an embodiment of the present invention;

[0032] Figure 2 is a flowchart of a specific data push method according to an embodiment of the present invention;

[0033] Figure 3 is a data push flow diagram according to an embodiment of the present invention;

[0034] Figure 4 is a schematic diagram of main modules of a data push device according to an embodiment of the present invention;

[0035] Figure 5 is an exemplary system architecture diagram to which embodiments of the present invention may be applied;

[0036] Figure 6 It is a schematic diagram of the structure of a computer system of a mobile device or a server suitable for implementing an embodiment of the present invention. DETAILED DESCRIPTION

[0037] The following is a description of exemplary embodiments of the present invention in conjunction with the accompanying drawings, including various details of the embodiments of the present invention to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present invention. Similarly, for clarity and conciseness, the description of well-known functions and structures is omitted in the following description.

[0038] The terms involved in this plan are explained as follows:

[0039] MySQL: An open source database system. Mainly used for data access operations.

[0040] Netty: Network communication middleware. Mainly used for network communication operations. Mainly used for the backend.

[0041] Disruptor: A local lock-free queue. Mainly used in scenarios where queue processing is required.

[0042] WebSocket: Network communication middleware. Mainly used for network communication operations. Mainly used for the front end.

[0043] MQ: message queue.

[0044] LMAX: A complex real-time data processing architecture used in the financial sector. The data is extremely real-time, but the architecture is very complex.

[0045] Ajax: Local data refresh technology used by the front end.

[0046] This solution can be applied to scenarios where data needs to be pushed, especially those that are extremely sensitive to data real-time requirements, such as real-time interactive data display in the financial field; real-time cluster monitoring in the Internet field, such as changes in underlying monitoring data, can be directly pushed to the monitoring screen or related business parties through the framework link; hardware control in the Internet of Things field, such as when the temperature in the greenhouse is higher than 40 degrees, the temperature alarm information is synchronized to the framework of this solution. This alarm information can be pushed to the upstream fan control device in real time to automatically turn on the fan. The entire process is completely automatic and does not require any human intervention.

[0047] See also Figure 1 , which shows a main flow chart of a data push method provided by an embodiment of the present invention, comprising the following steps:

[0048] S101: The back-end network communication middleware receives the change data, parses the attribute information of the change data, and obtains the change protocol;

[0049] S102: Performing business processing on the changed data using the business processing logic corresponding to the changed protocol to obtain a processing result;

[0050] S103: Determine the client corresponding to the changed protocol, and push the processing result to the client.

[0051] In the above implementation, for steps S101 to S103, the user obtains the data corresponding to each interface (i.e., business data) by calling the provided interface service, and writes the data into the MySQL database through this service. The data under different projects are different. For example, the data in the Internet monitoring large screen project is generally the CPU information and memory information of Docker, and the data in the Internet of Things project is generally the indicator exceeding standard information collected by a certain collection device.

[0052] Furthermore, there are many types of data in an interface, and which data needs to be monitored can be selected according to needs. In order to distinguish the data of different interfaces, data separation can be performed for different interfaces in the database.

[0053] When data is written into the MySQL database, canal can be automatically triggered to monitor the data in real time. Canal monitors whether the data in the MySQL database has changed. If so, it will actively select the changed data and send it to the consumer, which here refers to the back-end Netty Server (backend network communication middleware).

[0054] In addition to using the above-mentioned Netty Server implementation, this server can also be implemented using middleware such as Mina, depending on the business. Since Netty is based on Mina, it is a completely asynchronous model with better efficiency and reliability, so this solution prefers Netty.

[0055] Netty Server not only has the data consumption function (that is, pushing data to the Client), but also has the Client heartbeat detection, encoding and decoding, login authentication, disconnection and reconnection functions to ensure the stability of the overall link, including:

[0056] Heartbeat detection: It is mainly a way for the server and client to detect each other's activity, that is, both parties confirm that the other party is still connected to each other.

[0057] Encoding and decoding: The underlying network communication requires converting the binary stream to facilitate transmission on the network.

[0058] Login authentication: unified login service, machines that are not logged in cannot connect.

[0059] Reconnect after disconnection: Due to network factors, the system can automatically reconnect after disconnection.

[0060] Usually, after the data is changed, the change information will be stored in the attributes of the data, such as the change time, change content, and device information. The change information in this solution also includes the change protocol, such as 0S01, 0S02. Different change protocols indicate different operations on the data.

[0061] Netty Server processes the changed data according to the business processing logic corresponding to the changed protocol. For example, it categorizes the collected data, formats the monitoring data, performs switch circuit analysis on the hardware control data, and finally sends the processed data to the Client.

[0062] In addition, commands can also be encapsulated in the form of protocols, so the command corresponding to the changed protocol can be determined first, and the changed data can be processed by the business processing logic corresponding to the command. For example, the command to obtain cpu data is 0x01|cpu|load.

[0063] There are many types of Client here, which are determined by changing the protocol. For example, 0S01 means pushing the processing results to the mobile phone, 0S02 means pushing the processing results to the web page, and 0S03 means pushing the processing results to the email. The push methods of different clients are different. If it is a large monitoring screen, it will be pushed to the connected websocket client; if it is a hardware control project, it will be pushed to the control hardware device; if it is financial data, it will be pushed to the user in real time, etc. The display framework process of different companies is different. The core of this solution is how to push data in real time, so the display part will not be elaborated.

[0064] The method provided in the above embodiment actively pushes the perceived change data to the Netty Server through the data perception function of canal, and the Netty Server performs business logic processing on the change data, and finally actively pushes the processing result to the client, thereby solving the delay problem caused by the existing data pulling; the overall architecture is simple, suitable for all scenarios that need to push data, and has universal applicability.

[0065] See also Figure 2 , shows a schematic flow chart of an optional data push method according to an embodiment of the present invention, comprising the following steps:

[0066] S201: In response to an operation of writing data into a database, triggering real-time monitoring of the data in the database, so as to send the changed data to a backend network communication middleware when a change in the data is detected;

[0067] S202: The back-end network communication middleware receives the change data, parses the attribute information of the change data, and obtains the change protocol;

[0068] S203: Determine the business processing logic corresponding to the changed protocol, and send the changed data and the business processing logic to a local lock-free queue;

[0069] S204: The local lock-free queue performs business processing on the changed data using the business processing logic, obtains the processing result and sends it to the backend communication middleware;

[0070] S205: The backend communication middleware determines the client corresponding to the changed protocol, and pushes the processing result to the client.

[0071] In the above embodiment, for steps S201, S202 and S205, see Figure 1 The description of steps S101 and S103 is not repeated here.

[0072] In the above implementation, for steps S203 and S204, to ensure the efficiency of the communication link, the server will push the changed data to other components for business logic processing, such as Disruptor components, MQ, etc., which can be selected according to the business. Since the Disruptor component is lock-free and efficient, the Disruptor component is preferably used in this solution.

[0073] First, Netty Server determines the specific business processing logic corresponding to the change protocol, and hands the business processing logic to the disruptor for processing. It should be noted that the relationship between the change protocol and the business processing logic is pre-defined. Here, it is not to receive the client request and then pull it, but to actively analyze the business processing logic and actively push the data to the client after processing.

[0074] The Disruptor component can be considered as a business thread pool to reduce the service congestion of the Netty Server. For example, complex business operations are usually performed in the business thread pool instead of in the Netty Server.

[0075] The Disruptor component receives the changed data and business processing logic from the Netty Server, and uses the business processing logic to process the data. For example, it classifies the collected data, formats the monitoring data, performs switch circuit analysis on the hardware control data, etc. Finally, the processed data is sent to the Netty Server, which then sends it to the Client.

[0076] In addition, canal cannot currently send change data directly to the Disruptor component. Although the process is relatively less than this solution, it requires more additional configuration, so it cannot achieve high concurrency and high availability.

[0077] The method provided in the above embodiment is based on the Disruptor component to assist in processing business logic, so as to reduce the service pressure of NettyServer and reduce its blocking situation.

[0078] See also Figure 3 , shows a data push flow diagram according to an embodiment of the present invention, including the following steps:

[0079] 1. Data is written to the Internet by the user via PC, mobile phone, etc., and the Internet writes the data into the MySQL database;

[0080] 2. Use canal to monitor the data in the MySQL database, and when the data is changed, transmit the changed data to the Netty Server;

[0081] 3. Netty Server receives the changed data, parses the change protocol in the change attribute, determines the business processing logic corresponding to the change protocol, and sends the changed data and business processing logic to the Disruptor component;

[0082] 4. The Disruptor component decouples data from business processing to prevent complex business calculations from blocking and causing NettyServer performance degradation. This component is used for business logic processing because it is lock-free and efficient;

[0083] 5. The Disruptor component transmits the processing results to the Netty Server, which pushes the processing results to the User through websocket, or directly pushes the processing results to the User.

[0084] The method provided by the embodiment of the present invention has the following beneficial effects:

[0085] 1) The entire data flow is from bottom to top, layer by layer: after the data enters the MySQL database, the changed data will be actively pushed to the Netty Server through canal. After the Netty Server receives and processes the changed data, it will push the processing results to the connected client.

[0086] 2) The architecture is simple and has great versatility, such as real-time data push of IoT hardware control devices and real-time data push of the Internet.

[0087] See also Figure 4, shows a schematic diagram of the main modules of a data push device 400 provided in an embodiment of the present invention, the back-end network communication middleware includes a parsing module 401, a processing module 402 and a push module 403, wherein:

[0088] The parsing module 401 is used to receive the change data, parse the attribute information of the change data, and obtain the change agreement;

[0089] The processing module 402 is used to perform business processing on the changed data using the business processing logic corresponding to the changed protocol to obtain a processing result;

[0090] The push module 403 is used to determine the client corresponding to the changed protocol and push the processing result to the client.

[0091] The implementation device of the present invention also includes a monitoring module 404 (not shown in the figure), which is used to: in response to the operation of writing data into the database, trigger real-time monitoring of the data in the database, so as to send the changed data to the back-end network communication middleware when the data is monitored to change.

[0092] In the implementation device of the present invention, the monitoring module 404 is used to obtain data corresponding to each interface by calling the interface service and write the data into the database.

[0093] In the implementation device of the present invention, the processing module 402 is used to:

[0094] Sending the changed data and the business processing logic to a local lock-free queue;

[0095] The local lock-free queue performs business processing on the changed data using the business processing logic, obtains the processing result and sends it to the back-end communication middleware.

[0096] In addition, the specific implementation content of the device described in the embodiment of the present invention has been described in detail in the method described above, so the repeated content will not be described again here.

[0097] Figure 5 An exemplary system architecture 500 is shown in which embodiments of the present invention may be applied.

[0098] like Figure 5 As shown, the system architecture 500 may include terminal devices 501, 502, 503, a network 504 and a server 505 (just an example). The network 504 is used to provide a medium for communication links between the terminal devices 501, 502, 503 and the server 505. The network 504 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0099] The user can use the terminal devices 501, 502, 503 to interact with the server 505 through the network 504 to receive or send messages, etc. Various communication client applications can be installed on the terminal devices 501, 502, 503.

[0100] The terminal devices 501 , 502 , and 503 may be various electronic devices having display screens and supporting web browsing, and the server 505 may be a server providing various services.

[0101] It should be noted that the method provided in the embodiment of the present invention is generally executed by the server 505 , and accordingly, the device is generally set in the server 505 .

[0102] It should be understood that Figure 5 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0103] Reference below Figure 6 , which shows a schematic diagram of the structure of a computer system 600 of a terminal device suitable for implementing an embodiment of the present invention. Figure 6 The terminal device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0104] like Figure 6 As shown, the computer system 600 includes a central processing unit (CPU) 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage part 608 into a random access memory (RAM) 603. In the RAM 603, various programs and data required for the operation of the system 600 are also stored. The CPU 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0105] The following components are connected to the I / O interface 605: an input section 606 including a keyboard, a mouse, etc.; an output section 607 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 608 including a hard disk, etc.; and a communication section 609 including a network interface card such as a LAN card, a modem, etc. The communication section 609 performs communication processing via a network such as the Internet. A drive 610 is also connected to the I / O interface 605 as needed. A removable medium 611, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 610 as needed, so that a computer program read therefrom is installed into the storage section 608 as needed.

[0106] In particular, according to the embodiments disclosed in the present invention, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present invention include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 609, and / or installed from the removable medium 611. When the computer program is executed by the central processing unit (CPU) 601, the above-mentioned functions defined in the system of the present invention are executed.

[0107] It should be noted that the computer-readable medium shown in the present invention may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present invention, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In the present invention, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, optical cable, RF, etc., or any suitable combination of the above.

[0108] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present invention. In this regard, each box in the flow chart or block diagram can represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0109] The modules involved in the embodiments of the present invention may be implemented by software or hardware. The modules described may also be set in a processor. For example, they may be described as: a processor includes a parsing module, a processing module, and a push module. The names of these modules do not, in some cases, constitute limitations on the modules themselves. For example, the push module may also be described as a "data push module".

[0110] As another aspect, the present invention further provides a computer-readable medium, which may be included in the device described in the above embodiment; or may exist independently without being assembled into the device. The above computer-readable medium carries one or more programs, and when the above one or more programs are executed by a device, the device includes:

[0111] The back-end network communication middleware receives the change data, parses the attribute information of the change data, and obtains the change agreement;

[0112] Using the business processing logic corresponding to the change agreement, the change data is subjected to business processing to obtain a processing result;

[0113] A client corresponding to the changed protocol is determined, and the processing result is pushed to the client.

[0114] According to the technical solution of the embodiment of the present invention, the perceived change data is actively pushed to the Netty Server through the data perception function of the canal, and the Netty Server pushes the change data to the Disruptor component for business logic processing, and the final processing result is pushed to the client. The overall architecture is simple, suitable for all scenarios that need to push data, and has universal applicability.

[0115] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions may occur depending on design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A data push method, It is characterized in that include: The back-end network communication middleware receives the change data, analyzes the attribute information of the change data, and obtains the change protocol; wherein, by calling the interface service, the data corresponding to each interface is obtained; different change protocols indicate different operations on the data; Using the business processing logic corresponding to the change protocol, the change data is subjected to business processing to obtain a processing result; wherein, using the business processing logic corresponding to the change protocol to perform business processing on the change data includes: sending the change data and the business processing logic to a local lock-free queue; the local lock-free queue performs business processing on the change data using the business processing logic, obtains a processing result and sends it to the back-end communication middleware; A client corresponding to the changed protocol is determined, and the processing result is pushed to the client.

2. The method according to claim 1, It is characterized in that Before the back-end network communication middleware receives the change data, it also includes: In response to the operation of writing data into the database, real-time monitoring of the data in the database is triggered, so that when changes to the data are detected, the changed data is sent to the back-end network communication middleware.

3. The method according to claim 2, It is characterized in that Before the operation of writing the data into the database in response to the above, the method further includes: By calling the interface service, the data corresponding to each interface is obtained and written into the database.

4. A data push device, It is characterized in that The back-end network communication middleware includes parsing module, processing module and push module, among which: The parsing module is used to receive the change data, parse the attribute information of the change data, and obtain the change protocol; wherein, by calling the interface service, the data corresponding to each interface is obtained; different change protocols indicate different operations on the data; A processing module, used to perform business processing on the changed data using the business processing logic corresponding to the changed protocol to obtain a processing result; wherein, performing business processing on the changed data using the business processing logic corresponding to the changed protocol includes: sending the changed data and the business processing logic to a local lock-free queue; the local lock-free queue performs business processing on the changed data using the business processing logic, obtains a processing result and sends it to the back-end communication middleware; The push module is used to determine the client corresponding to the changed protocol and push the processing result to the client.

5. The device according to claim 4, It is characterized in that Also includes a monitoring module for: In response to the operation of writing data into the database, real-time monitoring of the data in the database is triggered, so that when changes to the data are detected, the changed data is sent to the back-end network communication middleware.

6. The device according to claim 5, It is characterized in that The monitoring module is used to obtain data corresponding to each interface by calling the interface service and write the data into the database.

7. An electronic device, It is characterized in that include: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 3.

8. A computer readable medium having a computer program stored thereon, It is characterized in that When the program is executed by a processor, the method according to any one of claims 1 to 3 is implemented.

Images