Efficient large-scale real-time data pushing system and method

Abstract

The invention relates to the technical field of data pushing, and discloses an efficient large-scale real-time data pushing system which comprises a system module assembly, and the system module assembly comprises an intelligent load balancing module, a distributed cache module, a pushing strategy making module, a multi-protocol adapter, a self-adaptive compression unit and a safety management assembly. The intelligent load balancing module dynamically monitors the load condition of each node, automatically adjusts a task allocation strategy, and avoids single-point overload; the invention further provides an efficient large-scale real-time data pushing method which comprises the following steps: S1, request sending: a user initiates a connection request through a client application program; s2, identity authentication: receiving a client request through a front-end layer, and performing authentication; and S3, establishing connection. According to the invention, by introducing multiple technical innovations such as intelligent load balancing, distributed caching and hierarchical pushing strategies, real-time data pushing for massive users is realized, and the system performance and the user experience are remarkably improved.

Description

Technical Field

[0001] The present invention relates to the technical field of data push, and specifically to an efficient large-scale real-time data push system and method. Background Art

[0002] With the development of Internet applications, such as the rise of social media, online transactions, Internet of Things (IoT), etc., the demand for real-time data is increasing day by day. Traditional data push mechanisms (such as long polling, short polling) are inefficient and have high latency, making it difficult to meet the requirements of modern applications for low latency and high concurrency.

[0003] Although existing protocols such as WebSocket and MQTT have improved real-time performance, they still have performance bottlenecks when facing a large number of users. Especially during peak periods, they may cause server overload and service interruption. Therefore, a new solution is needed to optimize the large-scale real-time data push process to ensure high throughput, low latency, and stability. Summary of the Invention

[0004] (1) Technical Problems to be Solved

[0005] Aiming at the deficiencies of the prior art, the present invention provides an efficient large-scale real-time data push system and method, mainly to solve the problem that although existing protocols such as WebSocket and MQTT have improved real-time performance, they still have performance bottlenecks when facing a large number of users. Especially during peak periods, they may cause server overload and service interruption.

[0006] (2) Technical Solutions

[0007] To achieve the above object, the present invention provides the following technical solutions:

[0008] An efficient large-scale real-time data push system includes a system module assembly. The system module assembly includes an intelligent load balancing module, a distributed cache module, a push strategy formulation module, a multi-protocol adapter, an adaptive compression unit, and a security management component. The intelligent load balancing module dynamically monitors the load conditions of each node, automatically adjusts the task allocation strategy to avoid single-point overload. The distributed cache module uses distributed cache technology to store hot data and reduce the database query pressure. The push strategy formulation module implements differential push frequencies and contents according to the user's subscription level and network conditions. The multi-protocol adapter is compatible with multiple communication protocols to meet the requirements of different devices and application scenarios. The security management component provides user access authentication, permission management, and data encryption measures to ensure the security and integrity of data transmission.

[0009] Furthermore, the intelligent load balancing module includes a dynamic load detection sub-module and a task reallocation sub-module, which monitors the workloads of all nodes in the system in real time and adjusts the task distribution strategy accordingly.

[0010] Based on the above solution, the distributed cache module includes a memory cache layer and a disk cache layer to ensure the fast reading of frequently accessed data and the reliable storage of large-capacity data.

[0011] As a further solution of the present invention, the push strategy formulation module includes a user behavior analysis engine, which analyzes the historical behavior patterns of users to provide personalized push content and services for each user.

[0012] The present invention also proposes an efficient large-scale real-time data push method, including the following steps:

[0013] S1: Send a request. The user initiates a connection request through the client application.

[0014] S2: Authentication. Receive the client request through the front-end layer and perform verification.

[0015] S3: Establish a connection. After passing the authentication of the front-end layer, forward the request to the back-end service to establish a stable connection.

[0016] S4: Prepare for pushing. The push service layer listens for new data notifications from the message middleware. Once eligible data is generated, immediately prepare for pushing.

[0017] S5: Push strategy. Determine the optimal push path and format according to the pre-configured push strategy.

[0018] S6: Push data. Push the data and compress the data before transmission to reduce the volume and accelerate the transmission.

[0019] S7: Data parsing. After the client receives the pushed data packet, parse it and display it to the user or perform corresponding actions.

[0020] As a further solution of the present invention, the S2 includes a verification module, which verifies illegal inputs and illegal request information.

[0021] Furthermore, the condition in S4 is a specified user ID message or a specified user group message. The S5 includes a business logic layer for processing core business logic and a data persistence layer for managing database operations to ensure data consistency and persistence.

[0022] On the basis of the foregoing solution, encryption information is added during the transmission in S6 to protect privacy. The encryption information includes mobile phone numbers, ID card numbers, and bank card numbers. S6 includes a message middleware and a push service layer. The message middleware serves as a bridge between data production and consumption, supporting asynchronous communication and decoupling. The push service layer: pushes updated data to the target client according to the set rules.

[0023] (III) Beneficial Effects

[0024] Compared with the prior art, the present invention provides an efficient large-scale real-time data push system and method, having the following beneficial effects:

[0025] 1. By introducing a number of technological innovations such as intelligent load balancing, distributed caching, and hierarchical push strategies, the present invention realizes real-time data push for a large number of users, significantly improving the performance of the system and the user experience.

[0026] 2. By adopting the distributed caching technology to store hot data, the present invention reduces the pressure of database queries, speeds up the response speed, and implements different push frequencies and contents according to the subscription level and network status of users, enhancing the user experience.

[0027] 3. The present invention is compatible with multiple communication protocols to meet the requirements of different devices and application scenarios, and provides user access authentication, permission management, and data encryption measures to ensure the security and integrity of data transmission. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 It is a schematic diagram of the module structure of the efficient large-scale real-time data push system proposed by the present invention.

[0029] Figure 2 It is a schematic diagram of the flow structure of the efficient large-scale real-time data push method proposed by the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0030] The following will clearly and completely describe the technical solutions in the embodiments of the present invention with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all of the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0031] Embodiment 1

[0032] Refer to Figure 1-2, An efficient large-scale real-time data push system, including a system module assembly. The system module assembly includes an intelligent load balancing module, a distributed cache module, a push policy formulation module, a multi-protocol adapter, an adaptive compression unit, and a security management component. The intelligent load balancing module dynamically monitors the load conditions of each node, automatically adjusts the task allocation strategy, and avoids single-point overload. The distributed cache module uses distributed cache technology to store hot data, reduces the database query pressure, and speeds up the response speed. The push policy formulation module implements different push frequencies and contents according to the user's subscription level and network conditions to enhance the user experience. The multi-protocol adapter is compatible with multiple communication protocols to meet the requirements of different devices and application scenarios. The security management component provides user access authentication, permission management, and data encryption measures to ensure the security and integrity of data transmission.

[0033] An efficient large-scale real-time data push method, including the following steps:

[0034] S1: Send a request. The user initiates a connection request through the client application.

[0035] S2: Authentication. Receive the client request through the front-end layer and perform verification. S2 includes a verification module that verifies illegal inputs and illegal request information.

[0036] S3: Establish a connection. After passing the authentication of the front-end layer, forward the request to the back-end service to establish a stable connection.

[0037] S4: Prepare for pushing. The push service layer listens for new data notifications from the message middleware. Once eligible data is generated, immediately prepare for pushing. The conditions in S4 are specified user ID messages and specified user group messages.

[0038] S5: Push policy. Determine the optimal push path and format according to the pre-configured push policy. S5 includes a business logic layer for processing core business logic and a data persistence layer for managing database operations to ensure data consistency and persistence.

[0039] S6: Push data. Push the data and compress the data before transmission to reduce the volume and accelerate the transmission. At the same time, encryption information will also be added to protect privacy. The encryption information in S6 includes mobile phone numbers, ID card numbers, and bank card numbers.

[0040] S7: Data parsing. After the client receives the pushed data packet, parse it and display it to the user or perform corresponding actions.

[0041] Embodiment 2

[0042] Refer to Figure 1-2, An efficient large-scale real-time data push system, including a system module assembly. The system module assembly includes an intelligent load balancing module, a distributed cache module, a push policy formulation module, a multi-protocol adapter, an adaptive compression unit, and a security management component. The intelligent load balancing module dynamically monitors the load conditions of each node, automatically adjusts the task allocation strategy, and avoids single-point overload. The distributed cache module uses distributed cache technology to store hot data, reduces the database query pressure, and speeds up the response speed. The push policy formulation module implements different push frequencies and contents according to the user's subscription level and network status, enhancing the user experience. The multi-protocol adapter is compatible with multiple communication protocols (such as WebSocket, MQTT) to meet the requirements of different devices and application scenarios. The security management component provides user access authentication, permission management, and data encryption measures to ensure the security and integrity of data transmission.

[0043] The intelligent load balancing module includes a dynamic load detection sub-module and a task reallocation sub-module, which are used to monitor the workload of all nodes in the system in real time and adjust the task distribution strategy accordingly. The distributed cache module includes a memory cache layer and a disk cache layer to ensure the fast reading of frequently accessed data and the reliable storage of large-capacity data. The push policy formulation module includes a user behavior analysis engine, which is used to analyze the historical behavior patterns of users, so as to provide personalized push content and services for each user.

[0044] An efficient large-scale real-time data push method, including the following steps:

[0045] S1: Send a request. The user initiates a connection request through the client application.

[0046] S2: Authentication. Receive the client request through the front-end layer and perform verification. S2 includes a verification module, which verifies illegal inputs and illegal request information.

[0047] S3: Establish a connection. After passing the authentication of the front-end layer, forward the request to the back-end service to establish a stable connection.

[0048] S4: Prepare for pushing. The push service layer listens for new data notifications from the message middleware. Once eligible data is generated, immediately prepare for pushing. The conditions in S4 are specified user ID messages and specified user group messages.

[0049] S5: Push policy. Determine the optimal push path and format according to the pre-configured push policy (such as factors like user subscription level, geographical location, network status, etc.). S5 includes a business logic layer for processing core business logic and a data persistence layer for managing database operations to ensure data consistency and persistence.

[0050] S6: Push data, push the data and compress the data before transmission to reduce the volume and accelerate the transmission. At the same time, encryption information will be added to protect privacy. The encryption information in S6 includes mobile phone numbers, ID card numbers, and bank card numbers. And S6 includes a message middleware and a push service layer. The message middleware serves as a bridge between data production and consumption, supporting asynchronous communication and decoupling. The push service layer: Push the updated data to the target client according to the set rules;

[0051] S7: Data parsing. After the client receives the pushed data packet, it parses and displays it to the user or performs corresponding actions.

[0052] In the description of this article, it should be noted that relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the term "including", "comprising" or any other variant thereof is intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements not only includes those elements, but also includes other elements not expressly listed, or also includes elements inherent to such process, method, article or device.

Claims

1. An efficient, large-scale, real-time data push system, including a system module assembly, characterized in that: The system module assembly includes an intelligent load balancing module, a distributed cache module, a push strategy formulation module, a multi-protocol adapter, an adaptive compression unit and a security management component. The intelligent load balancing module dynamically monitors the load conditions of each node and automatically adjusts the task allocation strategy to avoid single-point overload. The distributed cache module uses distributed cache technology to store hot data and reduce database query pressure. The push strategy formulation module implements differentiated push frequency and content according to the user's subscription level and network conditions. The multi-protocol adapter is compatible with multiple communication protocols to meet the needs of different devices and application scenarios. The security management component provides user access authentication, authority management and data encryption measures to ensure the security and integrity of data transmission.

2. The efficient, large-scale, real-time data push system according to claim 1, characterized in that: The intelligent load balancing module includes a dynamic load detection submodule and a task reallocation submodule, which monitors the workload of all nodes in the system in real time and adjusts the task distribution strategy accordingly.

3. The efficient, large-scale, real-time data push system according to claim 1, characterized in that: The distributed cache module includes a memory cache layer and a disk cache layer to ensure fast reading of high-frequency access data and reliable storage of large-capacity data.

4. The efficient, large-scale, real-time data push system according to claim 1, characterized in that: The push strategy formulation module includes a user behavior analysis engine that analyzes the user's historical behavior patterns to provide personalized push content and services for each user.

5. An efficient, large-scale, real-time data push method, characterized in that: The following steps are involved: S1: Send request. The user initiates a connection request through the client application. S2: Authentication, receiving client requests through the front-end layer and performing verification; S3: Establishes a connection. After passing the identity authentication of the front-end layer, it forwards the request to the back-end service to establish a stable connection. S4: Prepare to push. The push service layer listens to new data notifications from the message middleware. Once data that meets the conditions is generated, it is immediately prepared to push. S5: Push strategy: Determine the optimal push path and format based on the pre-configured push strategy; S6: Push data, push data and compress the data before transmission to reduce the volume and speed up the transmission; S7: Data parsing: After receiving the pushed data packet, the client parses it and displays it to the user or performs corresponding actions.

6. The efficient large-scale real-time data push method according to claim 5, characterized in that: The S2 includes a verification module, which verifies illegal input and illegal request information.

7. The efficient, large-scale, real-time data push method according to claim 5, characterized in that: The conditions in S4 are a specified user ID message and a specified user group message, and S5 includes a business logic layer for processing core business logic and a data persistence layer for managing database operations to ensure data consistency and persistence.

8. The efficient, large-scale, real-time data push method according to claim 5, characterized in that: The S6 adds encrypted information during transmission to protect privacy. The encrypted information includes mobile phone number, ID card number, and bank card number. The S6 includes a message middleware and a push service layer. The message middleware serves as a bridge between data production and consumption, supports asynchronous communication and decoupling, and the push service layer pushes updated data to the target client according to the set rules.

Images