Data consumption method, device, server and readable storage medium

Abstract

The present application provides a data consumption method, device, server and readable storage medium, and relates to the field of computer technology. The method includes: according to the user identification of the target client and the stored user configuration information, determining the identification of the target message queue to be monitored and the first target preset consumption speed, and monitoring the target message queue, the user configuration information includes the first preset consumption speed and the user identification, and the identification of the message queue is determined according to the corresponding user identification; when it is monitored that there is data to be consumed in the target message queue, consuming the data in the target consumption queue at a speed not greater than the first target preset consumption speed, and sending the consumed data to the target client through the websocket link, and the data in the target message queue that exceeds the first target preset consumption speed is discarded. In this way, stable real-time consumption of a large amount of data can be achieved, and it can be distributed to the target user.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a data consumption method, device, server and readable storage medium. Background Art

[0002] At present, the client consumes server data generally through back-end interface query. In scenarios requiring real-time performance, real-time consumption is achieved through interface polling to meet some high real-time requirements. However, in some scenarios, the real-time consumption of big data on the client is indeed a challenge. How to ensure the orderly and stable consumption of large amounts of data and distribute it to target users is a technical problem that technicians in this field need to solve urgently. Summary of the invention

[0003] The embodiments of the present application provide a data consumption method, device, server and readable storage medium, which can ensure the orderly, stable and real-time consumption of large amounts of data and distribute it to target users.

[0004] The embodiments of the present application can be implemented as follows:

[0005] In a first aspect, an embodiment of the present application provides a data consumption method, which is applied to a server connected to a client for communication, wherein at least one user configuration information is stored in the server, and the user configuration information includes a first preset consumption speed and a user identifier, and the method includes:

[0006] According to the user identifier of the target client and the stored user configuration information, determine the identifier of the target message queue to be monitored and the first target preset consumption speed, and monitor the target message queue, wherein the identifier of the message queue is determined according to the corresponding user identifier;

[0007] In the case of monitoring that there is data to be consumed in the target message queue, the data in the target consumption queue is consumed at a speed no greater than the first target preset consumption speed, and the consumed data is sent to the target client via a websocket link, wherein the data in the target message queue that exceeds the first target preset consumption speed is discarded.

[0008] In a second aspect, an embodiment of the present application provides a data consumption device, which is applied to a server connected to a client for communication, wherein at least one user configuration information is stored in the server, wherein the user configuration information includes a first preset consumption speed and a user identifier, and the device includes:

[0009] A first monitoring module, used to determine the identifier of a target message queue to be monitored and a first target preset consumption speed according to a user identifier of a target client and the stored user configuration information, and monitor the target message queue, wherein the identifier of the message queue is determined according to the corresponding user identifier;

[0010] A sending module is used to consume the data in the target consumption queue at a speed not greater than the first target preset consumption speed when it is monitored that there is data to be consumed in the target message queue, and send the consumed data to the target client through a websocket link, wherein the data in the target message queue that exceeds the first target preset consumption speed is discarded.

[0011] In a third aspect, an embodiment of the present application provides a server, including a processor and a memory, wherein the memory stores machine executable instructions that can be executed by the processor, and the processor can execute the machine executable instructions to implement the data consumption method described in the aforementioned embodiment.

[0012] In a fourth aspect, an embodiment of the present application provides a readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the data consumption method as described in the foregoing embodiment is implemented.

[0013] The data consumption method, device, server and readable storage medium provided by the embodiment of the present application determine the identifier of the target message queue to be monitored and the first target preset consumption speed according to the user identifier of the target client and the first preset consumption speed and user identifier included in the stored configuration information of each user, and monitor the target message queue, wherein the identifier of the message queue is determined according to the corresponding user identifier; when it is monitored that there is data to be consumed in the target message queue, the data in the target consumption queue is consumed at a speed not greater than the first target preset consumption speed, and the consumed data is sent to the target client through a websocket link, and the data in the target message queue that exceeds the first target preset consumption speed is discarded. In this way, it is convenient to ensure the orderly, stable and real-time consumption of a large amount of data and distribute it to the target user. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0015] Figure 1 A block diagram of a server provided in an embodiment of the present application;

[0016] Figure 2 One of the flow charts of the data consumption method provided in the embodiment of the present application;

[0017] Figure 3 The second flowchart of the data consumption method provided in the embodiment of the present application;

[0018] Figure 4 The third flowchart of the data consumption method provided in the embodiment of the present application;

[0019] Figure 5 A fourth flowchart of the data consumption method provided in an embodiment of the present application;

[0020] Figure 6 The fifth flowchart of the data consumption method provided in the embodiment of the present application;

[0021] Figure 7 Flow chart of the data consumption method provided in the embodiment of the present application is as follows;

[0022] Figure 8 A schematic diagram illustrating an example of a data consumption method provided in an embodiment of the present application;

[0023] Fig. 9 One of the block diagrams of the data consumption device provided in the embodiment of the present application;

[0024] Fig.10 The second block diagram of the data consumption device provided in the embodiment of the present application;

[0025] Fig.11 The third block diagram of the data consumption device provided in the embodiment of the present application.

[0026] Icon: 100 - server; 110 - memory; 120 - processor; 130 - communication unit; 200 - data consumption device; 201 - configuration module; 210 - second monitoring module; 220 - processing module; 230 - first monitoring module; 240 - sending module. DETAILED DESCRIPTION

[0027] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. The components of the embodiments of the present application described and shown in the drawings here can be arranged and designed in various different configurations.

[0028] Therefore, the following detailed description of the embodiments of the present application provided in the accompanying drawings is not intended to limit the scope of the present application for which protection is sought, but merely represents selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present application.

[0029] It should be noted that relational terms such as "first" and "second" are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0030] Currently, clients regularly update data on pages through interface polling. Polling refers to the browser sending HTTP requests to the server at regular intervals to obtain data. However, this method requires constant establishment of HTTP connections, which seriously wastes server and client resources. It is not real-time enough and has no strategy for large data volume scenarios, which means that it cannot guarantee the orderly and stable consumption of large amounts of data and its distribution to target users.

[0031] In order to alleviate the above problems, the embodiments of the present application provide a data consumption method, device, server and readable storage medium, which are convenient for ensuring the orderly, stable and real-time consumption of large amounts of data and distributing it to target users.

[0032] In conjunction with the accompanying drawings, some embodiments of the present application are described in detail below. In the absence of conflict, the following embodiments and features in the embodiments can be combined with each other.

[0033] Please refer to Figure 1 , Figure 1 A block diagram of a server 100 provided in an embodiment of the present application. The server 100 may be a single server or a cluster server, etc. The server 100 may include a memory 110, a processor 120, and a communication unit 130. The memory 110, the processor 120, and the communication unit 130 are electrically connected to each other directly or indirectly to achieve data transmission or interaction. For example, these components may be electrically connected to each other via one or more communication buses or signal lines.

[0034] The memory 110 is used to store programs or data. The memory 110 may be, but is not limited to, a random access memory (RAM), a read only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), etc.

[0035] The processor 120 is used to read / write data or programs stored in the memory 110 and execute corresponding functions. For example, the memory 110 stores a data consumption device 200, and the data consumption device 200 includes at least one software function module that can be stored in the memory 110 in the form of software or firmware. The processor 120 executes various functional applications and data processing by running software programs and modules stored in the memory 110, such as the data consumption device 200 in the embodiment of the present application, that is, realizing the data consumption method in the embodiment of the present application.

[0036] The communication unit 130 is used to establish a communication connection between the server 100 and other communication terminals through a network, and to send and receive data through the network.

[0037] It should be understood that Figure 1 The structure shown is only a schematic diagram of the structure of the server 100. The server 100 may also include Figure 1 More or fewer components as shown, or with Figure 1 Different configurations shown. Figure 1 Each component shown in the figure can be implemented by hardware, software or a combination thereof.

[0038] Please refer to Figure 2 , Figure 2 One of the flow diagrams of the data consumption method provided in the embodiment of the present application. The method can be applied to the server 100, in which at least one user configuration information is stored, and each user configuration information may include a first preset consumption speed and a user identifier. The user client can be a unique identifier of the user corresponding to the client, which can be set in combination with actual needs. The specific process of the data consumption method is described in detail below. The method may include steps S150 and S160.

[0039] Step S150: Determine the identifier of the target message queue to be monitored and the first target preset consumption speed according to the user identifier of the target client and the stored user configuration information, and monitor the target message queue.

[0040] In this embodiment, the target client is a client to which data needs to be sent, and can be specifically set in combination with actual needs, such as an online client. The target user configuration information corresponding to the target client can be determined based on the user identifier of the target client and the user identifier included in the stored user configuration information. Then, the first preset consumption speed included in the target user configuration information is used as the first target preset consumption speed corresponding to the target client. The identifier of the target message queue that needs to be monitored is also determined based on the user identifier of the target client. When the identifier of the target message queue corresponding to the target client is determined, the target message queue corresponding to the identifier of the target message queue is monitored.

[0041] Step S160, when it is monitored that there is data to be consumed in the target message queue, consume the data in the target consumption queue at a speed not greater than the first target preset consumption speed, and send the consumed data to the target client through the websocket link.

[0042] When data is monitored from the target message queue, the data in the target message queue is consumed at a first consumption speed, and the consumed data is sent to the target client via a websocket link. The websocket pushes the data to the target client via the network, so that the target client obtains the data in real time.

[0043] The first consumption speed is less than or equal to the first target preset consumption speed corresponding to the target client, and the specific value of the first consumption speed is determined according to the actual situation of the data in the target message queue. The data in the target message queue that exceeds the first target preset consumption speed is discarded.

[0044] When consuming the data in the target message queue, the specific value of the first consumption speed can be obtained by statistics, and when the value is about to reach the first target preset consumption speed, the data is discarded. That is, if the amount of data in the target message queue exceeds the first target preset consumption speed, a part of the data is discarded so that the actual consumption speed is not greater than the first target preset consumption speed.

[0045] In this way, data is sent to the target client via websocket, and there is no need to continuously establish http connections, thereby saving server and client resources and ensuring the real-time nature of the data. In addition, data is consumed based on the preset consumption speed of the first target, and data is discarded when the actual consumption speed is about to exceed the preset consumption speed of the first target, thereby ensuring stable real-time consumption of large amounts of data and distributing it to target users.

[0046] Optionally, in this embodiment, an online user list is maintained in the server. When the client is online, the client can actively send the local user ID to the server so that the server maintains the online user list. The online user list may include the user ID of the online user, and the user configuration information includes the user name and the consumption data configuration identifier. A user configuration information may include a consumption data configuration identifier or multiple consumption data configuration identifiers. Among them, the user name can be characters that are easy for staff to read and understand, such as xx enterprise; the user identifier is an identifier that is easy for machine recognition corresponding to the user name, for example, the user identifier of xxx enterprise is 500xxx. A consumption data configuration identifier is used to represent an identifier of a consumption data configuration information, which can be manually entered by the staff as the name of a certain consumption data configuration information. The consumption data configuration information can be a configuration of consumption data.

[0047] As a possible implementation, the identifier of the target message queue may be determined in the following manner: wherein the identifier of the message queue may be generated based on the user name and the consumption data configuration identifier, which facilitates determination of the target message queue and manual viewing.

[0048] First, a user identifier of at least one target client is determined from the online user list. Optionally, one or more online clients can be determined as the target client according to actual needs, or all online clients determined from the online user list can be used as the target client.

[0049] When the target client is determined, the target user configuration information corresponding to each target client can be determined based on the user identifier of the target client and the users in the stored user configuration information; then, the identifier of the target message queue corresponding to the target client can be determined based on the user name and consumption data configuration identifier included in the target user configuration information, thereby determining the target message queue corresponding to the target client.

[0050] The user configuration information may also include a user name. When the identifier of the target message queue corresponding to the target client is determined, the target message queue may be monitored, and when data is monitored, the data is sent to the target client using the user name in the user configuration information corresponding to the target client. The target client monitors the corresponding event name to obtain data in real time.

[0051] Optionally, in a possible implementation, the user configuration information may further include whether there is an associated user, and if there is an associated user, the user configuration information may further include an associated user identifier, wherein the associated user identifier may be determined based on the associated user name input by the staff and other user configuration information.

[0052] The identifier of each online user can be obtained according to the online user list, and then the user group corresponding to each online user can be determined according to the user identifier of each online user and the user identifier and the associated user identifier in the stored user configuration information, wherein the user group corresponding to an online user includes the online user and the associated user of the online user determined based on the associated user identifier, and the associated user of the online user is the online user. Optionally, the user group corresponding to an online user can be all online users, or some online users and some offline users, which can be set according to actual needs.

[0053] When the data in the target message queue is monitored, the data can be sent to each online client in the target user group corresponding to the target client with the configured event name. That is, according to the event name corresponding to each online user in the target user group corresponding to the target client, the data is sent to each online client in the target user group through the websocket link. Among them, the target user group includes the target client. In this way, the clients in the target user group can obtain data in real time by monitoring the corresponding event name.

[0054] Optionally, only when the user configuration information of a user includes an associated user identifier, the user has an associated user, and the associated user is the user corresponding to the associated user identifier. If the user configuration information of a user does not include an associated user identifier, it means that the user has no associated user. For example, if the user configuration information of user A includes the associated user identifier B, and the user configuration information of user B does not include the associated user identifier A, it means that the associated user of user A is B, and user B has no associated user.

[0055] That is, it is equivalent to saying that the associated user identifier is the parent identifier, the parent can consume the data associated with the child, and the child cannot consume the data associated with the parent. In this way, the client that needs to obtain the corresponding data can obtain the corresponding data.

[0056] As a possible implementation, the data in the message queue may include a user ID, and the data may be sent to a client having the user ID and an associated client of the client (ie, a client of an associated user). In this way, the data may be sent to a user with authority.

[0057] Optionally, a heartbeat detection and reconnection mechanism needs to be implemented between the server and the client so as to send data to the corresponding client.

[0058] Please refer to Figure 3 , Figure 3 The second flowchart of the data consumption method provided in the embodiment of the present application is shown in FIG.

[0059] Step S120, determining a target message queue topic to be monitored, and monitoring the target message queue topic.

[0060] The message queue topic stores the data produced by the producer. The target message queue topic can be specified by the staff or automatically determined by the server according to the corresponding rules.

[0061] Optionally, the server maintains an online user list, which includes user identifiers of online users. The server also stores at least one consumption data configuration information, each consumption data configuration information includes a consumption data configuration identifier and a message queue topic identifier, and the user configuration information also includes a consumption data configuration identifier.

[0062] As an optional implementation method, the user identifier of each online user can be obtained according to the online user list. For each online user, the user configuration information corresponding to the online user is obtained according to the user identifier of the online user and the stored user configuration information. The consumption data configuration identifier in the user configuration information corresponding to the online user can be used as the consumption data configuration identifier corresponding to the online user. The consumption data configuration information corresponding to the consumption data configuration identifier corresponding to the online user is obtained, and the message queue topic identifier in the consumption data configuration information is used as the target consumption data configuration identifier corresponding to the online user. Then, the target message queue topic (i.e., message queue topic) identifier corresponding to the online user can be obtained from the target consumption data configuration information corresponding to the target consumption data configuration identifier corresponding to the online user. Wherein, topic represents the name of each topic in the message queue system.

[0063] When the target message queue topic identifier corresponding to each online user is obtained, the message queue topic corresponding to each target message queue topic identifier can be used as the target message queue topic to be monitored. That is, the target message queue topic identifier is the identifier of the target message queue topic. In this way, the message queue topic where the data associated with each online user is located can be monitored so that the data can be sent to each online user later.

[0064] Step S130, when it is monitored that there is data to be consumed in the target message queue topic, the data is obtained from the target message queue topic, and the obtained data is pushed to the corresponding message queue.

[0065] In the process of monitoring the target message queue topic, if data is monitored, the data is obtained from the target message queue topic and pushed to the corresponding message queue. The identifier of the corresponding message queue can be determined according to the user identifier and user configuration information corresponding to the target message queue topic.

[0066] Optionally, the consumption data configuration information may also include the message queue type, whether to persist, the table name corresponding to persistence, and the logic of processing and cleaning. The message queue type is the queue type of the message queue topic corresponding to the message queue topic identifier included in the consumption data configuration information, such as kafka. The table name corresponding to persistence needs to be applied for and initialized in advance.

[0067] You can listen to the target message queue topic according to the message queue type in the consumption data configuration information corresponding to the target message queue topic. When listening to the target message queue topic, when data is listened to, the data can be processed and cleaned according to the processing and cleaning logic in the consumption data configuration information corresponding to the target message queue topic. If the consumption data configuration information corresponding to the target message queue topic includes persistence, the data obtained after processing and cleaning will be asynchronously stored in the database, and finally the obtained data will be pushed to the corresponding message queue.

[0068] Please refer to Figure 4 , Figure 4 The third flowchart of the data consumption method provided in the embodiment of the present application is shown in FIG.

[0069] Step S141, counting the second consumption speed of the data of the target message queue topic.

[0070] Step S142, determining whether to suspend or resume consumption of data on the target message queue topic according to the second preset consumption speed and the second consumption speed, and executing a corresponding processing method.

[0071] The second consumption speed of the data of each target message queue topic can be counted. Then, for each target message queue topic, the second consumption speed of the target message queue topic is compared with the second preset consumption speed corresponding to the target message queue topic. Among them, the second preset consumption speed corresponding to the target message queue topic can be set according to the estimated data processing capacity of the machine. The second preset consumption speeds corresponding to different message queue topics can be the same or different, and can be set in combination with actual needs.

[0072] If the second consumption speed of the target message queue topic reaches the second preset consumption speed corresponding to the target message queue topic, the consumption of the data of the target message queue topic is suspended to avoid the large amount of data from dragging down the machine. If the data of the target message queue topic is being consumed and the second consumption speed of the target message queue topic does not reach the second preset consumption speed, the data of the target message queue topic continues to be consumed. If the consumption of the data of the target message queue topic is currently suspended and the second consumption speed of the target message queue topic does not reach the second preset consumption speed, the consumption of the data of the target message queue topic can be resumed. In this way, the consumption of the topic can be subject to flow restrictions to avoid large amounts of data from dragging down the machine.

[0073] Please refer to Figure 5 , Figure 5 This is a fourth flow chart of the data consumption method provided in the embodiment of the present application. In this embodiment, the method may further include step S145 and step S146.

[0074] Step S145, counting the second consumption speed and generation speed of the data of the target message queue topic.

[0075] Step S146: When the second consumption speed continues to be lower than the generation speed for a period longer than a preset period, a prompt message is generated to prompt that the consumption capacity needs to be improved.

[0076] In this embodiment, for each target message queue topic, the second consumption speed and generation speed of the data of the target message queue topic can be obtained by statistics, and the duration of the second consumption speed being continuously less than the generation speed can be obtained by statistics. If the duration is greater than the preset duration, it indicates that the consumption capacity of the machine is insufficient, and a prompt message can be generated and sent to the corresponding display terminal for display, so as to prompt the user to consider increasing the consumption capacity by adding more machines.

[0077] Please refer to Figure 6 , Figure 6 The fifth flowchart of the data consumption method provided in the embodiment of the present application is as follows. In this embodiment, the method may further include step 170.

[0078] Step S170, when the online users change, the online user list is updated accordingly, and according to the updated online user list, the monitoring of the corresponding message queue topic is canceled or established, and the user group corresponding to each online user is determined.

[0079] In this embodiment, the connection and disconnection of the websocket can be monitored, and the online user list can be updated according to the changes of the monitored online users. In the case of changes in online users, the monitoring of the corresponding message queue topic can be canceled or established according to the updated online user list, and the user group corresponding to each online user can be determined. In this way, the data that the online user is concerned about can be sent to the corresponding online user.

[0080] For example, at time 1, user 1 is online and user 2 is offline, then the message queue topic that needs to be monitored at this time is the message queue topic corresponding to user 1, and there is no need to monitor the message queue topic corresponding to user 2; if user 2 is also online at time 2, then the message queue topics that need to be monitored at this time are the message queue topics corresponding to user 1 and the message queue topics corresponding to user 2. In this way, both online users 1 and 2 can receive the data they are concerned about.

[0081] Please refer again Figure 6 In this embodiment, the method may further include step S180.

[0082] Step S180, monitoring changes in the user configuration information and the consumption data configuration information, and canceling or establishing monitoring of the corresponding message queue topic according to the changes, and canceling or establishing monitoring of the corresponding message queue according to the changes.

[0083] In this embodiment, the changes of the user configuration information and the consumption data configuration information can also be dynamically monitored, and the monitoring of the corresponding message queue topic and the corresponding message queue can be canceled or established according to the changes, so that the monitoring object corresponds to the changed user configuration information and the consumption data configuration information. In this way, it can be ensured that the data that the user is concerned about is sent to the corresponding user.

[0084] Please refer to Figure 7 , Figure 7 This is a sixth flow chart of the data consumption method provided in the embodiment of the present application. In this embodiment, the method may further include step S110.

[0085] Step S110, receiving a configuration operation input by a user through a configuration interface, and obtaining the user configuration information and consumption data configuration information according to the configuration operation.

[0086] In this embodiment, the staff can input the configuration operation to the server in advance through the configuration interface. The server can obtain the user configuration information and consumption data configuration information according to the received configuration operation. The user configuration information and consumption data configuration information can also be persisted in the database. For example, persisted in the redis database, which is a high-performance memory storage database. When the user configuration information and consumption data configuration information are needed, the user configuration information and consumption data configuration information can be pulled from the database.

[0087] It is understandable that after the user configuration information and the consumption data configuration information are generated for the first time, the staff can also perform operations such as adding, deleting, and modifying the user configuration information and the consumption data configuration information through the configuration interface.

[0088] Please refer to Figure 8 , Figure 8 The following is a schematic diagram of an example of a data consumption method provided in an embodiment of the present application. The server may include a data processing layer, a message center, and a configuration interface. The data processing layer is used to connect to various data sources, clean and persist the data, and push it to the message center. The message center is used to group users, consume the data pushed by the data processing layer, and push the data to the corresponding groups. The configuration interface is used to enable staff to configure the configurable items of the entire process. The data consumption method is described below by way of example.

[0089] The message center and data processing layer can be configured through the configuration interface.

[0090] First, for the data processing layer, new consumption data configuration information can be created: name (i.e., consumption data configuration identifier), message queue type, topic (i.e., message queue topic identifier), whether to persist, the corresponding table name for persistence (need to be applied for and initialized in advance), and processing and cleaning logic. Secondly, configure the accessible users (i.e., client users who can connect to the server with websocket), that is, configure the user configuration information: user name, whether to associate with the parent (equivalent to whether there is an associated user) and the user name of the associated parent (the user identifier of the parent can be obtained according to the user name of the parent), the name of one or more associated consumption data configuration information, the upper limit of the data consumption speed (i.e., the first preset consumption speed, which may have a default value and cannot exceed the upper limit configured by the system), user identifier, and time name. Among them, when the application scenario is for an enterprise, the user identifier can be a unique user identifier commonly used within the enterprise. The above configurations are all persisted in the redis database.

[0091] Message queues are generally used in big data scenarios. The data processing layer has built-in support for several common message queues to consume, process, cleanse, and persist data.

[0092] The data processing layer can first pull the online user list from the redis database, and pull the consumption data configuration information and user configuration information from the redis database, so as to match the user configuration information of the online user, and then determine the message queue topic corresponding to the online user. Then, the message queue topic corresponding to the online user can be monitored for data. After the data is monitored, the data is processed and cleaned according to the processing and cleaning logic in the consumption data configuration information corresponding to the online user. If persistence is required, the processed data is asynchronously stored in the database, and finally the processed data is pushed to the message queue consumed by the message center.

[0093] The message center needs to maintain changes in online users and can notify the data processing layer through the pub / sub mechanism of redis. The data processing layer dynamically cancels or establishes subscriptions to message queue topics based on changes in online users.

[0094] The data processing layer limits the consumption of message queue topics to prevent large amounts of data from bringing down the machine. The data processing capacity of the machine can be estimated, and the consumption traffic of message queue topics can be counted. Based on the above data processing capacity and the traffic statistics, consumption can be suspended and resumed in a timely manner.

[0095] If the consumption speed of data of a message queue topic continues to be lower than the data generation speed for a period longer than a preset period, a corresponding prompt message can be generated to prompt the user to increase the consumption capacity of the message queue topic by adding machines.

[0096] The data of the message queue topic may include a user identifier, and the data including the user identifier may be subsequently sent to a client corresponding to the user identifier.

[0097] The name of the message queue consumed by the message center can be generated based on the user name corresponding to the message queue and the name of the consumption data configuration. The data processing layer can determine the message queue where the data needs to be placed based on this principle. This method makes it easier for the message center to consume the data in the message queue. The message queue consumed by the message center can be a queue in redis.

[0098] For example, the same message queue topic consumed by the data processing layer may contain data corresponding to different users. When the data processing layer pushes data to the queue in redis, it can separate the data of different users to form a row of message queues for easy consumption by the message center.

[0099] The data processing layer can also dynamically monitor changes in the consumption data configuration information and user configuration information, and cancel or establish a subscription to the message queue topic according to the changes.

[0100] The client can establish a websocket link with the message center and pass in the user ID corresponding to the client user. The message center can pull the user configuration information and message data configuration information based on the user ID of the online client user, generate a mapping of associated children, and assign the parent and child to the group corresponding to the child; and determine the message queue to be monitored based on the name and user name in the consumption data configuration information corresponding to the online user. When the data is monitored, the data is consumed and sent to the corresponding group with the event name in the user configuration information corresponding to each online user in the group. The websocket will be pushed to the corresponding client through the network, and the client can obtain data in real time by listening to the corresponding event name. There are several key points:

[0101] Among them, the message center needs to monitor the connection and disconnection of the websocket, maintain online users, and notify changes through the pub / sub mechanism of redis so that the data processing layer can update the message queue topic to be monitored accordingly.

[0102] The message center may also collect statistics on the consumption rate of the message queue corresponding to each online user. If the consumption rate cannot reach the first preset consumption rate in the user configuration information corresponding to the message queue, the data in the message queue needs to be discarded.

[0103] The client and the message center need to implement heartbeat detection and reconnection mechanisms to ensure that data is sent to the corresponding online client.

[0104] The message center can also dynamically monitor changes in user configuration information and message data configuration information, and re-acquire various information corresponding to online users based on the changes, so as to execute corresponding logic and send corresponding data to corresponding online users.

[0105] In this embodiment, the data processing layer unifies the connection of message queues corresponding to various data sources, dynamically cancels and establishes data subscriptions through online user monitoring, and achieves stable and real-time consumption of big data through the flow limiting strategy of the data processing layer and the consumption rate configuration and data discarding strategy of the message center. In addition, compared with polling, by establishing a websocket connection between the server and the client, grouping users, and sending data to the corresponding groups through the websocket connection, the resource consumption of the server can be greatly reduced, and the real-time performance is also guaranteed.

[0106] In order to execute the corresponding steps in the above embodiments and various possible methods, a data consumption device 200 is provided below. Optionally, the data consumption device 200 may adopt the above Figure 1 The device structure of the server 100 is shown in FIG. Fig. 9 , Fig. 9 This is one of the block diagrams of the data consumption device 200 provided in the embodiment of the present application. It should be noted that the basic principle and technical effect of the data consumption device 200 provided in this embodiment are the same as those of the above-mentioned embodiments. For the sake of brief description, for parts not mentioned in this embodiment, reference can be made to the corresponding contents in the above-mentioned embodiments. The data consumption device 200 can be applied to a server 100 that is communicatively connected to a client, and at least one user configuration information is stored in the server 100, and the user configuration information includes a first preset consumption speed and a user identifier. The data consumption device 200 may include: a first monitoring module 230 and a sending module 240.

[0107] The first monitoring module 230 is used to determine the identifier of the target message queue to be monitored and the first target preset consumption speed according to the user identifier of the target client and the stored user configuration information, and monitor the target message queue. The identifier of the message queue is determined according to the corresponding user identifier.

[0108] The sending module 240 is used to consume the data in the target consumption queue at a speed not greater than the first target preset consumption speed when monitoring that there is data to be consumed in the target message queue, and send the consumed data to the target client through the websocket link. The data in the target message queue that exceeds the first target preset consumption speed is discarded.

[0109] Optionally, in this embodiment, an online user list is maintained in the server 100, the online user list includes user identifiers of online users, the user configuration information includes user names and consumption data configuration identifiers, and the first monitoring module 230 is specifically used to: determine the user identifier of at least one target client from the online user list; determine the user name and consumption data configuration identifier corresponding to the target client according to the user identifier of the target client and the stored user configuration information; determine the identifier of the target message queue according to the user name and consumption data configuration identifier corresponding to the target client.

[0110] Optionally, in this embodiment, the server 100 maintains an online user list, the online user list includes user identifiers of online users, the user configuration information includes event names, and in the case where the user has associated users, the user configuration information also includes associated user identifiers, and the first monitoring module 230 is further used to: determine the user group corresponding to each online user according to the user identifier of each online user and the user identifier and associated user identifier in the user configuration information, wherein the user group corresponding to an online user includes the online user and the associated users of the online user determined based on the associated user identifier. The sending module 240 is specifically used to: send data to each online client in the target user group corresponding to the target client through a websocket link according to the event name corresponding to each online user in the target user group.

[0111] Please refer to Fig.10 , Fig.10 The second block diagram of the data consumption device 200 provided in the embodiment of the present application. The data consumption device 200 may further include: a second monitoring module 210 and a processing module 220.

[0112] The second monitoring module 210 is used to determine a target message queue topic to be monitored and monitor the target message queue topic.

[0113] The processing module 220 is used to obtain data from the target message queue topic when it is monitored that there is data to be consumed in the target message queue topic, and push the obtained data to the corresponding message queue.

[0114] Optionally, in this embodiment, the server 100 maintains an online user list, the online user list includes user identifiers of online users, the server also stores at least one consumption data configuration information, the consumption data configuration information includes a consumption data configuration identifier and a message queue topic identifier, and the user configuration information also includes a consumption data configuration identifier. The second monitoring module 210 is specifically used to: obtain a target consumption data configuration identifier corresponding to each online user according to the user identifier of each online user and the stored user configuration information; obtain a target message queue topic identifier corresponding to each online user according to the target consumption data configuration information corresponding to the target consumption data configuration identifier, wherein the target message queue topic identifier is the identifier of the target message queue topic.

[0115] Optionally, in this embodiment, the processing module 220 is also used to: count the second consumption speed of the data of the target message queue topic; determine to suspend the consumption of the data of the target message queue topic or resume the consumption of the data of the target message queue topic based on the second preset consumption speed and the second consumption speed, and execute the corresponding processing method.

[0116] Optionally, in this embodiment, the processing module 220 is also used to: count the second consumption speed and generation speed of the data of the target message queue topic; when the second consumption speed continues to be less than the generation speed for a period longer than a preset period, generate a prompt message for prompting the need to improve consumption capacity.

[0117] Optionally, in this embodiment, the first monitoring module 230 may also be used to: update the online user list accordingly when the online user changes. The second monitoring module 210 may also be used to cancel or establish monitoring of the corresponding message queue topic according to the updated online user list. The first monitoring module 230 may also be used to determine the user group corresponding to each online user.

[0118] Optionally, in this embodiment, the first monitoring module 230 can also be used to monitor changes in the user configuration information and the consumption data configuration information, and cancel or establish monitoring of the corresponding message queue according to the changes. The second monitoring module 210 can also be used to monitor changes in the user configuration information and the consumption data configuration information, and cancel or establish monitoring of the message queue topic according to the changes.

[0119] Please refer to Fig.11 , Fig.11 The third block diagram of the data consumption device 200 provided in the embodiment of the present application. The data consumption device 200 may further include a configuration module 201 .

[0120] The configuration module 201 is used to receive configuration operations input by the user through the configuration interface, and obtain the user configuration information and consumption data configuration information according to the configuration operations.

[0121] Optionally, the above modules can be stored in the form of software or firmware. Figure 1 The memory 110 shown in the figure may be fixed in the operating system (OS) of the server 100 and may be Figure 1 Meanwhile, the data and program codes required for executing the above modules may be stored in the memory 110.

[0122] An embodiment of the present application also provides a readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the data consumption method is implemented.

[0123] In summary, the embodiments of the present application provide a data consumption method, device, server and readable storage medium, which determines the identifier of the target message queue to be monitored and the first target preset consumption speed according to the user identifier of the target client and the first preset consumption speed and user identifier included in the stored configuration information of each user, and monitors the target message queue, wherein the identifier of the message queue is determined according to the corresponding user identifier; when it is monitored that there is data to be consumed in the target message queue, the data in the target consumption queue is consumed at a speed not greater than the first target preset consumption speed, and the consumed data is sent to the target client through a websocket link, and the data in the target message queue that exceeds the first target preset consumption speed is discarded. In this way, it is convenient to ensure the orderly, stable and real-time consumption of a large amount of data and distribute it to the target users.

[0124] In several embodiments provided in the present application, it should be understood that the disclosed devices and methods can also be implemented in other ways. The device embodiments described above are merely schematic. For example, the flowcharts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the devices, methods and computer program products according to multiple embodiments of the present application. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of a code, and the module, a program segment or a part of a code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart can be implemented with a dedicated hardware-based system that performs a specified function or action, or can be implemented with a combination of dedicated hardware and computer instructions.

[0125] In addition, the functional modules in the various embodiments of the present application may be integrated together to form an independent part, or each module may exist separately, or two or more modules may be integrated to form an independent part.

[0126] If the functions are implemented in the form of software function modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0127] The above description is only an optional embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A data consumption method, characterized in that: The method is applied to a server connected to a client for communication, wherein the server stores at least one user configuration information, wherein the user configuration information includes a first preset consumption speed and a user identifier, and comprises: According to the user identifier of the target client and the stored user configuration information, determine the identifier of the target message queue to be monitored and the first target preset consumption speed, and monitor the target message queue, wherein the identifier of the message queue is determined according to the corresponding user identifier; In the case of monitoring that there is data to be consumed in the target message queue, the data in the target consumption queue is consumed at a speed no greater than the first target preset consumption speed, and the consumed data is sent to the target client via a websocket link, wherein the data in the target message queue that exceeds the first target preset consumption speed is discarded.

2. The method according to claim 1, characterized in that: The server maintains an online user list, the online user list includes user identifiers of online users, the user configuration information includes a user name and a consumption data configuration identifier, and the identifier of the target message queue to be monitored is determined according to the user identifier of the target client and the stored user configuration information, including: Determine a user identifier of at least one target client from the online user list; Determine the user name and consumption data configuration identifier corresponding to the target client according to the user identifier of the target client and the stored user configuration information; The identifier of the target message queue is determined according to the user name and consumption data configuration identifier corresponding to the target client.

3. The method according to claim 1, characterized in that The server maintains an online user list, the online user list includes user identifiers of online users, the user configuration information includes an event name, and in the case where the user has associated users, the user configuration information also includes associated user identifiers, The method further comprises: Determine a user group corresponding to each online user according to the user identifier of each online user and the user identifier and the associated user identifier in the user configuration information, wherein the user group corresponding to an online user includes the online user and the associated users of the online user determined based on the associated user identifier; The step of sending the consumed data to the target client via a websocket link includes: According to the event name corresponding to each online user in the target user group corresponding to the target client, the data is sent to each online client in the target user group through a websocket link.

4. The method according to claim 1, characterized in that The method further comprises: Determine a target message queue topic to be monitored, and monitor the target message queue topic; When it is monitored that there is data to be consumed in the target message queue topic, the data is obtained from the target message queue topic, and the obtained data is pushed to the corresponding message queue.

5. The method according to claim 4, characterized in that The server maintains an online user list, the online user list includes user identifiers of online users, the server also stores at least one consumption data configuration information, the consumption data configuration information includes a consumption data configuration identifier and a message queue topic identifier, the user configuration information also includes a consumption data configuration identifier, and the determining of a target message queue topic to be monitored includes: Obtaining a target consumption data configuration identifier corresponding to each online user according to the user identifier of each online user and the stored user configuration information; According to the target consumption data configuration information corresponding to the target consumption data configuration identifier, a target message queue topic identifier corresponding to each online user is obtained, wherein the target message queue topic identifier is an identifier of the target message queue topic.

6. The method according to claim 4, characterized in that The method further comprises: Counting a second consumption speed of data on the target message queue topic; According to the second preset consumption speed and the second consumption speed, the consumption of the data of the target message queue topic is suspended or the consumption of the data of the target message queue topic is resumed.

7. The method according to claim 4, characterized in that The method further comprises: Counting the second consumption speed and generation speed of the data of the target message queue topic; When the second consumption speed continues to be lower than the generation speed for a period longer than a preset period, prompt information for prompting the user to improve consumption capacity is generated.

8. The method according to claim 2, 3 or 5, characterized in that: The method further comprises: In the case of changes in online users, the online user list is updated accordingly, and according to the updated online user list, monitoring of the corresponding message queue topic is canceled or established, and the user group corresponding to each online user is determined; and / or, Monitor changes in the user configuration information and the consumption data configuration information, and cancel or establish monitoring of corresponding message queue topics according to the changes, and cancel or establish monitoring of corresponding message queues according to the changes.

9. The method according to any one of claims 5 to 7, characterized in that: The method further comprises: Receive the configuration operation input by the user through the configuration interface, and obtain the user configuration information and consumption data configuration information according to the configuration operation.

10. A data consumption device, characterized in that: The device is applied to a server connected to a client for communication, wherein the server stores at least one user configuration information, wherein the user configuration information includes a first preset consumption speed and a user identifier, and comprises: A first monitoring module, used to determine the identifier of a target message queue to be monitored and a first target preset consumption speed according to a user identifier of a target client and the stored user configuration information, and monitor the target message queue, wherein the identifier of the message queue is determined according to the corresponding user identifier; A sending module is used to consume the data in the target consumption queue at a speed not greater than the first target preset consumption speed when it is monitored that there is data to be consumed in the target message queue, and send the consumed data to the target client through a websocket link, wherein the data in the target message queue that exceeds the first target preset consumption speed is discarded.

11. A server, characterized in that: It includes a processor and a memory, wherein the memory stores machine executable instructions that can be executed by the processor, and the processor can execute the machine executable instructions to implement the data consumption method described in any one of claims 1-9.

12. A readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the data consumption method according to any one of claims 1 to 9 is implemented.

Images