A log management method, device and related equipment

Abstract

The application relates to the technical field of data processing, in particular to a log management method and device and related equipment. The method aggregates the core management strategy of allocation in units of devices, so that most or even all of the user flow logs from the same front-end device have the same storage target as the back-end node on the front-end device, thereby maximizing the proportion of directly storing the user flow logs in the local back-end node. The user flow log forwarding amount in the back-end node cluster is significantly reduced, the internal network bandwidth is saved, the server load of the back-end node is reduced, and the processing efficiency and stability of the whole system are improved.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a log management method, apparatus and related equipment. Background Technology

[0002] Home broadband service is a fundamental service provided by telecom operators. With the intelligent upgrading of this service, operators have an increasing need for in-depth analysis and management of broadband user behavior, leading to the emergence of the iBRAS (Intelligent Broadband Remote Access Service System) business scenario to meet this demand.

[0003] In a typical iBRAS architecture, router devices equipped with intelligent analysis cards (referred to as the iBRAS front end) are responsible for collecting and uploading broadband user flow log data; a server cluster running broadband user operation and maintenance analysis software (referred to as the iBRAS back end) is responsible for receiving, storing, and analyzing this massive amount of flow log data. User flow log data is characterized by its huge volume (for example, 30 days of data from 10,000 users would require approximately 5TB of storage space) but low requirements for data reliability.

[0004] A city typically supports millions of broadband users, requiring multiple iBRAS front-end devices and multiple iBRAS back-end servers to work collaboratively. To ensure the rational utilization of storage and computing resources on each back-end server and avoid single-point overload, it is essential to achieve a balanced distribution of user flow log data across the nodes of the back-end server cluster. Existing technologies typically employ a data distribution scheme based on global user hashing: after the iBRAS back-end has all user information, it performs hash calculations on identifiers such as usernames, mapping each user to a specific back-end server node as the storage target for their flow logs; simultaneously, each iBRAS front-end device is statically configured with the target back-end server address for sending its flow logs.

[0005] However, this existing technical solution has the following significant drawbacks: Excessive east-west traffic within the backend cluster: Because hash calculations are global and random, there's a high probability that the streaming log data received by one iBRAS backend server will be stored on a different server. This generates a large amount of data forwarding between backend nodes (i.e., east-west traffic), which, in large-scale user scenarios, severely consumes internal cluster network bandwidth, increases server processing load, and affects overall system stability.

[0006] Inefficient and inflexible configuration: The targets sent by the iBRAS front-end devices need to be manually and statically configured. When expanding the network (adding new devices) or adjusting services, manual configuration is required for each device, which is inefficient, error-prone, and cannot adapt to dynamically changing business needs.

[0007] Therefore, there is an urgent need for an iBRAS user flow log processing method that can effectively reduce internal traffic in the backend cluster and achieve automated configuration while ensuring balanced data storage. Summary of the Invention

[0008] This application provides a log management method, apparatus, and related equipment.

[0009] Firstly, this application provides a log management method applied to a backend node cluster, wherein the backend node cluster includes a plurality of backend nodes; the method includes: Obtain the number of users currently connected to each front-end device, and calculate the expected number of users x to be carried by each back-end node in the back-end node cluster based on the total number of users currently connected to each front-end device and the number of back-end nodes included in the back-end node cluster. Based on the number of users currently connected to each front-end device and the expected number of users x to be carried by each back-end node, the following categories of front-end devices are defined: the first category is front-end devices where the difference between the current number of connected users and x is less than or equal to a preset value; the second category is front-end devices where the difference between the current number of connected users and x is greater than the preset value and the current number of connected users is greater than x; and the third category is front-end devices where the difference between the current number of connected users and x is greater than the preset value and the current number of connected users is less than x. For each type of front-end device, the upload target of the flow log of the currently connected user of the type of front-end device is configured as an idle back-end node in the back-end node cluster, and the storage node of the flow log of the currently connected user of the type of front-end device is configured as the back-end node. For each type of second-class front-end device, the upload target of the flow logs of the currently connected users of the second-class front-end device is configured as an idle back-end node in the back-end node cluster, and all users currently connected to the second-class front-end device are divided into a first user subset and a second user subset, and the storage node of the flow logs of the users included in the first user subset is configured as the back-end node, and the second subset is marked as users to be assigned, wherein the first subset includes x users; The third type of front-end devices are divided into multiple sets of third type front-end devices, so that the total number of users currently connected to each set of third type front-end devices is close to x. For each set of third type front-end devices, the upload target of the flow logs of the users currently connected to each set of third type front-end devices is configured as an idle back-end node in the back-end node cluster, and the storage node of the flow logs of the users currently connected to each set of third type front-end devices is configured as the back-end node. Configure the storage node for the flow logs of each user included in each second subset as the backend node where the current actual number of users supported has not yet reached x.

[0010] Optionally, the method further includes: Based on the established mapping relationship between each front-end device and the target back-end node of the stream logs it sends to the user, the communication address information of the target back-end node of the user's stream logs is sent to each front-end device, so that each front-end device can establish a communication link with the target back-end node based on the communication address information of the target back-end node of the user's stream logs.

[0011] Optionally, when any backend node in the backend node cluster receives a stream log sent by its corresponding frontend device, it extracts the user information of the stream log and determines the configured backend node for storing the user's stream log. If the backend node used to store the user's stream logs is a local node, then process and store them directly; If the backend node used to store the user's stream logs is another backend node, the stream logs will be forwarded to that other backend node for processing and storage through the internal network of the backend node cluster.

[0012] Optionally, when a change in the business status is detected, the method described in the first aspect above is re-executed, wherein the change in the business status includes: the number of users carried by any front-end device increases or decreases, and the increase or decrease in the number of users is greater than a set value, or the number of front-end devices increases or decreases.

[0013] Optionally, the method further includes: After a user goes online through the front-end device in the back-end node cluster, the mapping relationship between the user and the front-end device that went online is recorded. The steps to obtain the number of currently connected users on each front-end device include: Based on the mapping relationship between each user and its online front-end device recorded locally, obtain the number of users currently connected to each front-end device.

[0014] Secondly, this application provides a log management device applied to a backend node cluster, the backend node cluster including a plurality of backend nodes; the device includes: The acquisition unit is used to acquire the number of users currently connected to each front-end device, and calculate the expected number of users x to be carried by each back-end node in the back-end node cluster based on the total number of users currently connected to each front-end device and the number of back-end nodes included in the back-end node cluster. The determining unit is used to determine, based on the number of users currently connected to each front-end device and the number of users expected to be carried by each back-end node x, the following types of front-end devices: the first type of front-end devices whose difference between the current number of connected users and x is less than or equal to a preset value; the second type of front-end devices whose difference between the current number of connected users and x is greater than the preset value and whose current number of connected users is greater than x; and the third type of front-end devices whose difference between the current number of connected users and x is greater than the preset value and whose current number of connected users is less than x. The configuration unit is used to configure, for each type of front-end device, the upload target of the current user's flow log of the first type of front-end device as an idle back-end node in the back-end node cluster, and to configure the storage node of the current user's flow log of the first type of front-end device as the back-end node. The configuration unit is further configured to, for each second type of front-end device, configure the upload target of the flow logs of the currently connected users of the second type of front-end device as an idle back-end node in the back-end node cluster, divide all the currently connected users of the second type of front-end device into a first user subset and a second user subset, configure the storage node of the flow logs of the users included in the first user subset as the back-end node, and mark the second subset as users to be assigned, wherein the first subset includes x users; The configuration unit is further configured to divide the third type of front-end devices into multiple sets of third type of front-end devices, such that the total number of users currently connected to each set of third type of front-end devices is close to x. For each set of third type of front-end devices, the upload target of the flow logs of the currently connected users of each set of third type of front-end devices is configured as an idle back-end node in the back-end node cluster, and the storage node of the flow logs of the currently connected users of each set of third type of front-end devices is configured as the back-end node. The configuration unit is further configured to configure the storage node for the flow logs of each user included in each second subset as a backend node for which the current actual number of users carried has not yet reached x.

[0015] Optionally, the device further includes: The sending unit is used to send the communication address information of the target backend node of the stream log of the user to each frontend device based on the determined mapping relationship between each frontend device and the target backend node of the stream log of the user it sends, so that each frontend device can establish a communication link with the target backend node based on the communication address information of the target backend node of the stream log of the user it sends.

[0016] Optionally, the apparatus further includes a log processing unit, which is specifically used for: When a stream log is received from its corresponding front-end device, the user information of the stream log is extracted, and the configured back-end node for storing the user's stream log is determined. If the backend node used to store the user's stream logs is a local node, then process and store them directly; If the backend node used to store the user's stream logs is another backend node, the stream logs will be forwarded to that other backend node for processing and storage through the internal network of the backend node cluster.

[0017] Optionally, the device further includes a monitoring unit. When the monitoring unit detects a change in the service status, it triggers the device to re-execute the steps of the method described in the first aspect, wherein the change in the service status includes: the number of users carried by any front-end device increases or decreases, and the increase or decrease in the number of users is greater than a set value, or the number of front-end devices increases or decreases.

[0018] Optionally, the device further includes: The recording unit is used to record the mapping relationship between the user and the front-end device after the user goes online through the front-end device in the back-end node cluster. When obtaining the number of currently connected users on each front-end device, the obtaining unit is specifically used for: Based on the mapping relationship between each user and its online front-end device recorded locally, obtain the number of users currently connected to each front-end device.

[0019] Thirdly, embodiments of this application provide a log management device, which includes: Memory, used to store program instructions; A processor is configured to invoke program instructions stored in the memory and execute the steps of the method as described in any one of the first aspects above, according to the obtained program instructions.

[0020] Fourthly, embodiments of this application also provide a computer-readable storage medium storing computer-executable instructions for causing a computer to perform the steps of the method as described in any of the first aspects above.

[0021] In summary, the log management method provided in this application is applied to a backend node cluster, which includes a plurality of backend nodes. The method includes: obtaining the number of users currently connected to each frontend device, and calculating the expected number of users x to be carried by each backend node in the backend node cluster based on the total number of users currently connected to each frontend device and the number of backend nodes included in the backend node cluster; determining, based on the number of users currently connected to each frontend device and the expected number of users x to be carried by each backend node, a first type of frontend device where the difference between the current number of connected users and x is less than or equal to a preset value, a second type of frontend device where the difference between the current number of connected users and x is greater than a preset value and the current number of connected users is greater than x, and a third type of frontend device where the difference between the current number of connected users and x is greater than a preset value and the current number of connected users is less than x; for each first type of frontend device, configuring the upload target of the stream logs of the currently connected users of the first type of frontend device as an idle backend node in the backend node cluster, and configuring the storage node of the stream logs of the currently connected users of the first type of frontend device as that backend node. For each second-type front-end device, the upload target for the flow logs of the currently connected users of that second-type front-end device is configured as an idle back-end node in the back-end node cluster. All currently connected users of that second-type front-end device are divided into a first user subset and a second user subset. The storage node for the flow logs of the users included in the first user subset is configured as the back-end node. The second subset is marked as users to be assigned. The first subset includes x users. Third-type front-end devices are divided into multiple third-type front-end device sets such that the total number of currently connected users in each third-type front-end device set is close to x. For each third-type front-end device set, the upload target for the flow logs of the currently connected users of each third-type front-end device in that third-type front-end device set is configured as an idle back-end node in the back-end node cluster. The storage node for the flow logs of the currently connected users of each third-type front-end device in that third-type front-end device set is configured as the back-end node. The storage node for the flow logs of each user in each second subset is configured as a back-end node whose current actual number of connected users has not yet reached x.

[0022] The log management method provided in this application, through the core strategy of "aggregation and allocation on a device-by-device basis," ensures that the storage target of most or even all user stream logs from the same front-end device is consistent with the upload target of the front-end device. This maximizes the proportion of stream logs directly stored locally on the receiving node (back-end node). This significantly reduces data forwarding traffic within the back-end node cluster, saves internal network bandwidth, reduces server load, and improves the processing efficiency and stability of the entire system. Attached Figure Description

[0023] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments of this application or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this application. For those skilled in the art, other drawings can be obtained based on these drawings of the embodiments of this application.

[0024] Figure 1 A detailed flowchart of a log management method provided in an embodiment of this application; Figure 2 A schematic diagram illustrating the connection between a front-end device and a back-end node provided in an embodiment of this application; Figure 3 This is a schematic diagram of the structure of a log management device provided in an embodiment of this application; Figure 4 This is a schematic diagram of the hardware architecture of a log management device provided in an embodiment of this application. Detailed Implementation

[0025] The terminology used in the embodiments of this application is for the purpose of describing particular embodiments only and is not intended to limit the application. The singular forms “a,” “the,” and “the” as used in this application and claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to any and all possible combinations comprising one or more of the associated listed items.

[0026] It should be understood that although the terms first, second, third, etc., may be used to describe various information in embodiments of this application, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, without departing from the scope of this application, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" may also be interpreted as "when," "when," or "in response to a determination."

[0027] For example, see Figure 1 The diagram shown is a detailed flowchart of a log management method provided in this application embodiment, applied to a backend node cluster, wherein the backend node cluster includes several backend nodes; the method includes the following steps: Step 100: Obtain the number of users currently connected to each front-end device, and calculate the expected number of users x to be carried by each back-end node in the back-end node cluster based on the total number of users currently connected to each front-end device and the number of back-end nodes included in the back-end node cluster.

[0028] In this embodiment of the application, after a user goes online through a front-end device in the back-end node cluster, the mapping relationship between the user and the front-end device is recorded; therefore, when obtaining the number of users currently connected to each front-end device, a preferred implementation is as follows: Based on the mapping relationship between each user and its online front-end device recorded locally, obtain the number of users currently connected to each front-end device.

[0029] For example, broadband users complete the online process in the iBRAS backend (backend node) through the authentication process. The iBRAS backend records the information of each online user and its corresponding online source iBRAS frontend device (frontend device), forming a user-frontend device mapping relationship.

[0030] In other words, once a user logs into the iBRAS backend, the iBRAS backend possesses the user's basic information, the frontend device they are on, and other relevant information. For example, Table 1 illustrates the mapping relationship between users and frontend devices: Table 1 In this way, the backend node can determine the number of users currently connected to each frontend device based on the above mapping table.

[0031] Furthermore, the total number of users currently connected to each front-end device is calculated, and based on the total number of users currently connected to each front-end device and the number of back-end nodes included in the back-end node cluster, the expected number of users x to be carried by each back-end node in the back-end node cluster is calculated.

[0032] In other words, assuming there are m iBRAS backend nodes, the number of users supported by each backend node is calculated according to the following formula, and the storage usage of each node remains balanced: x = (dev1 + dev2 + ... + devn) / m, (where devn is the number of users online from device n, and m is the number of backend nodes) For example, this application embodiment uses an initial deployment scenario of a system containing 5 iBRAS front-end devices (devices 1-5) and 3 iBRAS back-end nodes (nodes 1-3) as an example. Assume that after all users are online, the back-end node cluster counts the current number of users connected to each front-end device as follows: Device 1: 500,000, Device 2: 450,000, Device 3: 200,000, Device 4: 150,000, Device 5: 50,000. The total number of users currently connected to each front-end device is: 500,000 + 450,000 + 200,000 + 150,000 + 50,000 = 1,350,000. With 3 back-end nodes, the expected number of users each back-end node will support is x: 1,350,000 / 3 = 450,000. For example, see Table 2, which illustrates the statistics of the current number of users connected to each front-end device. Table 2 Step 110: Based on the number of users currently connected to each front-end device and the expected number of users x to be carried by each back-end node, determine the following front-end devices: the first type of front-end device is the one where the difference between the number of users currently connected and x is less than or equal to a preset value; the second type of front-end device is the one where the difference between the number of users currently connected and x is greater than a preset value and the number of users currently connected is greater than x; and the third type of front-end device is the one where the difference between the number of users currently connected and x is greater than a preset value and the number of users currently connected is less than x.

[0033] As mentioned above, the expected number of users x for each backend node is 135 / 3 = 450,000. Based on the above division rules (assuming a preset value of 10,000, but it can also be other values ​​(e.g., 0), in actual applications, it can be customized according to user needs), device 2 is determined to be the first type of front-end device, device 1 is the second type of front-end device, and devices 3, 4 and 5 are the third type of front-end nodes.

[0034] Step 120: For each type of front-end device, configure the upload target of the flow log of the currently connected user of the type of front-end device as an idle back-end node in the back-end node cluster, and configure the storage node of the flow log of the currently connected user of the type of front-end device as the back-end node.

[0035] In this embodiment of the application, for each first type of front-end device (e.g., dev2, front-end device 2), since the number of users currently connected to the first type of front-end device is equal to or close to the number of users expected to be carried by the back-end node (i.e., the above average value x (450,000)), the storage target of the flow logs of the users currently connected to the front-end device 1 can be allocated to the same back-end node (e.g., back-end node 1), and the back-end node for the flow logs of the users sent by the first type of front-end device can also be configured as back-end node 1.

[0036] Step 130: For each type of second-class front-end device, configure the upload target of the flow logs of the currently connected users of the second-class front-end device as an idle back-end node in the back-end node cluster, divide all the currently connected users of the second-class front-end device into a first user subset and a second user subset, configure the storage node of the flow logs of the users included in the first user subset as the back-end node, and mark the second subset as users to be assigned, wherein the first subset includes x users.

[0037] In this embodiment of the application, for each second type of front-end device (e.g., dev1, front-end device 1), since the number of users currently connected to the second type of front-end device (500,000) is greater than the number of users expected to be carried by the back-end node (i.e., the above average value x (450,000)), the users currently connected to the second type of front-end device can be divided into a first subset (including 450,000 users) and a second subset (including 50,000 users); the storage target of the flow logs of the users included in the first subset is allocated to the same back-end node (e.g., back-end node 2), and the back-end node for the flow logs of the users uploaded by the first type of front-end device is also configured as back-end node 2; the storage target of the flow logs of the users included in the second subset is to be allocated.

[0038] Step 140: Divide the third type of front-end devices into multiple sets of third type of front-end devices, such that the total number of users currently connected to each set of third type of front-end devices is close to x. For each set of third type of front-end devices, configure the upload target of the flow logs of the currently connected users of each set of third type of front-end devices as an idle back-end node in the back-end node cluster, and configure the storage node of the flow logs of the currently connected users of each set of third type of front-end devices as that back-end node.

[0039] In this embodiment of the application, for the second type of front-end devices, multiple third type of front-end devices (e.g., dev3, dev4, and dev5) are combined to obtain at least one set of third type of front-end devices. The total number of users currently connected to each third type of front-end device set (20+15+5=400,000) is close to x (e.g., x is 450,000, and the difference from x is less than or equal to a preset value). For each third type of front-end device set, the storage target of the flow logs of the users currently connected to each third type of front-end device included in the third type of front-end device set is allocated to the same back-end node (e.g., back-end node 3), and the back-end node for the flow logs of the users sent by each third type of front-end device included in the third type of front-end device set is also configured as back-end node 3.

[0040] Step 150: Configure the storage node for the flow logs of each user included in each second subset as the backend node for which the current actual number of users has not yet reached x.

[0041] In this embodiment of the application, the second subset of the front-end device 1 also includes 50,000 users, and the back-end node 3 currently carries 400,000 users. Therefore, the storage target of the flow logs of the 50,000 users included in the second subset of the front-end device 1 can be allocated to the back-end node 3.

[0042] For example, the backend nodes allocated for storing user flow logs for each currently connected user on each front-end device, and the configuration for uploading user flow logs to the front-end devices are shown in Table 3: Table 3 For example, participation Figure 2 The diagram shown illustrates the connection between a front-end device and a back-end node according to an embodiment of this application. After allocating the storage nodes (back-end nodes) for the flow logs of currently connected users on each front-end device and configuring the upload targets for the flow logs of currently connected users on each front-end device as described above, only the flow log data of 50,000 users will flow from node 2 to node 3 among the various nodes in the iBRAS back-end. The flow logs of other users are received by the back-end nodes and stored and maintained locally.

[0043] In this embodiment of the application, the above-mentioned log management method may further include the following steps: Based on the established mapping relationship between each front-end device and the target back-end node of the stream logs it sends to the user, the communication address information of the target back-end node of the user's stream logs is sent to each front-end device, so that each front-end device can establish a communication link with the target back-end node based on the communication address information of the target back-end node of the user's stream logs.

[0044] The configuration of the flow logs on the device is no longer fixed by manual configuration, but is configured through the iBRAS backend via netconf.

[0045] As shown in Table 4, the configuration of the example above is as follows: Device 1 sends to Node 2, Device 2 sends to Node 1, and Device 3, Device 4, and Device 5 send to Node 3.

[0046] Table 4 Therefore, in this embodiment of the application, when any backend node in the backend node cluster receives the stream log sent by its corresponding frontend device, it extracts the user information of the stream log and determines the backend node configured to store the user's stream log. If the backend node used to store the user's stream logs is a local node, then process and store them directly; If the backend node used to store the user's stream logs is another backend node, the stream logs will be forwarded to that other backend node for processing and storage through the internal network of the backend node cluster.

[0047] Specifically, each iBRAS backend server receives user stream log data from the device. It then extracts user information from the stream log data and retrieves the target storage server for that user's data.

[0048] After processing the user's stream log data, if the user's data storage target is local, it can be stored locally directly. If the user's data storage target is not local, the corresponding target server is connected via network and the data is stored on the corresponding target server.

[0049] Furthermore, in this embodiment of the application, when a change in the business status is detected, the above-mentioned log management method is re-executed. The change in the business status includes: the number of users carried by any front-end device increases or decreases, and the increase or decrease in the number of users is greater than a set value, or the number of front-end devices increases or decreases.

[0050] In practical applications, new business may occur during nighttime when traffic is infrequent. In such cases, the storage target for broadband user flow logs and the data upload configuration for front-end devices will be dynamically recalculated and redistributed to the devices. For example, if the number of users on a device increases—such as dev5 increasing from 50,000 to 200,000 users in the example above, while the number of users on other devs remains unchanged—then the expected number of users each iBRAS backend node can support is: x = (50 + 45 + 20 + 15 + 20) / 3 = 500,000. This means that each iBRAS node can support the flow logs of 500,000 users, ensuring a relatively balanced storage environment.

[0051] Following the same method of allocating user data storage targets as described above, the allocation results are shown in Table 5: Table 5 In summary, in the iBRAS backend, the flow logs of 50,000 users on node 3 will pass through the east-west network and be stored on node 2. The flow logs of the other 1.45 million users will be received, processed, and stored locally from the receiving node.

[0052] For example, when a large number of new users are added, such as when new users come online on backend nodes via new frontend devices (dev6 is a newly added frontend device with 150,000 users currently connected, while the number of users connected to other devices remains unchanged), then the expected number of users that each iBRAS backend node can support is: x = (50 + 45 + 20 + 15 + 20) / 3 = 500,000. That is, each iBRAS node can support the streaming logs of 500,000 users, ensuring a basically balanced storage load.

[0053] Following the same method of allocating user data storage targets as described above, the allocation results are shown in Table 6: Table 6 In summary, in the iBRAS backend, the flow logs of 50,000 users on node 3 will pass through the east-west network and be stored on node 2. The flow logs of the other 1.45 million users will be received, processed, and stored locally from the receiving node.

[0054] Based on the same inventive concept as the above-described embodiments, see, for example, the following: Figure 3 The diagram shown is a structural schematic of a log management device provided in an embodiment of this application. This device is applied to a backend node cluster, which includes a plurality of backend nodes. The device includes: The acquisition unit 30 is used to acquire the number of users currently connected to each front-end device, and calculate the expected number of users x to be carried by each back-end node in the back-end node cluster based on the total number of users currently connected to each front-end device and the number of back-end nodes included in the back-end node cluster. The determining unit 31 is used to determine, based on the number of users currently connected to each front-end device and the number of users expected to be carried by each back-end node x, the following types of front-end devices: the first type of front-end devices whose difference between the current number of connected users and x is less than or equal to a preset value; the second type of front-end devices whose difference between the current number of connected users and x is greater than a preset value and whose current number of connected users is greater than x; and the third type of front-end devices whose difference between the current number of connected users and x is greater than a preset value and whose current number of connected users is less than x. Configuration unit 32 is used to configure, for each type of front-end device, the upload target of the current user's flow log of the first type of front-end device as an idle back-end node in the back-end node cluster, and configure the storage node of the current user's flow log of the first type of front-end device as the back-end node. The configuration unit 32 is further configured to, for each second type of front-end device, configure the upload target of the flow logs of the currently connected users of the second type of front-end device as an idle back-end node in the back-end node cluster, divide all the currently connected users of the second type of front-end device into a first user subset and a second user subset, configure the storage node of the flow logs of the users included in the first user subset as the back-end node, and mark the second subset as users to be assigned, wherein the first subset includes x users; The configuration unit 32 is further configured to divide the third type of front-end devices into multiple third type of front-end device sets, such that the total number of users currently connected to each third type of front-end device set is close to x. For each third type of front-end device set, the upload target of the flow logs of the currently connected users of each third type of front-end device included in the third type of front-end device set is configured as an idle back-end node in the back-end node cluster, and the storage node of the flow logs of the currently connected users of each third type of front-end device included in the third type of front-end device set is configured as the back-end node. The configuration unit 32 is further configured to configure the storage node of the flow log of each user included in each second subset as the backend node where the current actual number of users carried has not yet reached x.

[0055] Optionally, the device further includes: The sending unit is used to send the communication address information of the target backend node of the stream log of the user to each frontend device based on the determined mapping relationship between each frontend device and the target backend node of the stream log of the user it sends, so that each frontend device can establish a communication link with the target backend node based on the communication address information of the target backend node of the stream log of the user it sends.

[0056] Optionally, the apparatus further includes a log processing unit, which is specifically used for: When a stream log is received from its corresponding front-end device, the user information of the stream log is extracted, and the configured back-end node for storing the user's stream log is determined. If the backend node used to store the user's stream logs is a local node, then process and store them directly; If the backend node used to store the user's stream logs is another backend node, the stream logs will be forwarded to that other backend node for processing and storage through the internal network of the backend node cluster.

[0057] Optionally, the device further includes a monitoring unit. When the monitoring unit detects a change in the service status, it triggers the device to re-execute the steps of the method described in the first aspect, wherein the change in the service status includes: the number of users carried by any front-end device increases or decreases, and the increase or decrease in the number of users is greater than a set value, or the number of front-end devices increases or decreases.

[0058] Optionally, the device further includes: The recording unit is used to record the mapping relationship between the user and the front-end device after the user goes online through the front-end device in the back-end node cluster. When obtaining the number of users currently connected to each front-end device, the obtaining unit 30 is specifically used for: Based on the mapping relationship between each user and its online front-end device recorded locally, obtain the number of users currently connected to each front-end device.

[0059] These units can be one or more integrated circuits configured to implement the above methods, such as one or more Application Specific Integrated Circuits (ASICs), one or more digital signal processors (DSPs), or one or more Field Programmable Gate Arrays (FPGAs). Alternatively, when one of these units is implemented using processing element scheduler code, the processing element can be a general-purpose processor, such as a Central Processing Unit (CPU) or other processor capable of calling program code. Furthermore, these units can be integrated together to form a system-on-a-chip (SOC).

[0060] Furthermore, regarding the log management device provided in this application embodiment, from a hardware perspective, the hardware architecture diagram of the log management device can be found in [reference needed]. Figure 4 As shown, the log management device may include: a memory 40 and a processor 41. The memory 40 is used to store program instructions; the processor 41 calls the program instructions stored in the memory 40 and executes the above method embodiment according to the obtained program instructions. The specific implementation method and technical effect are similar, and will not be described again here.

[0061] Optionally, this application also provides a backend node device, including at least one processing element (or chip) for performing the above method embodiments.

[0062] Optionally, this application also provides a program product, such as a computer-readable storage medium storing computer-executable instructions for causing the computer to perform the above-described method embodiments.

[0063] Here, a machine-readable storage medium can be any electronic, magnetic, optical, or other physical storage device that can contain or store information, such as executable instructions, data, etc. For example, a machine-readable storage medium can be: RAM (Random Access Memory), volatile memory, non-volatile memory, flash memory, storage drives (such as hard disk drives), solid-state drives, any type of storage disk (such as optical discs, DVDs, etc.), or similar storage media, or combinations thereof.

[0064] The systems, devices, modules, or units described in the above embodiments can be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer, which can take the form of a personal computer, laptop computer, cellular phone, camera phone, smartphone, personal digital assistant, media player, navigation device, email sending and receiving device, game console, tablet computer, wearable device, or any combination of these devices.

[0065] For ease of description, the above devices are described separately by function as various units. Of course, in implementing this application, the functions of each unit can be implemented in one or more software and / or hardware.

[0066] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, embodiments of this application can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0067] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0068] Furthermore, these computer program instructions can also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in the process. Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0069] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0070] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A log management method, characterized in that, Applied to a backend node cluster, wherein the backend node cluster includes a plurality of backend nodes; the method includes: Obtain the number of users currently connected to each front-end device, and calculate the expected number of users x to be carried by each back-end node in the back-end node cluster based on the total number of users currently connected to each front-end device and the number of back-end nodes included in the back-end node cluster. Based on the number of users currently connected to each front-end device and the expected number of users x to be carried by each back-end node, the following categories of front-end devices are defined: the first category is front-end devices where the difference between the current number of connected users and x is less than or equal to a preset value; the second category is front-end devices where the difference between the current number of connected users and x is greater than the preset value and the current number of connected users is greater than x; and the third category is front-end devices where the difference between the current number of connected users and x is greater than the preset value and the current number of connected users is less than x. For each type of front-end device, the upload target of the flow log of the currently connected user of the type of front-end device is configured as an idle back-end node in the back-end node cluster, and the storage node of the flow log of the currently connected user of the type of front-end device is configured as the back-end node. For each type of second-class front-end device, the upload target of the flow logs of the currently connected users of the second-class front-end device is configured as an idle back-end node in the back-end node cluster, and all users currently connected to the second-class front-end device are divided into a first user subset and a second user subset, and the storage node of the flow logs of the users included in the first user subset is configured as the back-end node, and the second subset is marked as users to be assigned, wherein the first subset includes x users; The third type of front-end devices are divided into multiple sets of third type front-end devices, so that the total number of users currently connected to each set of third type front-end devices is close to x. For each set of third type front-end devices, the upload target of the flow logs of the users currently connected to each set of third type front-end devices is configured as an idle back-end node in the back-end node cluster, and the storage node of the flow logs of the users currently connected to each set of third type front-end devices is configured as the back-end node. Configure the storage node for the flow logs of each user included in each second subset as the backend node whose current actual number of users has not yet reached x.

2. The method as described in claim 1, characterized in that, The method further includes: Based on the established mapping relationship between each front-end device and the target back-end node of the stream logs it sends to the user, the communication address information of the target back-end node of the user's stream logs is sent to each front-end device, so that each front-end device can establish a communication link with the target back-end node based on the communication address information of the target back-end node of the user's stream logs.

3. The method as described in claim 1 or 2, characterized in that, When any backend node in the backend node cluster receives a stream log sent by its corresponding frontend device, it extracts the user information of the stream log and determines the backend node configured to store the user's stream log. If the backend node used to store the user's stream logs is a local node, then process and store them directly; If the backend node used to store the user's stream logs is another backend node, the stream logs will be forwarded to that other backend node for processing and storage through the internal network of the backend node cluster.

4. The method as described in claim 1, characterized in that, When a change in the business status is detected, the method described in claim 1 is re-executed, wherein the change in the business status includes: the number of users carried by any front-end device increases or decreases, and the increase or decrease in the number of users is greater than a set value, or the number of front-end devices increases or decreases.

5. The method as described in claim 1, characterized in that, The method further includes: After a user goes online through the front-end device in the back-end node cluster, the mapping relationship between the user and the front-end device that went online is recorded. The steps to obtain the number of currently connected users on each front-end device include: Based on the mapping relationship between each user and its online front-end device recorded locally, obtain the number of users currently connected to each front-end device.

6. A log management device, characterized in that, Applied to a backend node cluster, the backend node cluster comprising a plurality of backend nodes; the device includes: The acquisition unit is used to acquire the number of users currently connected to each front-end device, and calculate the expected number of users x to be carried by each back-end node in the back-end node cluster based on the total number of users currently connected to each front-end device and the number of back-end nodes included in the back-end node cluster. The determining unit is used to determine, based on the number of users currently connected to each front-end device and the number of users expected to be carried by each back-end node x, the following types of front-end devices: the first type of front-end devices whose difference between the current number of connected users and x is less than or equal to a preset value; the second type of front-end devices whose difference between the current number of connected users and x is greater than the preset value and whose current number of connected users is greater than x; and the third type of front-end devices whose difference between the current number of connected users and x is greater than the preset value and whose current number of connected users is less than x. The configuration unit is used to configure, for each type of front-end device, the upload target of the current user's flow log of the first type of front-end device as an idle back-end node in the back-end node cluster, and to configure the storage node of the current user's flow log of the first type of front-end device as the back-end node. The configuration unit is further configured to, for each second type of front-end device, configure the upload target of the flow logs of the currently connected users of the second type of front-end device as an idle back-end node in the back-end node cluster, divide all the currently connected users of the second type of front-end device into a first user subset and a second user subset, configure the storage node of the flow logs of the users included in the first user subset as the back-end node, and mark the second subset as users to be assigned, wherein the first subset includes x users; The configuration unit is further configured to divide the third type of front-end devices into multiple sets of third type of front-end devices, such that the total number of users currently connected to each set of third type of front-end devices is close to x. For each set of third type of front-end devices, the upload target of the flow logs of the currently connected users of each set of third type of front-end devices is configured as an idle back-end node in the back-end node cluster, and the storage node of the flow logs of the currently connected users of each set of third type of front-end devices is configured as the back-end node. The configuration unit is further configured to configure the storage node for the flow logs of each user included in each second subset as a backend node for which the current actual number of users carried has not yet reached x.

7. The apparatus as claimed in claim 6, characterized in that, The device further includes: The sending unit is used to send the communication address information of the target backend node of the stream log of the user to each frontend device based on the determined mapping relationship between each frontend device and the target backend node of the stream log of the user it sends, so that each frontend device can establish a communication link with the target backend node based on the communication address information of the target backend node of the stream log of the user it sends.

8. The apparatus as claimed in claim 6 or 7, characterized in that, The device further includes a log processing unit, which is specifically used for: When a stream log is received from its corresponding front-end device, the user information of the stream log is extracted, and the configured back-end node for storing the user's stream log is determined. If the backend node used to store the user's stream logs is a local node, then process and store them directly; If the backend node used to store the user's stream logs is another backend node, the stream logs will be forwarded to that other backend node for processing and storage through the internal network of the backend node cluster.

9. The apparatus as claimed in claim 6, characterized in that, The device further includes: The recording unit is used to record the mapping relationship between the user and the front-end device after the user goes online through the front-end device in the back-end node cluster. When obtaining the number of currently connected users on each front-end device, the obtaining unit is specifically used for: Based on the mapping relationship between each user and its online front-end device recorded locally, obtain the number of users currently connected to each front-end device.

10. A log management device, characterized in that, The log management device includes: Memory, used to store program instructions; A processor is configured to invoke program instructions stored in the memory and execute the steps of the method as described in any one of claims 1-5 according to the obtained program instructions.

11. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions for causing the computer to perform the steps of the method as described in any one of claims 1-5.

Images