Data acquisition method, device, electronic device, medium and computer program product

Abstract

The present invention provides a data acquisition method, apparatus, electronic device, medium, and computer program product. The method comprises: obtaining, by a server, all currently configured server nodes and current device data from a database table; adding a field to the database table for storing a device identification value; and traversing the current device data, performing a modulo calculation on the device identification value of the current device data and the number of all currently configured server nodes to obtain the device data of the server. The present invention effectively utilizes server resources, addresses the shortcomings of the active-standby mode, and enables all deployed server nodes to operate simultaneously.

Description

Technical Field

[0001] The present invention relates to the field of computer applications, and in particular to a data acquisition method, device, electronic equipment, medium and computer program product. Background Art

[0002] In a cluster environment, a service is deployed with N+1 servers, including one primary server and multiple backup servers. When the service is started on these N+1 servers, only the primary server can retrieve data, while the other backup servers are inoperative. This wastes server resources and results in inefficient table retrieval, with data retrieval taking a long time to display. Furthermore, the primary server consumes high resources (CPU and memory), while the backup servers are not fully utilized, resulting in significant waste of server resources. Summary of the Invention

[0003] The present invention provides a data acquisition method, device, electronic device, medium and computer program product, which are used to solve the defect of waste of server resources in the active-standby service mode in the prior art and realize the effective utilization of server resources.

[0004] The present invention provides a data acquisition method, comprising:

[0005] This server obtains all currently configured server nodes and current device data in the database table;

[0006] The server adds a field to the database table, wherein the field is used to store the device identification value;

[0007] This server traverses the current device data, and takes the remainder of the device identification value of the current device data and the node number of all the currently configured server nodes. If the remainder is consistent with the node value of this server, the current device data is obtained as the device data of this server.

[0008] According to a data acquisition method provided by the present invention, the data type of the field is BigInt.

[0009] According to a data acquisition method provided by the present invention, the device identification value of the current device data is modulo the number of all currently configured server nodes to obtain the device data of the current server, including:

[0010] The device identification value of the current device data is used as the dividend, the number of all currently configured server nodes is used as the divisor, and device data that satisfies the remainder is obtained as the device data of the current server.

[0011] According to a data acquisition method provided by the present invention, the device identification value is obtained by calculating the device ID, and the device ID of each device is unique.

[0012] According to a data acquisition method provided by the present invention, the device identification value includes a CRC value of the device ID whose data type is Long.

[0013] According to a data acquisition method provided by the present invention, all the currently configured server nodes do not distinguish between primary and backup services.

[0014] The present invention also provides a data acquisition device, comprising:

[0015] The acquisition module is used to obtain all currently configured server nodes and current device data in the database table;

[0016] A modification module, configured to add a field to a database table, wherein the field is configured to store a device identification value;

[0017] The query module is used to traverse the current device data, take the remainder of the device identification value of the current device data and the node number of all the currently configured server nodes, and if the remainder is consistent with the node value of the current server, obtain the current device data as the device data of the current server.

[0018] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of any of the above-described data acquisition methods when executing the program.

[0019] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which implements the steps of any of the above-mentioned data acquisition methods when executed by a processor.

[0020] The present invention also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the steps of any of the above-mentioned data acquisition methods are implemented.

[0021] The present invention provides a data acquisition method, apparatus, electronic device, medium, and computer program product. These methods disable the active / standby service mode for services deployed in a cluster, add a field for storing device identification values, and use database function remainder calculations to retrieve data. This effectively utilizes server resources, allowing all deployed server nodes to operate simultaneously. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the technical solutions in the present invention or the prior art, a brief introduction is given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0023] Figure 1 This is one of the flow charts of the data acquisition method provided by an embodiment of the present invention;

[0024] Figure 2 This is the second flow chart of the data acquisition method provided by the embodiment of the present invention;

[0025] Figure 3 is a structural diagram of a data acquisition device provided by an embodiment of the present invention;

[0026] Figure 4 It is a structural diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0027] To make the objectives, technical solutions, and advantages of the present invention more clear, the technical solutions of the present invention will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the embodiments described are only some of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present invention.

[0028] When a server cluster running in master-slave mode faces a large number of devices and a large amount of data, there is often a problem of wasted server resources. For example, when the amount of calculated data is large, only the master server obtains the calculation results of the device, and the backup server does not work. The memory and CPU utilization of the master service are high. Therefore, the present invention proposes a data acquisition method to solve the above problem. In this embodiment, the device data includes: message push strategy, terminal information collection strategy, Internet behavior audit strategy and other data that are handed over to the device for calculation. The types of device data in this embodiment include: ip (device ip), ipType (ip type), ipNumber (calculated ip number value), mac (device mac value), devOnlyId (unique id), organizationId (affiliated organization), regState (registration status), deleteState (deletion status), assetStatus (asset status), approvalState (approval status), agentPackType (device type).

[0029] The following combination Figure 1 The data acquisition method according to an embodiment of the present invention includes:

[0030] Step 101: The server obtains all currently configured server nodes and current device data in the database table;

[0031] It should be noted that the calculation result data of the device data is identified by the device's devOnlyId and onlyCrc (the crc value calculated based on the unique id). CRC is a cyclic redundancy check code. It is generally called crc check, abbreviated as crc. The calculation results of all devices are stored in the same table. The current device data in the database table is any one of all the device data stored before it is classified into different servers. In this embodiment, there are 500,000 device data stored in the database table. If each device data corresponds to 3 policies, that is, each device has 3 device data.

[0032] Step 102: The server adds a field to the database table, where the field is used to store the device identification value.

[0033] Step 103: The server traverses the current device data, and calculates the remainder of the device identification value of the current device data and the node number of all the currently configured server nodes. If the remainder is consistent with the node value of the server, the current device data is obtained as the device data of the server.

[0034] It should be noted that the term "all currently configured server nodes" refers to all nodes configured by the configuration center for the current service. Prior to executing the data acquisition method of the present invention, the active / standby mode algorithm has been disabled by the configuration center for this server, and each server node is not configured for active / standby mode. In step 101, this server monitors the database table in real time through the configuration thread, acquiring the current device data from the database table in real time. This server refers to any one of the deployed server nodes. The server node value refers to the sequence number of this server node among all nodes.

[0035] The data acquisition method of the present invention overcomes the shortcomings of traditional clusters in active-standby mode by adding a field for storing device identification values ​​and using a database function to obtain data. This method effectively utilizes server resources and allows all deployed server nodes to run simultaneously.

[0036] In at least one embodiment of the present invention, the data type of the field is BigInt. This data type is selected because the field is stored in a database and is subsequently used as dividend data, and the dividend must meet the numeric type. At the same time, to accommodate the CRC value type of long, the field type is selected as BigInt.

[0037] In at least one embodiment of the present invention, the step of taking the modulus of the device identification value of the current device data and the number of all currently configured server nodes, and obtaining the current device data as the device data of the current server if the modulus is consistent with the node value of the current server, includes:

[0038] The device identification value of the current device data is used as the dividend, and the number of all server nodes currently configured is used as the divisor. If the remainder is consistent with the node value of the current server, the current device data is obtained as the device data of the current server.

[0039] It should be noted that the remainder operation is performed using the remainder function in the database, as shown in Formula 1:

[0040] mod(numer1,number2)=number3 Formula 1

[0041] Here, numer1 represents the device identification value stored in the newly added BigInt field, number2 represents the number of all server nodes, and number3 represents the node value of the current server. The node value of this server refers to the sequence number of this server among all server nodes. For example, if mod(10,3) = 1, the device identification value of the current device data is 10 divided by 3 nodes, and the remainder is 1, which is the data of the server with sequence number 1.

[0042] In the data acquisition method of the embodiment of the present invention, the service node is used as the divisor and remainder of the function (MOD), so that the service can be deployed on one or more servers and can calculate and obtain correct data well.

[0043] In at least one embodiment of the present invention, the device identification value is obtained by calculating the device ID, and the device ID of each device is unique.

[0044] It should be noted that the device ID format is as follows:

[0045] 8e79c7f681cfbe86b3c7df52c05504b3.

[0046] The data obtained by each server is unique to a fixed device, ensuring data consistency. This is because the Long CRC value calculated for each devOnlyId is the same, and the remainder is also the same. Therefore, if each service node remains unchanged, the data obtained is data for the same devOnlyId, avoiding data confusion caused by different service nodes obtaining different data for the same devOnlyId.

[0047] In at least one embodiment of the present invention, the device identification value includes a CRC value of the device ID whose data type is Long.

[0048] In at least one embodiment of the present invention, all server nodes currently configured do not distinguish between primary and backup services. Service cluster deployment does not distinguish between primary and backup services, and multiple services run simultaneously, effectively utilizing server resources and improving program operation efficiency.

[0049] The present invention also includes a method for monitoring data during service operation using the data acquisition method of the above embodiment. The method first monitors and then acquires data. A thread is started and continuously runs to detect whether there is data in the data table. If there is data, the data is acquired. If there is no data, no operation is performed. After collecting data, the data in the database table is synchronized and cached in Redis. This includes:

[0050] 1. Deploy cluster service. This service deploys multiple server nodes.

[0051] It should be noted that, in the embodiment of the present invention, if the device data is less than 10,000, this service is deployed on a single machine; if the device data is 10,000-40,000, this service is deployed on 4 server nodes; if the device data is 40,000-100,000, this service is deployed on 13 server nodes; if the device data is more than 100,000, this service is deployed on servers divided according to the specific amount of computing devices.

[0052] 2. The database table adds the onlyCrc field value that stores the Long type crc.

[0053] 3. Start the service, run the business, check the log files and observe the database table data.

[0054] 4. Observe the database table. When there is data, the program will obtain device data in real time.

[0055] 5. Check whether the final result data is disordered.

[0056] It should be noted that data confusion can occur when both the message push policy and the terminal information collection policy are calculated using all device data, while one selects newly added data and the other selects the newly added data after calculation. This results in device data for the message push policy but no data for the terminal information collection policy. If a device is processed simultaneously by multiple nodes at different times with different values, the processing results will be inconsistent.

[0057] 6. Observe server resources (CPU and memory usage of the service) during service program operation.

[0058] like Figure 2 As shown, an embodiment of the present invention discloses a data acquisition method, including:

[0059] Step 201: The server obtains all nodes from the configuration center;

[0060] Step 202: Obtain the total number of nodes and the node value of the server node;

[0061] Step 203: Assemble respective node values;

[0062] It should be noted that when data is assembled, it will be assembled into a set value map for transmission. If it is not assembled, the total number of nodes and the node value of this service will be transmitted separately. The assembled value format is as follows:

[0063] Map<String,Integer> map=new HashMap<>()

[0064] Step 204: Calculate the Long CRC value of devOnlyId and store it in the database.

[0065] Step 205: Query the device data of each server node using the modulo function (mod);

[0066] Step 206: Each server node obtains a different value of the same devOnlyId;

[0067] It should be noted that if a device calculates the message push policy, terminal information collection policy, and online behavior audit policy, it will have three values. The device devOnlyId is the same.

[0068] Step 207: Obtain device data of the same server node, that is, data of the same device devOnlyId.

[0069] The method of this embodiment observes that the server resources (CPU and memory occupied by the service) are small during the operation of the service program, and the deployed service node can be fully utilized.

[0070] The data acquisition device provided by the present invention is described below. The data acquisition device described below and the data acquisition method described above can be referred to in correspondence with each other. Figure 3 As shown, the present invention discloses a data acquisition device, comprising:

[0071] The acquisition module 301 is used to obtain all currently configured server nodes and current device data in the database table;

[0072] A modification module 302 is configured to add a field to a database table, wherein the field is configured to store a device identification value;

[0073] The query module 303 is configured to traverse the current device data, perform modulo calculations on the device identification value of the current device data and the number of all currently configured server nodes, and obtain the current device data of the server.

[0074] The data acquisition device of the embodiment of the present invention overcomes the shortcomings of traditional clusters in active-standby mode by adding a field for storing device identification values ​​and using a database function to obtain data. This effectively utilizes server resources and allows all deployed server nodes to run simultaneously.

[0075] In at least one embodiment of the present invention, the data type of the field is BigInt.

[0076] In at least one embodiment of the present invention, the device identification value of the current device data is modulo the number of all currently configured server nodes, and if the remainder is consistent with the node value of the current server, the current device data is obtained as the device data of the current server, including:

[0077] The device identification value of the current device data is used as the dividend, and the number of all server nodes currently configured is used as the divisor. If the remainder is consistent with the node value of the current server, the current device data is obtained as the device data of the current server.

[0078] In at least one embodiment of the present invention, the device identification value is obtained by calculating the device ID, and the device ID of each device is unique.

[0079] In at least one embodiment of the present invention, the device identification value includes a CRC value of the device ID whose data type is Long.

[0080] In at least one embodiment of the present invention, all server nodes currently configured do not distinguish between primary and backup services.

[0081] Figure 4 An example of a physical structure diagram of an electronic device is shown below. Figure 4 As shown, the electronic device may include: a processor 410, a communication interface 420, a memory 430, and a communication bus 440, wherein the processor 410, the communication interface 420, and the memory 430 communicate with each other via the communication bus 440. The processor 410 may call the logic instructions in the memory 430 to execute the data acquisition method, which includes:

[0082] This server obtains all currently configured server nodes and current device data in the database table;

[0083] The server adds a field to the database table, wherein the field is used to store the device identification value;

[0084] The server traverses the current device data, modulo the device identification value of the current device data and the number of all server nodes currently configured, and obtains the device data of the server.

[0085] In addition, the logic instructions in the above-mentioned memory 430 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when sold or used as an independent product. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0086] On the other hand, the present invention further provides a computer program product, comprising a computer program. The computer program may be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can perform the data acquisition method provided by each of the above methods, comprising:

[0087] This server obtains all currently configured server nodes and current device data in the database table;

[0088] The server adds a field to the database table, wherein the field is used to store the device identification value;

[0089] The server traverses the current device data, modulo the device identification value of the current device data and the number of all server nodes currently configured, and obtains the device data of the server.

[0090] In another aspect, the present invention further provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to perform the data acquisition method provided by the above methods, the method comprising:

[0091] This server obtains all currently configured server nodes and current device data in the database table;

[0092] The server adds a field to the database table, wherein the field is used to store the device identification value;

[0093] The server traverses the current device data, modulo the device identification value of the current device data and the number of all server nodes currently configured, and obtains the device data of the server.

[0094] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, i.e., they may be located in one location or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of the present embodiment. Persons of ordinary skill in the art will be able to understand and implement the present invention without inventive effort.

[0095] Through the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus a necessary general hardware platform, or of course, by hardware. Based on this understanding, the essence of the above technical solution or the part that contributes to the existing technology can be embodied in the form of a software product. The computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a magnetic disk, an optical disk, etc., and includes a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or certain parts of the embodiments.

[0096] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A data acquisition method, characterized in that: include: When the master-slave mode algorithm is disabled and each server node is not configured for master-slave mode, this server obtains all currently configured server nodes and the current device data in the database table. The device data includes the data calculated by the device for the message push policy, terminal information collection policy, and Internet behavior audit policy. The calculation results of the device data are stored in the same database table. The current device data in the database table is any one of the device data stored before being classified into different servers. This server adds a field in the database table, which is used to store the device identification value. The data type of the field is BigInt, where the device identification value is calculated based on the device ID and the device ID of each device is unique. The server traverses the current device data, calculates the remainder of the device identification value of the current device data and the node number of all the currently configured server nodes, and if the remainder is consistent with the node value of the server, obtains the current device data as the device data of the server; The device identification value of the current device data is modulo the number of all currently configured server nodes, and if the remainder is consistent with the node value of the current server, the current device data is obtained as the device data of the current server, including: The device identification value of the current device data is used as the dividend, and the number of nodes of all the currently configured server nodes is used as the divisor. If the remainder is consistent with the node value of this server, the current device data is obtained as the device data of this server, wherein the node value of this server refers to the serial number of this server among all server nodes.

2. The data acquisition method according to claim 1, characterized in that: The device identification value is obtained by calculating the device ID, and the device ID of each device is unique.

3. The data acquisition method according to claim 2, characterized in that: The device identification value includes a CRC value of the device ID whose data type is Long.

4. The data acquisition method according to claim 1, wherein: All server nodes in the current configuration do not distinguish between primary and backup services.

5. A data acquisition device, characterized in that: include: The acquisition module is used to obtain the current device data of all currently configured server nodes and the database table when the active-standby mode algorithm is disabled and each server node is not configured for active-standby mode. The device data includes data calculated by the device for the message push policy, terminal information collection policy, and online behavior audit policy. The calculated data of the device data is stored in the same database table. The current device data in the database table is any one of all the device data stored before being classified into different servers. A modification module is used to add a field in the database table, wherein the field is used to store a device identification value, and the data type of the field is BigInt, wherein the device identification value is obtained by calculating the device ID and the device ID of each device is unique; A query module is used to traverse the current device data, take the modulus of the device identification value of the current device data and the number of nodes of all server nodes currently configured, and if the remainder is consistent with the node value of this server, obtain the current device data as the device data of this server; taking the modulus of the device identification value of the current device data and the number of nodes of all server nodes currently configured, and if the remainder is consistent with the node value of this server, obtain the current device data as the device data of this server, including: taking the device identification value of the current device data as the dividend, taking the number of nodes of all server nodes currently configured as the divisor, and if the remainder is consistent with the node value of this server, obtaining the current device data as the device data of this server, wherein the node value of this server refers to the serial number of this server among all server nodes.

6. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the steps of the data acquisition method according to any one of claims 1 to 4 are implemented.

7. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the data acquisition method according to any one of claims 1 to 4 are implemented.

8. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the data acquisition method according to any one of claims 1 to 4 are implemented.

Images