Data collection method, device, electronic device and storage medium

By identifying data categories and using the master-slip replication and failover combined with the fault tolerance mechanism, the problem of data loss during big data collection is solved, and zero-lost data transmission from business databases and front-end pages to message middleware is realized, ensuring data Collection integrity and reliability.

CN114371962BActive Publication Date: 2024-11-26PINGAN INT SMART CITY TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202210056348.7
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-01-18
Publication Date
2024-11-26
Estimated Expiration
2042-01-18

AI Technical Summary

Technical Problem

During the big data acquisition process, data loss is prone to data loss from nginx to data acquisition layer flume and flume to kafka storage layer, resulting in incomplete data acquisition.

Method used

By identifying data categories, the master-slip replication and failover method combined with a pre-built fault tolerance mechanism is adopted to ensure that data is lost from the business database and front-end page to the message middleware, ensuring the integrity of data collection.

Benefits of technology

It realizes zero loss of data acquisition, ensures the integrity and reliability of data acquisition, and avoids data loss and downtime during transmission.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114371962B_ABST
    Figure CN114371962B_ABST
Patent Text Reader

Abstract

The present invention relates to the field of data processing, and discloses a data collection method, including: identifying the data category of data collection requirements, the data category including behavior data and business data; when the data category is business data, obtaining the data to be collected from the business database, loading the data to be collected in the business database into the data synchronization component by means of master-slave replication, and storing the data of the data synchronization component into the message middleware based on the pre-built fault-tolerant mechanism; when the data category is behavior data, obtaining the data to be collected from the front-end page, loading the data to be collected in the front-end page into the data collection layer of the log collection system by means of failover, and storing the data in the data collection layer into the cache queue; loading the data in the cache queue into the distribution layer of the log collection system, and storing the data of the distribution layer into the message middleware based on the pre-built fault-tolerant mechanism. The present invention can achieve zero loss of data collection and ensure the integrity of data collection.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of data processing, and in particular to a data acquisition method, device, electronic equipment and computer-readable storage medium. Background Art

[0002] With the growing development of big data, data collection has become an indispensable process. At present, big data collection is usually implemented based on components such as nginx+flume+kafka. Among them, the nginx component is used to receive the user behavior log reverse proxy and write it to the disk. The flume component directly reads the logs continuously written by the Nginx component from the disk directory and writes them to the kafka component. However, data loss is prone to occur from nginx to the data collection layer flume, and from flume to the kafka storage layer, resulting in incomplete data collection. Summary of the invention

[0003] The present invention provides a data collection method, device, electronic device and computer-readable storage medium, the main purpose of which is to achieve zero loss of data collection and ensure the integrity of data collection.

[0004] To achieve the above object, the present invention provides a data collection method, comprising:

[0005] Identify data categories for data collection requirements, including behavioral data and business data;

[0006] When the data category is business data, the data to be collected is obtained from the business database, and the data to be collected in the business database is loaded into the data synchronization component by using the master-slave replication method, and the data in the data synchronization component is stored in the message middleware based on the pre-built fault-tolerant mechanism;

[0007] When the data category is behavioral data, the data to be collected is obtained from the front-end page, the data to be collected in the front-end page is loaded into the data collection layer of the log collection system by means of failover, and the data in the data collection layer is stored in the cache queue;

[0008] The data in the cache queue is loaded into the distribution layer of the log collection system, and the data in the distribution layer is stored in the message middleware based on the pre-built fault tolerance mechanism.

[0009] Optionally, the step of loading the data to be collected in the business database into the data synchronization component by using a master-slave replication method includes:

[0010] Configure the main acquisition node and the backup acquisition node of the data synchronization component;

[0011] Using the main acquisition node to collect the data to be collected from the business database, and monitoring in real time whether the main acquisition node collects data from the business database abnormally;

[0012] If the main collection node does not have an abnormality, continue to collect the data to be collected, and transmit the collected data to the data synchronization component;

[0013] If the main acquisition node is abnormal, the standby acquisition node is switched to acquire the data to be acquired, and the acquired data is transmitted to the data synchronization component.

[0014] Optionally, the collecting the to-be-collected data into the business database by using the main collection node includes:

[0015] A log file of the main collection node is configured in the business database, and the data to be collected is read from the business database based on the log file.

[0016] Optionally, storing the data in the data synchronization component in the message middleware based on the pre-built fault tolerance mechanism includes:

[0017] Real-time monitoring of whether the message middleware has any abnormality during the storage process of the data of the data synchronization component;

[0018] If the message middleware is abnormal, the data of the data synchronization component is temporarily stored in the local disk until the message middleware recovers from the abnormality and then the data of the local disk is uploaded to the message middleware;

[0019] If no abnormality occurs in the message middleware, the data storage in the data synchronization component continues to be executed.

[0020] Optionally, the method of loading the data to be collected in the front-end page into the data collection layer of the log collection system by using a failover method includes:

[0021] Distribute the data to be collected in the front-end page to the data collection layer of multiple log collection systems deployed in advance by using a reverse proxy method;

[0022] Identify whether the log collection system crashes during the distribution of the data to be collected in the front-end page, and if the log collection system crashes, forward the data to be collected in the front-end page to the data collection layer of the log collection system that has not crashed.

[0023] Optionally, the method of distributing the data to be collected in the front-end page to a data collection layer in which multiple log collection systems are pre-deployed by means of a reverse proxy includes:

[0024] In response to the distribution request of the data to be collected in the front-end page, the distribution request is forwarded to the data collection layer of the pre-deployed multiple log collection systems in a load balancing manner.

[0025] Optionally, loading the data in the cache queue into the distribution layer of the log collection system includes:

[0026] In response to a data collection requirement of a distribution layer in the log collection system, pulling target data of the data collection requirement from the cache queue;

[0027] The target data is loaded into the distribution layer of the log collection system by adopting a failover method.

[0028] In order to solve the above problem, the present invention further provides a data acquisition device, the device comprising:

[0029] A data category identification module, used to identify the data category of data collection requirements, wherein the data category includes behavioral data and business data;

[0030] The data master-slave acquisition module is used to obtain the data to be collected from the business database when the data category is business data, and load the data to be collected in the business database into the data synchronization component by using the master-slave replication method, and store the data in the data synchronization component into the message middleware based on the pre-built fault-tolerant mechanism;

[0031] A data cache module, used for obtaining the data to be collected from the front-end page when the data category is behavioral data, loading the data to be collected in the front-end page into the data collection layer of the log collection system by means of failover, and storing the data in the data collection layer into a cache queue;

[0032] The data distribution and collection module is used to load the data in the cache queue into the distribution layer of the log collection system, and store the data in the distribution layer into the message middleware based on the pre-built fault tolerance mechanism.

[0033] In order to solve the above problem, the present invention further provides an electronic device, the electronic device comprising:

[0034] at least one processor; and,

[0035] a memory communicatively connected to the at least one processor; wherein,

[0036] The memory stores a computer program that can be executed by the at least one processor, and the computer program is executed by the at least one processor to implement the data collection method described above.

[0037] In order to solve the above problem, the present invention also provides a computer-readable storage medium, in which at least one computer program is stored. The at least one computer program is executed by a processor in an electronic device to implement the above data collection method.

[0038] It can be seen that the embodiment of the present invention can select different data collection methods to perform data collection by identifying the data category of the data to be collected, thereby ensuring the reliability of data collection. Secondly, when the data category is business data, the embodiment of the present invention avoids the phenomenon of downtime of the data synchronization component when collecting data from the business database through the master-slave replication method, and combines the fault-tolerant mechanism to ensure that the data to be collected in the data synchronization component is stored in the message middleware when the message middleware is down, thereby ensuring zero loss of data collection from the business database and ensuring the integrity of data collection; further, when the data category is behavior data, the embodiment of the present invention adopts the fault transfer + cache queue + fault-tolerant mechanism to ensure zero loss of data when collecting data from the front-end page through the log collection system, and when the log collection system transmits the data to be collected in the front-end page to the message middleware, thereby ensuring the integrity of data collection. Therefore, a data collection method, device, electronic device and computer-readable storage medium proposed in the embodiment of the present invention can achieve zero loss of data collection and ensure the integrity of data collection. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 A schematic diagram of a flow chart of a data collection method provided by an embodiment of the present invention;

[0040] Figure 2 A flowchart of one of the steps in the data collection method provided by one embodiment of the present invention;

[0041] Figure 3 A schematic diagram of a module of a data acquisition device provided by an embodiment of the present invention;

[0042] Figure 4 A schematic diagram of the internal structure of an electronic device for implementing a data collection method provided by an embodiment of the present invention;

[0043] The realization of the purpose, functional features and advantages of the present invention will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0044] It should be understood that the specific embodiments described herein are only used to explain the present invention, and are not used to limit the present invention.

[0045] The embodiment of the present invention provides a data collection method. The execution subject of the data collection method includes but is not limited to at least one of the electronic devices such as a server, a terminal, etc. that can be configured to execute the method provided by the embodiment of the present invention. In other words, the data collection method can be executed by software or hardware installed on a terminal device or a server device, and the software can be a blockchain platform. The server includes but is not limited to: a single server, a server cluster, a cloud server or a cloud server cluster, etc. The server can be an independent server, or it can be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (Content Delivery Network, CDN), and big data and artificial intelligence platforms.

[0046] Reference Figure 1 FIG. 1 is a flow chart of a data collection method provided by an embodiment of the present invention. In the embodiment of the present invention, the data collection method includes:

[0047] S1. Identify data categories required for data collection, where the data categories include behavioral data and business data.

[0048] In an embodiment of the present invention, the data collection demand is generated based on different user needs, such as the demand of user A is to collect user health data, and the demand of user B is to collect user identity data. Furthermore, the embodiment of the present invention identifies the data category of the data collection demand to select different data collection methods to perform data collection, thereby ensuring the reliability of data collection. The data category includes a behavior data category and a business data category. The behavior data is used to characterize the data browsed by the user on the website or APP, and the business data is used to characterize the data filled in by the user on the website or APP. For example, there is a mall order website, and the user browses electronic products on the mall order website, collects the mobile phone recommended products on the mall order website, and places an order to purchase a certain model of mobile phone. Therefore, the data generated by the user's browsing and collection behavior can be used as the behavior data in the present invention, and the data filled in by the user when placing an order can be used as the business data in the present invention.

[0049] As an embodiment of the present invention, the identification of the data category of the data collection requirement includes: obtaining the collection object of the data collection requirement, identifying the data source of the data collection requirement according to the collection object, and if the data source is a front-end page, the data category of the data collection requirement is classified as behavioral data, and if the data source is a back-end database, the data category of the data collection requirement is classified as business data. The collection object refers to the collection target of the data collection requirement, which is obtained by identifying the data field in the data collection requirement.

[0050] It should be noted that the front-end page refers to the page for human-computer interaction, which serves as a browsing carrier for users to browse page data. Therefore, the embodiment of the present invention divides the data browsed on the front-end page into behavioral data. The back-end database is used to save the data filled in by the user on the front-end page. The common MySQL database serves as a storage carrier for user interaction data. Therefore, the embodiment of the present invention divides the data stored in the front-end page into behavioral data.

[0051] S2. When the data category is business data, the data to be collected is obtained from the business database, and the data to be collected in the business database is loaded into the data synchronization component by using the master-slave replication method, and the data in the data synchronization component is stored in the message middleware based on the pre-built fault-tolerant mechanism.

[0052] In an embodiment of the present invention, the data synchronization component can be understood as data middleware, which includes a canal component for monitoring data changes in the database to realize data collection of the database, thereby ensuring synchronous backup of data in the database. In an embodiment of the present invention, the data source of the business data is the background database. Therefore, the embodiment of the present invention obtains the data to be collected from the business database, and loads the data to be collected in the business database into the data synchronization component using a master-slave replication method, so as to achieve zero-loss collection of the data to be collected in the business database. The master-slave replication method can be understood as configuring a master data collection layer and a backup data collection layer in the data synchronization component. When the master data collection layer goes down during data collection, the backup data collection layer is switched to perform data collection, thereby ensuring zero loss of data collection.

[0053] As an embodiment of the present invention, see Figure 3 As shown, the method of using the master-slave replication method to load the data to be collected in the business database into the data synchronization component includes:

[0054] S201, configuring the main acquisition node and the backup acquisition node of the data synchronization component;

[0055] S202, using the main acquisition node to collect the data to be collected from the business database, and monitoring in real time whether the main acquisition node collects data from the business database abnormally;

[0056] S203: If the main collection node does not have an abnormality, continue to collect the data to be collected, and transmit the collected data to the data synchronization component;

[0057] S204: If an abnormality occurs in the main acquisition node, switch the standby acquisition node to acquire the data to be acquired, and transmit the acquired data to the data synchronization component.

[0058] The configuration of the main acquisition node and the backup acquisition node can be implemented in Java language, the abnormal monitoring of the main acquisition node can be implemented by a monitoring script, the monitoring script can be compiled in JavaScript script language, and the switching of the backup acquisition node can be automatically switched by the configuration center zookeeper. The abnormalities of the main acquisition node include network abnormalities, server downtime, and thread crashes.

[0059] Furthermore, in an optional embodiment of the present invention, the configuration of the master acquisition node and the backup acquisition node of the data synchronization component includes: determining the number of servers of the data synchronization component in response to business needs, compiling the master server and the slave server of the data synchronization component using the Java language according to the number of servers, and using the master server and the slave server as the master acquisition node and the backup acquisition node, respectively.

[0060] Further, in an optional embodiment of the present invention, the collecting the data to be collected from the business database using the main collection node includes: configuring a log file of the main collection node in the business database, and reading the data to be collected from the business database based on the log file. The log file includes a binary log file, which is used to monitor the data change status of the business database, such as data addition, update, etc.

[0061] Furthermore, in an embodiment of the present invention, the message middleware is a high-throughput distributed publish-subscribe message system, which includes a kafka component for real-time processing and storage of all action flow data of consumers on the website. In an embodiment of the present invention, the data to be collected in the business data is user data generated in real time. Therefore, the embodiment of the present invention stores the real-time data generated by the user in the message middleware to ensure real-time consumption processing of the data to be collected.

[0062] It should be noted that in the process of data transmission from the data synchronization component to the message middleware, data loss may occur due to network and other reasons. Therefore, the embodiment of the present invention adopts a pre-built fault-tolerant mechanism to store the data in the data synchronization component in the message middleware to ensure zero loss of data collection. Among them, the fault-tolerant mechanism refers to the process of data transmission from the data synchronization component to the message middleware, real-time monitoring of whether the message middleware has an abnormality, and when the message middleware has an abnormality, the data is stored in the local disk to ensure the continued transmission of data when the message middleware recovers from the abnormality.

[0063] As an embodiment of the present invention, the data in the data synchronization component is stored in the message middleware based on a pre-built fault-tolerant mechanism, including: real-time monitoring of whether the message middleware has any abnormality during the storage process of the data of the data synchronization component; if the message middleware has an abnormality, the data of the data synchronization component is temporarily stored in the local disk until the message middleware recovers from the abnormality and then the data on the local disk is uploaded to the message middleware; if the message middleware has no abnormality, the data storage in the data synchronization component continues. Optionally, the abnormality monitoring of the message middleware can be implemented through the above-mentioned monitoring script.

[0064] S3. When the data category is behavioral data, the data to be collected is obtained from the front-end page, and the data to be collected in the front-end page is loaded into the data collection layer of the log collection system by means of failover, and the data in the data collection layer is stored in the cache queue.

[0065] It should be understood that when the data category is behavioral data, the data source representing the data to be collected is the front-end page, which performs data transmission through the HTTP protocol. Since a large amount of data will be generated through the front-end page, in order to ensure that the subsequent log collection system can fully receive the large amount of data from the front-end page, the embodiment of the present invention obtains the data to be collected from the front-end page, and uses a failover method to load the data to be collected in the front-end page into the data collection layer of the log collection system, so as to achieve zero-loss collection of the data to be collected.

[0066] The log collection system (flume) refers to a highly available, highly reliable, distributed system for massive log collection, aggregation and transmission. The data collection layer of the log collection system can be understood as a data receiver, which is used to collect data generated by the front-end page.

[0067] As an embodiment of the present invention, the method of using failover to load the data to be collected in the front-end page into the data collection layer of the log collection system includes: using a reverse proxy method to distribute the data to be collected in the front-end page to the data collection layer of multiple log collection systems deployed in advance, identifying whether the log collection system crashes during the distribution of the data to be collected in the front-end page, and when the log collection system crashes, forwarding the data to be collected in the front-end page to the data collection layer of the log collection system that has not crashed.

[0068] In an optional embodiment, the use of a reverse proxy to distribute the data to be collected in the front-end page to the data collection layer of the pre-deployed multiple log collection systems includes: in response to the distribution request of the data to be collected in the front-end page, forwarding the distribution request to the data collection layer of the pre-deployed multiple log collection systems in a load balancing manner. The distribution request can be triggered by a front-end LVS (Linux Virtual Server), and the load balancing method can be implemented by Nginx.

[0069] Furthermore, the embodiment of the present invention stores the data in the data collection layer in a cache queue to realize asynchronous processing of the data in the data collection layer and improve the processing efficiency of the data in the data collection layer. The cache queue can be a storage container of the log collection system, such as a channel.

[0070] S4. Load the data in the cache queue into the distribution layer of the log collection system, and store the data in the distribution layer into the message middleware based on the pre-built fault tolerance mechanism.

[0071] The embodiment of the present invention realizes asynchronous collection of the data to be collected and improves data collection efficiency by loading the data in the cache queue into the distribution layer of the log collection system. Optionally, the loading of the data in the cache queue into the distribution layer of the log collection system includes: in response to the data collection demand of the distribution layer in the log collection system, pulling the target data of the data collection demand from the cache queue, and loading the target data into the distribution layer of the log collection system by means of failover.

[0072] Among them, the data collection requirements are generated based on different business scenarios. For example, for medical scenarios, the data collection requirements may be to collect medical data, and for financial scenarios, the data collection requirements may be to collect financial data. The target data can be pulled by matching the collection objects in the data collection requirements. The failover method can refer to the description of S3 above, which is used to ensure zero data loss during the data loading process.

[0073] Furthermore, the present invention implements the storage of the data in the distribution layer in the message middleware based on the pre-built fault-tolerant mechanism, so as to ensure zero data loss of the data to be collected by the message middleware in the collection front-end page, and to ensure the comprehensiveness of the stored data. The pre-built fault-tolerant mechanism can refer to the description of S2 above, and will not be further elaborated here.

[0074] It can be seen that the embodiment of the present invention can select different data collection methods to perform data collection by identifying the data category of the data to be collected, thereby ensuring the reliability of data collection. Secondly, when the data category is business data, the embodiment of the present invention avoids the phenomenon of downtime of the data synchronization component when collecting data from the business database through the master-slave replication method, and combines the fault-tolerant mechanism to ensure that the data to be collected in the data synchronization component is stored in the message middleware when the message middleware is down, thereby ensuring zero loss of data collection from the business database and ensuring the integrity of data collection; further, when the data category is behavior data, the embodiment of the present invention adopts the fault transfer + cache queue + fault-tolerant mechanism to ensure zero loss of data when collecting data from the front-end page through the log collection system and when the log collection system transmits the data to be collected in the front-end page to the message middleware, thereby ensuring the integrity of data collection. Therefore, a data collection proposed in the embodiment of the present invention can achieve zero loss of data collection and ensure the integrity of data collection.

[0075] like Figure 3 The figure shows a functional module diagram of the data acquisition device of the present invention.

[0076] The data acquisition device 100 of the present invention can be installed in an electronic device. According to the functions to be implemented, the data acquisition device can include a data category identification module 101, a data master and backup acquisition module 102, a data cache module 103, and a data distribution acquisition module 104. The module of the present invention can also be called a unit, which refers to a series of computer program segments that can be executed by a processor of an electronic device and can complete fixed functions, which are stored in the memory of the electronic device.

[0077] In this embodiment, the functions of each module / unit are as follows:

[0078] The data category identification module 101 is used to identify the data category of the data collection requirement, and the data category includes behavioral data and business data;

[0079] The data master-slave acquisition module 102 is used to obtain the data to be collected from the business database when the data category is business data, and load the data to be collected in the business database into the data synchronization component by using the master-slave replication method, and store the data in the data synchronization component into the message middleware based on the pre-built fault-tolerant mechanism;

[0080] The data cache module 103 is used to obtain the data to be collected from the front-end page when the data category is behavioral data, load the data to be collected in the front-end page into the data collection layer of the log collection system by means of failover, and store the data in the data collection layer into the cache queue;

[0081] The data distribution and collection module 104 is used to load the data in the cache queue into the distribution layer of the log collection system, and store the data in the distribution layer into the message middleware based on the pre-built fault tolerance mechanism.

[0082] In detail, each module in the data acquisition device 100 in the embodiment of the present invention is used in the same manner as described above. Figure 1 The data collection method described in the present invention has the same technical means and can produce the same technical effects, so I will not go into details here.

[0083] like Figure 4 FIG. 1 is a schematic diagram of the structure of an electronic device 1 for implementing the data acquisition method of the present invention.

[0084] The electronic device 1 may include a processor 10 , a memory 11 , a communication bus 12 , and a communication interface 13 , and may also include a computer program stored in the memory 11 and executable on the processor 10 , such as a data acquisition program.

[0085] In some embodiments, the processor 10 may be composed of an integrated circuit, for example, a single packaged integrated circuit, or a plurality of packaged integrated circuits with the same or different functions, including one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and combinations of various control chips. The processor 10 is the control core (ControlUnit) of the electronic device 1, and uses various interfaces and lines to connect various components of the entire electronic device 1, and executes various functions and processes data of the electronic device 1 by running or executing programs or modules (such as executing data acquisition programs, etc.) stored in the memory 11, and calling data stored in the memory 11.

[0086] The memory 11 includes at least one type of readable storage medium, and the readable storage medium includes flash memory, mobile hard disk, multimedia card, card-type memory (for example: SD or DX memory, etc.), magnetic memory, disk, optical disk, etc. In some embodiments, the memory 11 can be an internal storage unit of the electronic device 1, such as a mobile hard disk of the electronic device 1. In other embodiments, the memory 11 can also be an external storage device of the electronic device 1, such as a plug-in mobile hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (SecureDigital, SD) card, a flash card (Flash Card), etc. equipped on the electronic device 1. Further, the memory 11 can also include both an internal storage unit of the electronic device 1 and an external storage device. The memory 11 can not only be used to store application software and various types of data installed in the electronic device 1, such as the code of the data acquisition program, etc., but also can be used to temporarily store data that has been output or is to be output.

[0087] The communication bus 12 may be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. The bus is configured to realize connection and communication between the memory 11 and at least one processor 10, etc.

[0088] The communication interface 13 is used for communication between the above-mentioned electronic device 1 and other devices, including a network interface and an employee interface. Optionally, the network interface may include a wired interface and / or a wireless interface (such as a WI-FI interface, a Bluetooth interface, etc.), which is generally used to establish a communication connection between the electronic device 1 and other electronic devices 1. The employee interface may be a display (Display), an input unit (such as a keyboard (Keyboard)), and optionally, the employee interface may also be a standard wired interface, a wireless interface. Optionally, in some embodiments, the display may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, and an OLED (Organic Light-Emitting Diode) touch device, etc. Among them, the display may also be appropriately referred to as a display screen or a display unit, which is used to display information processed in the electronic device 1 and to display a visual employee interface.

[0089] Figure 4 Only the electronic device 1 having components is shown, and those skilled in the art can understand that Figure 4The structure shown does not constitute a limitation on the electronic device 1, and may include fewer or more components than shown in the figure, or combine certain components, or arrange the components differently.

[0090] For example, although not shown, the electronic device 1 may also include a power source (such as a battery) for supplying power to each component. Preferably, the power source may be logically connected to the at least one processor 10 through a power management device, so that the power management device can realize functions such as charging management, discharging management, and power consumption management. The power source may also include any components such as one or more DC or AC power sources, recharging devices, power failure detection circuits, power converters or inverters, power status indicators, etc. The electronic device 1 may also include a variety of sensors, Bluetooth modules, Wi-Fi modules, etc., which will not be repeated here.

[0091] It should be understood that the embodiment is for illustrative purposes only and the scope of the patent invention is not limited to this structure.

[0092] The data acquisition program stored in the memory 11 of the electronic device 1 is a combination of multiple computer programs. When running in the processor 10, it can achieve:

[0093] Identify data categories for data collection requirements, including behavioral data and business data;

[0094] When the data category is business data, the data to be collected is obtained from the business database, and the data to be collected in the business database is loaded into the data synchronization component by using the master-slave replication method, and the data in the data synchronization component is stored in the message middleware based on the pre-built fault-tolerant mechanism;

[0095] When the data category is behavioral data, the data to be collected is obtained from the front-end page, the data to be collected in the front-end page is loaded into the data collection layer of the log collection system by means of failover, and the data in the data collection layer is stored in the cache queue;

[0096] The data in the cache queue is loaded into the distribution layer of the log collection system, and the data in the distribution layer is stored in the message middleware based on the pre-built fault tolerance mechanism.

[0097] Specifically, the specific implementation method of the processor 10 for the above computer program can refer to Figure 1 The description of the relevant steps in the corresponding embodiments will not be repeated here.

[0098] Furthermore, if the module / unit integrated in the electronic device 1 is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a non-volatile computer-readable storage medium. The computer-readable storage medium can be volatile or non-volatile. For example, the computer-readable medium can include: any entity or device capable of carrying the computer program code, a recording medium, a USB flash drive, a mobile hard disk, a magnetic disk, an optical disk, a computer memory, and a read-only memory (ROM).

[0099] The present invention further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor of the electronic device 1, the computer program can implement:

[0100] Identify data categories for data collection requirements, including behavioral data and business data;

[0101] When the data category is business data, the data to be collected is obtained from the business database, and the data to be collected in the business database is loaded into the data synchronization component by using the master-slave replication method, and the data in the data synchronization component is stored in the message middleware based on the pre-built fault-tolerant mechanism;

[0102] When the data category is behavioral data, the data to be collected is obtained from the front-end page, the data to be collected in the front-end page is loaded into the data collection layer of the log collection system by means of failover, and the data in the data collection layer is stored in the cache queue;

[0103] The data in the cache queue is loaded into the distribution layer of the log collection system, and the data in the distribution layer is stored in the message middleware based on the pre-built fault tolerance mechanism.

[0104] In the several embodiments provided by the present invention, it should be understood that the disclosed devices, apparatuses and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of the modules is only a logical function division, and there may be other division methods in actual implementation.

[0105] The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0106] In addition, each functional module in each embodiment of the present invention may be integrated into one processing unit, each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of hardware plus software functional modules.

[0107] It is obvious to those skilled in the art that the present invention is not limited to the details of the above exemplary embodiments, and that the present invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the present invention.

[0108] Therefore, no matter from which point of view, the embodiments should be regarded as illustrative and non-restrictive, and the scope of the present invention is limited by the appended claims rather than the above description, so it is intended that all changes falling within the meaning and scope of the equivalent elements of the claims are included in the present invention. Any attached figure mark in the claims should not be regarded as limiting the claims involved.

[0109] The blockchain referred to in this invention is a new application mode of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanism, encryption algorithm, etc. Blockchain is essentially a decentralized database, a string of data blocks generated by cryptographic methods. Each data block contains a batch of network transaction information, which is used to verify the validity of its information (anti-counterfeiting) and generate the next block. Blockchain can include the underlying blockchain platform, platform product service layer, and application service layer.

[0110] The embodiments of the present invention can acquire and process relevant data based on artificial intelligence technology. Artificial Intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0111] In addition, it is clear that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. Multiple units or devices stated in the system claim can also be implemented by one unit or device through software or hardware. The second and other words are used to indicate names, but not to indicate any particular order.

[0112] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solution of the present invention can be modified or replaced by equivalents without departing from the spirit and scope of the technical solution of the present invention.

Claims

1. A data collection method, characterized in that: The method comprises: Identify data categories for data collection requirements, including behavioral data and business data; When the data category is business data, the data to be collected is obtained from the business database, and the data to be collected in the business database is loaded into the data synchronization component by using the master-slave replication method, and the data in the data synchronization component is stored in the message middleware based on the pre-built fault-tolerant mechanism; When the data category is behavioral data, the data to be collected is obtained from the front-end page, the data to be collected in the front-end page is loaded into the data collection layer of the log collection system by means of failover, and the data in the data collection layer is stored in the cache queue; Loading the data in the cache queue into the distribution layer of the log collection system, and storing the data in the distribution layer into the message middleware based on the pre-built fault tolerance mechanism; The method of using a master-slave replication method to load the data to be collected in the business database into the data synchronization component includes: configuring a master collection node and a backup collection node of the data synchronization component; using the master collection node to collect the data to be collected from the business database, and monitoring in real time whether there is any abnormality in the data collection of the business database by the master collection node; if there is no abnormality in the master collection node, continuing to collect the data to be collected, and transmitting the collected data to the data synchronization component; if there is an abnormality in the master collection node, switching the backup collection node to collect the data to be collected, and transmitting the collected data to the data synchronization component; The using the main acquisition node to collect the data to be collected into the business database includes: configuring a log file of the main acquisition node in the business database, and reading the data to be collected from the business database based on the log file; The pre-built fault-tolerant mechanism based on which the data in the data synchronization component is stored in the message middleware includes: real-time monitoring of whether the message middleware has any abnormality during the storage process of the data of the data synchronization component; if the message middleware has any abnormality, the data of the data synchronization component is temporarily stored in the local disk until the message middleware recovers from the abnormality and then the data on the local disk is uploaded to the message middleware; if the message middleware has no abnormality, the data storage in the data synchronization component continues to be executed.

2. The data collection method according to claim 1, characterized in that: The method of loading the data to be collected in the front-end page into the data collection layer of the log collection system by adopting a failover method includes: Distribute the data to be collected in the front-end page to the data collection layer of multiple log collection systems pre-deployed by using a reverse proxy method; Identify whether the log collection system crashes during the distribution of the data to be collected in the front-end page, and if the log collection system crashes, forward the data to be collected in the front-end page to the data collection layer of the log collection system that has not crashed.

3. The data collection method according to claim 2, characterized in that: The method of using a reverse proxy to distribute the data to be collected in the front-end page to the data collection layer of the pre-deployed multiple log collection systems includes: In response to the distribution request of the data to be collected in the front-end page, the distribution request is forwarded to the data collection layer of the pre-deployed multiple log collection systems in a load balancing manner.

4. The data collection method according to claim 1, characterized in that: The step of loading the data in the cache queue into the distribution layer of the log collection system includes: In response to a data collection requirement of a distribution layer in the log collection system, pulling target data of the data collection requirement from the cache queue; The target data is loaded into the distribution layer of the log collection system by adopting a failover method.

5. A data acquisition device, used to implement the data acquisition method according to any one of claims 1 to 4, characterized in that: The device comprises: A data category identification module, used to identify the data category of data collection requirements, wherein the data category includes behavioral data and business data; The data master-slave acquisition module is used to obtain the data to be collected from the business database when the data category is business data, and load the data to be collected in the business database into the data synchronization component by using the master-slave replication method, and store the data in the data synchronization component into the message middleware based on the pre-built fault-tolerant mechanism; A data cache module, used for obtaining the data to be collected from the front-end page when the data category is behavioral data, loading the data to be collected in the front-end page into the data collection layer of the log collection system by means of failover, and storing the data in the data collection layer into a cache queue; The data distribution and collection module is used to load the data in the cache queue into the distribution layer of the log collection system, and store the data in the distribution layer into the message middleware based on the pre-built fault tolerance mechanism.

6. An electronic device, characterized in that: The electronic device comprises: at least one processor; and, a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can perform the data collection method according to any one of claims 1 to 4.

7. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the data collection method according to any one of claims 1 to 4 is implemented.