Data acquisition method and device, equipment, storage medium and product

Abstract

The invention provides a data acquisition method, device and equipment, a storage medium and a product, and relates to the technical field of computers, the method comprises the following steps: configuring a development environment required by data acquisition; in the development environment, creating a data acquisition micro-service item; configuring a plurality of data acquisition modules of the data acquisition micro-service item based on the type of the to-be-acquired data; each data acquisition module is used for acquiring one type of data; and starting the data acquisition micro-service item, and respectively acquiring various types of to-be-acquired data based on the plurality of data acquisition modules. By means of the mode, unified collection of different types of data is achieved, and the availability and practicability of the data can be effectively improved.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a data acquisition method, device, equipment, storage medium and product. Background Art

[0002] In an era of continuous development of informatization and widespread application of the Internet, data collection has very important applications in various fields. In the digital age, distributed systems are widely used, and the data that enterprises need to collect is often scattered in multiple business systems and stored and managed by different business systems.

[0003] Due to the diversity and complexity of data types in different business systems, enterprises often find it difficult to uniformly collect and manage different types of data in different business systems, resulting in low data availability and practicality. Therefore, there is an urgent need for a data collection method that can uniformly collect different types of data to improve data availability and practicality. Summary of the invention

[0004] The present invention provides a data collection method, device, equipment, storage medium and product to solve the defects in the prior art of lacking a data collection method that can uniformly collect different types of data, and low data availability and practicality.

[0005] The present invention provides a data collection method, comprising: configuring a development environment required for data collection; creating a data collection microservice project in the development environment; configuring multiple data collection modules of the data collection microservice project based on the type of data to be collected; each data collection module is used to collect one type of data; starting the data collection microservice project, and collecting multiple types of data to be collected based on multiple data collection modules.

[0006] According to a data collection method provided by the present invention, multiple data collection modules include a network device collection module, a security device collection module and an audio and video collection module; the network device collection module is used to collect network device information; the network device information includes one or more of network device asset information, network device performance information, network link information and network device port performance information; the security device collection module is used to collect security information of terminal devices; the security information includes one or more of security device asset information, security assessment information and security system assessment item information; the audio and video collection module is used to collect audio and video information; the audio and video information includes one or more of the operating status information of the audio and video equipment, audio and video alarm information, audio and video overall satisfaction information, video, voice, display and control equipment and service static information.

[0007] According to a data acquisition method provided by the present invention, after collecting multiple types of data to be collected respectively based on multiple data acquisition modules, it also includes: determining the load of each data acquisition module; based on the load of each data acquisition module, judging whether there is an overloaded data acquisition module with a load greater than or equal to a preset threshold; if there is an overloaded data acquisition module with a load greater than or equal to the preset threshold, configuring a new data acquisition module according to the type of data to be collected collected by the overloaded data acquisition module; the new data acquisition module and the overloaded data acquisition module are used to collect the same type of data to be collected.

[0008] According to a data acquisition method provided by the present invention, after collecting multiple types of data to be collected respectively based on multiple data acquisition modules, the method also includes: determining the fault status of each data acquisition module; judging whether there is a faulty data acquisition module based on the fault status of each data acquisition module; if there is a faulty data acquisition module, suspending data acquisition of the faulty data acquisition module, and continuing to collect the corresponding type of data to be collected through the non-faulty data acquisition module.

[0009] According to a data acquisition method provided by the present invention, after collecting multiple types of data to be collected respectively based on multiple data acquisition modules, it also includes: performing data cleaning processing and data conversion processing on the multiple types of data to be collected respectively to obtain multiple types of data to be stored; the multiple types of data to be stored have the same data format; and storing the multiple types of data to be stored in a time series database.

[0010] According to a data collection method provided by the present invention, before starting a data collection microservice project, it also includes: loading a configuration file of the data collection microservice project; wherein the configuration file is managed by a configuration management platform.

[0011] The present invention also provides a data acquisition device, including: a configuration module, used to configure the development environment required for data acquisition; a creation module, used to create a data acquisition microservice project in the development environment; a development module, used to configure multiple data acquisition modules of the data acquisition microservice project based on the type of data to be collected; each data acquisition module is used to collect one type of data; and a collection module, used to start the data acquisition microservice project, and based on multiple data acquisition modules, respectively collect multiple types of data to be collected.

[0012] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements any of the above-mentioned data collection methods when executing the computer program.

[0013] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, and the computer program, when executed by a processor, implements any of the above-mentioned data acquisition methods.

[0014] The present invention also provides a computer program product, including a computer program, and when the computer program is executed by a processor, any of the above-mentioned data collection methods is implemented.

[0015] The data collection method, apparatus, equipment, storage medium and product provided by the present invention create a data collection microservice project in the development environment required for data collection, and configure multiple data collection modules of the data collection microservice project based on the type of data to be collected. Each data collection module is used to collect one type of data. After starting the data collection microservice project, multiple types of data to be collected can be collected separately based on multiple data collection modules, thereby realizing unified collection of different types of data, which can effectively improve the availability and practicality of data. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the technical solutions in the present invention or the prior art, a brief introduction will be given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0017] Figure 1 This is one of the flow charts of the data acquisition method provided by the present invention.

[0018] Figure 2 This is the second flow chart of the data collection method provided by the present invention.

[0019] Figure 3 It is a service architecture diagram of the microservice project provided by the present invention.

[0020] Figure 4 It is an architecture diagram of the VictoriaMetrics database provided by the present invention.

[0021] Figure 5 This is the third flow chart of the data acquisition method provided by the present invention.

[0022] Figure 6 It is a schematic diagram of the configuration flow of NACOS provided by the present invention.

[0023] Figure 7 It is a structural schematic diagram of the data acquisition device provided by the present invention.

[0024] Figure 8 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0025] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0026] See also Figure 1 and Figure 2 , Figure 1 It is one of the flow charts of the data collection method provided by the present invention. Figure 2 This is the second flow chart of the data collection method provided by the present invention. Figure 1 As shown, in this embodiment, the data collection method includes steps S110 to S140, and each step is specifically as follows: S110: Configure the development environment required for data collection.

[0027] Specifically, Figure 2 As shown, configure the development environment required for data collection and ensure that development tools such as JDK, MAVEN (project management tool), and IDE (integrated development environment) are configured.

[0028] S120: Create a data collection microservice project in the development environment.

[0029] After the development environment required for data collection is configured, you can create a SpringBoot project in the development environment as a data collection microservice project and configure the necessary dependencies for the data collection microservice project.

[0030] S130: Based on the type of data to be collected, configure multiple data collection modules of the data collection microservice project.

[0031] Each data collection module is used to collect one type of data.

[0032] Furthermore, according to the type of data to be collected (ie, the collection object), multiple data collection modules of the data collection microservice project are developed and configured, and the data collection modules include but are not limited to a network device collection module, a security device collection module, and an audio and video collection module.

[0033] S140: Start a data collection microservice project, and collect various types of data to be collected based on multiple data collection modules.

[0034] The data collection method provided in this embodiment creates a data collection microservice project in the development environment required for data collection, and configures multiple data collection modules of the data collection microservice project based on the type of data to be collected. Each data collection module is used to collect one type of data. After starting the data collection microservice project, multiple types of data to be collected can be collected separately based on multiple data collection modules, thereby realizing unified collection of different types of data, which can effectively improve the availability and practicality of data.

[0035] In some embodiments, multiple data acquisition modules include a network device acquisition module, a security device acquisition module and an audio and video acquisition module; the network device acquisition module is used to collect network device information; the network device information includes one or more of network device asset information, network device performance information, network link information and network device port performance information; the security device acquisition module is used to collect security information of terminal devices; security information includes one or more of security device asset information, security assessment information and security system assessment item information; the audio and video acquisition module is used to collect audio and video information; the audio and video information includes one or more of the operating status information of the audio and video equipment, audio and video alarm information, audio and video overall satisfaction information, video, voice, display and control equipment and service static information.

[0036] See also Figure 3 , Figure 3 It is a service architecture diagram of the microservice project provided by the present invention.

[0037] like Figure 3 As shown in the figure, the data collection microservice project fully utilizes the advantage of the microservice architecture in supporting multi-module expansion. It collects various types of data by setting up multiple data collection modules, such as network equipment collection module, security equipment collection module, audio and video collection module and other modules.

[0038] Specifically, the network device collection module is used to collect network device information; the network device information includes one or more of network device asset information, network device performance information, network link information and network device port performance information.

[0039] Among them, network equipment asset information includes the IP address of the network equipment, the equipment manufacturer of the network equipment, the model of the network equipment, etc.; network equipment performance information includes CPU utilization, memory utilization, packet loss rate, delay, etc.; network link information includes link information between different network devices and between network devices and servers, and collection indicators include latency, jitter, packet loss rate, link status, etc.; network equipment port performance information includes port input rate, port output rate, bandwidth utilization, packet loss rate, number of error packets, port input traffic, port output traffic, etc.

[0040] The security equipment collection module can obtain the security information of each terminal device through the security system, collect the online status of each terminal and various security events; the security information includes one or more of the security equipment asset information, security assessment information and security system assessment item information.

[0041] Among them, security equipment asset information includes equipment type, equipment ID, equipment IP address, etc.; security assessment information includes the performance and risk analysis of the security system; security system assessment item information includes assessment category, assessment reason, assessment result, etc.

[0042] The audio and video acquisition module is used to collect audio and video information; the audio and video information includes one or more of the operating status information of the audio and video equipment, audio and video alarm information, audio and video overall satisfaction information, video, voice, display and control equipment and service static information.

[0043] Among them, the operating status information of audio and video equipment generally includes offline status and online status; audio and video alarm information includes alarm level, alarm description, device name, alarm time, etc.; video, voice, display and control equipment and service static information includes manufacturer, equipment type, category, name, equipment ID, equipment IP address, etc.

[0044] The data collection method provided in this embodiment adopts a microservice architecture to collect multiple types of data through multiple data collection modules. The advantages of this microservice architecture are high flexibility and maintainability: Since each data collection module is relatively independent, developers can choose the most suitable technology according to the needs of specific services and flexibly set different modules, which can effectively improve development efficiency and data collection efficiency, and adapt to the collection of different types of data.

[0045] In some embodiments, after collecting multiple types of data to be collected based on multiple data collection modules, it also includes: determining the load of each data collection module; based on the load of each data collection module, judging whether there is an overloaded data collection module with a load greater than or equal to a preset threshold; if there is an overloaded data collection module with a load greater than or equal to the preset threshold, configuring a new data collection module according to the type of data to be collected collected by the overloaded data collection module; the new data collection module and the overloaded data collection module are used to collect the same type of data to be collected.

[0046] In this embodiment, a microservice architecture is adopted to collect multiple types of data through multiple data collection modules. Different from conventional data collection methods, the microservice architecture is highly scalable and supports independent expansion. Each microservice (i.e., a data collection module) can be independently expanded according to its own load and demand. That is, if the load of a service increases, more instances can be added to the service alone without affecting other services.

[0047] Specifically, when multiple types of data are collected respectively through multiple data acquisition modules, the load of each data acquisition module can be monitored and determined in real time, and based on the load of the data acquisition module, it is determined whether the load of the data acquisition module is greater than or equal to a preset threshold.

[0048] If the load of the data acquisition module is greater than or equal to the preset threshold, the data acquisition module is considered to be an overloaded data acquisition module. At this time, a new data acquisition module can be configured according to the type of data to be collected by the overloaded data acquisition module. The new data acquisition module and the overloaded data acquisition module can be used to collect the same type of data to be collected.

[0049] The data collection method provided in this embodiment adopts a microservice architecture to collect various types of data through multiple data collection modules, and supports on-demand expansion. It can specifically expand the services that need to be expanded according to business needs, avoiding waste of resources due to excessive expansion of the entire application.

[0050] In some embodiments, after collecting multiple types of data to be collected based on multiple data acquisition modules, it also includes: determining the fault status of each data acquisition module; judging whether there is a faulty data acquisition module based on the fault status of each data acquisition module; if there is a faulty data acquisition module, pausing data acquisition of the faulty data acquisition module, and continuing to collect the corresponding type of data to be collected through the non-faulty data acquisition module.

[0051] In this embodiment, a microservice architecture is adopted to collect multiple types of data through multiple data collection modules. Different from the conventional data collection method, the microservice architecture can achieve fault isolation: if a microservice fails, it will only affect the service itself, but will not affect other microservices in the entire application. Other microservices can still continue to operate normally, thereby improving the overall availability of the system.

[0052] Specifically, when multiple types of data are collected respectively through multiple data collection modules, for each data collection module, the fault state of the data collection module can be monitored and determined in real time, and based on the fault state of the data collection module, it can be determined whether the data collection module is faulty.

[0053] If the data acquisition module has a fault, the data acquisition of the data acquisition module will be suspended, and other data acquisition modules that do not have a fault will continue to provide services and collect the corresponding type of data to be collected.

[0054] The data collection method provided in this embodiment realizes fault isolation of different data collection modules through a microservice architecture. When a failure occurs in some microservices, it will only affect the failed service itself without affecting other microservices. Other microservices can still continue to operate normally, thereby improving the overall availability of the system.

[0055] In some embodiments, after collecting multiple types of data to be collected based on multiple data collection modules, it also includes: performing data cleaning processing and data conversion processing on the multiple types of data to be collected to obtain multiple types of data to be stored; the multiple types of data to be stored have the same data format; and storing the multiple types of data to be stored in a time series database.

[0056] It is understandable that the data formats of different types of data are not unified, which may affect the subsequent storage and query of data. Therefore, after obtaining multiple types of data to be collected, data cleaning processing and data conversion processing can be performed on the multiple types of data to be collected respectively to obtain multiple types of data to be stored. The multiple types of data to be stored have the same data format.

[0057] For example, the data formats of the collected network equipment alarm information, audio and video alarm information, security alarm information, etc. are not unified. The above data can be cleaned and converted to convert them into a unified data format for storage.

[0058] Furthermore, after obtaining multiple types of data to be stored, the multiple types of data to be stored can be stored in a time series database.

[0059] Among them, the time series database is VictoriaMetrics.

[0060] It should be noted that with the rapid development of the Internet, data storage has become a key issue. Since the data in existing distributed systems has the characteristics of high concurrency, high speed and massive volume, traditional databases often find it difficult to meet the data storage needs. Therefore, time series databases have emerged. Time series databases need to solve two main problems when storing data: reducing storage costs and ensuring query performance. One way to reduce storage costs is to compress data and use the characteristics of the data itself to compress the data volume.

[0061] Specifically, in this embodiment, VictoriaMetrics is used as a time series database. VictoriaMetrics is a time series database that supports high-throughput data ingestion and query, and provides a rich API interface (application programming interface) that is compatible with Prometheus. VictoriaMetrics uses a customized and efficient data structure VMStorage, which manages data in layers between memory and disk to ensure the read and write performance of the database. At the same time, VictoriaMetrics has a unique compression algorithm that can improve storage efficiency and reduce the system's demand for hardware resources. VictoriaMetrics supports vectorized query technology, which can process large amounts of data in parallel to achieve rapid response. Its instant aggregation capability means that low-latency queries can be achieved even on large-scale data sets. The single-instance VictoriaMetrics already has excellent performance, but for larger-scale needs, its cluster version VMinsert can be deployed for distributed data ingestion to improve query load capacity.

[0062] Optionally, data is inserted into the database through the API interface provided by VictoriaMetrics.

[0063] See also Figure 4 , Figure 4 It is an architecture diagram of the VictoriaMetrics database provided by the present invention.

[0064] like Figure 4As shown in the figure, Grafana (an open source data visualization and monitoring platform) can interact with the VictoriaMetrics cluster (VictoriaMetricscluster) through vmselect. VictoriaMetrics is compatible with Prometheus. Other clients using the Prometheus API can also interact with the VictoriaMetrics cluster through vmselect. A load balancer can be used to distribute requests between the client and the VictoriaMetrics cluster to ensure high availability and load balancing of the system. vmselect is a stateless service that obtains and merges data from vmstorage during queries. vmselect fully supports PromQL and can be used as a Prometheus data source in Grafana. vmstorage is a stateful service for storing time series data. It is responsible for persistent storage of data. Vminsert is a stateless service for distributing time series data to available vmstorage nodes. Multiple Prometheus instances can write data to the VictoriaMetrics cluster. Support for writing InfluxDB line protocol data to the VictoriaMetrics cluster. Support writing Graphite's plain text protocol data to VictoriaMetrics cluster. Support writing OpenTSDB's put protocol data to VictoriaMetrics cluster.

[0065] vmselect and vminsert are stateless services, which means they do not store state information and can be horizontally scaled to handle more requests.

[0066] vmstorage is a stateful service because it is responsible for storing time series data.

[0067] The entire architecture of VictoriaMetrics is designed to provide a high-performance, scalable monitoring solution that can handle large amounts of time series data and support multiple data writing protocols, while improving system availability and responsiveness by using a load balancer.

[0068] In some embodiments, before starting the data collection microservice project, it also includes: loading a configuration file of the data collection microservice project; wherein the configuration file is managed by a configuration management platform.

[0069] It should be noted that in the microservice architecture, when a microservice project is split from a single application into multiple service nodes (such as multiple data acquisition modules) on a distributed system, the configuration files of the microservice project must also be migrated (split), which will cause the configuration files to be dispersed. At the same time, in order to ensure the normal operation of each microservice, the dispersed configuration files also contain a lot of redundant configuration information.

[0070] Therefore, in order to further optimize the operation and management efficiency of the data collection microservice project, the configuration files can be managed through a unified configuration management platform.

[0071] In this embodiment, NACOS is selected as the configuration management platform. NACOS is used to uniformly configure and manage the configurations involved in the system, and each microservice (data collection module) does not need to manage the configuration by itself.

[0072] Among them, NACOS can provide key / value storage for storing configuration and other metadata, and provide server-side and client-side support for externalized configuration in distributed systems.

[0073] See also Figure 5 and Figure 6 , Figure 5 This is the third flow chart of the data collection method provided by the present invention. Figure 6 It is a schematic diagram of the configuration flow of NACOS provided by the present invention.

[0074] like Figure 5 As shown, before starting the data collection microservice project, you can pre-configure the dependencies and references of NACOS, and use the configured NACOS to uniformly configure and manage the configuration files involved in the network device collection module, security device collection module, and audio and video collection module.

[0075] like Figure 6 As shown in the figure, after completing the configuration of NACOS, when the data collection microservice project is started, the NACOS address and configuration file id can be obtained through the file bootstrap.yml (the file bootstrap.yml is usually used to configure the externalized configuration of the application); the application uses the configuration file id as an identifier to obtain the corresponding configuration file from the NACOS server; each microservice can read the local configuration file application.yml, which contains the default configuration of the application; after merging the local configuration file application.yml with the configuration file obtained from the NACOS server, a Spring container can be created, which will load all beans (Spring-managed objects), and each microservice starts running.

[0076] The data collection method provided in this embodiment involves data collection of different business types, and can collect and store different types of data such as network equipment, audio and video, and security in a unified manner. Through NACOS, the configuration files involved in the program are managed in a unified manner, and the configuration files can be pushed to the application end in real time, and they can take effect without restarting the service, greatly improving the efficiency of development and operation and maintenance; through VictoriaMetrics database storage, real-time and accurate data can be provided for upper-level applications.

[0077] The present invention also provides a data acquisition device. Figure 7 , Figure 7 7 is a schematic diagram of the structure of the data acquisition device provided by the present invention. In this embodiment, the data acquisition device includes a configuration module 710 , a creation module 720 , a development module 730 and a collection module 740 .

[0078] Configuration module 710 is used to configure the development environment required for data collection.

[0079] The creation module 720 is used to create a data acquisition microservice project in a development environment.

[0080] The development module 730 is used to configure multiple data collection modules of the data collection microservice project based on the type of data to be collected; each data collection module is used to collect one type of data.

[0081] The collection module 740 is used to start the data collection microservice project and collect various types of data to be collected based on multiple data collection modules.

[0082] In some embodiments, multiple data acquisition modules include a network device acquisition module, a security device acquisition module and an audio and video acquisition module; the network device acquisition module is used to collect network device information; the network device information includes one or more of network device asset information, network device performance information, network link information and network device port performance information; the security device acquisition module is used to collect security information of terminal devices; security information includes one or more of security device asset information, security assessment information and security system assessment item information; the audio and video acquisition module is used to collect audio and video information; the audio and video information includes one or more of the operating status information of the audio and video equipment, audio and video alarm information, audio and video overall satisfaction information, video, voice, display and control equipment and service static information.

[0083] In some embodiments, the acquisition module 740 is used to determine the load of each data acquisition module; based on the load of each data acquisition module, determine whether there is an overloaded data acquisition module with a load greater than or equal to a preset threshold; if there is an overloaded data acquisition module with a load greater than or equal to the preset threshold, configure a new data acquisition module according to the type of data to be collected by the overloaded data acquisition module; the new data acquisition module and the overloaded data acquisition module are used to collect the same type of data to be collected.

[0084] In some embodiments, the acquisition module 740 is used to determine the fault status of each data acquisition module; based on the fault status of each data acquisition module, determine whether there is a faulty data acquisition module; if there is a faulty data acquisition module, suspend data acquisition of the faulty data acquisition module, and continue to collect the corresponding type of data to be collected through the non-faulty data acquisition module.

[0085] In some embodiments, the acquisition module 740 is used to perform data cleaning and data conversion processing on various types of data to be collected, respectively, to obtain various types of data to be stored; the various types of data to be stored have the same data format; and the various types of data to be stored are stored in a time series database.

[0086] In some embodiments, the collection module 740 is used to load a configuration file of a data collection microservice project; wherein the configuration file is managed by a configuration management platform.

[0087] The invention also provides an electronic device. Figure 8 is a schematic diagram of the structure of the electronic device provided by the present invention, such as Figure 8 As shown, the electronic device may include: a processor 810, a communication interface 820, a memory 830 and a communication bus 840, wherein the processor 810, the communication interface 820 and the memory 830 communicate with each other through the communication bus 840. The processor 810 may call the logic instructions in the memory 830 to execute the data acquisition method.

[0088] In addition, the logic instructions in the above-mentioned memory 830 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0089] The present invention also provides a non-transitory computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the data acquisition method provided by the above methods is implemented.

[0090] The present invention also provides a computer program product, which includes a computer program. The computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the data acquisition methods provided by the above methods.

[0091] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0092] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0093] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A data collection method, characterized in that: include: Configure the development environment required for data collection; In the development environment, create a data collection microservice project; Based on the type of data to be collected, configure multiple data collection modules of the data collection microservice project; Each of the data collection modules is used to collect one type of data; The data collection microservice project is started, and based on the multiple data collection modules, multiple types of data to be collected are collected respectively.

2. The data collection method according to claim 1, characterized in that: The plurality of data acquisition modules include a network equipment acquisition module, a security equipment acquisition module and an audio and video acquisition module; The network device collection module is used to collect network device information; the network device information includes one or more of network device asset information, network device performance information, network link information and network device port performance information; The security equipment collection module is used to collect security information of terminal equipment; the security information includes one or more of security equipment asset information, security assessment information and security system assessment item information; The audio and video acquisition module is used to collect audio and video information; the audio and video information includes one or more of the operating status information of the audio and video equipment, audio and video alarm information, audio and video overall satisfaction information, video, voice, display and control equipment and service static information.

3. The data collection method according to claim 1, characterized in that: After the plurality of data collection modules are used to collect a plurality of types of data to be collected, the method further includes: Determining the load of each of the data acquisition modules; Based on the load of each of the data acquisition modules, determining whether there is an overloaded data acquisition module whose load is greater than or equal to a preset threshold; If there is an overloaded data acquisition module whose load is greater than or equal to a preset threshold, a new data acquisition module is configured according to the type of data to be collected by the overloaded data acquisition module; the new data acquisition module and the overloaded data acquisition module are used to collect the same type of data to be collected.

4. The data collection method according to claim 1, characterized in that: After the plurality of data collection modules are used to collect a plurality of types of data to be collected, the method further includes: Determining a fault status of each of the data acquisition modules; Based on the fault status of each of the data acquisition modules, determining whether there is a faulty data acquisition module; If there is a faulty data collection module, the data collection of the faulty data collection module is suspended, and the corresponding type of data to be collected continues to be collected through the non-faulty data collection module.

5. The data collection method according to claim 1, characterized in that: After the plurality of data collection modules are used to collect a plurality of types of data to be collected, the method further includes: Performing data cleaning and data conversion processing on the various types of data to be collected respectively to obtain various types of data to be stored; the various types of data to be stored have the same data format; The multiple types of data to be stored are stored in a time series database.

6. The data collection method according to claim 1, characterized in that: Before starting the data collection microservice project, it also includes: Load the configuration file of the data acquisition microservice project; wherein the configuration file is managed by a configuration management platform.

7. A data acquisition device, characterized in that: include: Configuration module, used to configure the development environment required for data collection; Creating a module for creating a data collection microservice project in the development environment; A development module, used for configuring multiple data collection modules of the data collection microservice project based on the type of data to be collected; Each of the data collection modules is used to collect one type of data; The collection module is used to start the data collection microservice project and collect multiple types of data to be collected based on multiple data collection modules.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the data collection method according to any one of claims 1 to 6 is implemented.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the data collection method according to any one of claims 1 to 6 is implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the data collection method according to any one of claims 1 to 6 is implemented.

Images