Data format conversion method and device, equipment, storage medium and program product

Abstract

The embodiment of the invention provides a data format conversion method and device, equipment, a storage medium and a program product, and is applied to the field of cloud computing. The method comprises the following steps: acquiring to-be-processed monitoring data through a processing terminal; a processing terminal determines a target preset format conversion strategy corresponding to a target service from a plurality of preset format conversion strategies according to the target service corresponding to to-be-processed monitoring data; and performing format conversion on the to-be-processed monitoring data through the processing terminal according to the target preset format conversion strategy to generate target monitoring data. The method is used for improving the data processing efficiency.

Description

Technical Field

[0001] This application relates to the field of cloud computing, and in particular to a data format conversion method, apparatus, device, storage medium, and program product. Background Technology

[0002] With the widespread application of cloud computing technology, monitoring the service stability and performance at the infrastructure layer of cloud computing architecture has become essential to ensuring the stable operation of cloud computing services. By acquiring and analyzing monitoring data at the infrastructure layer, we can intuitively reflect the performance and operational stability information of the infrastructure.

[0003] In existing technologies, the collection of monitoring data mainly relies on the data interfaces provided by the infrastructure. By using the data interfaces corresponding to various infrastructures, monitoring data corresponding to various infrastructures can be obtained. Based on the monitoring data, the operational status of the infrastructure for a specific task or a specific time can be analyzed to obtain the performance information and operational stability information of the infrastructure.

[0004] Because various infrastructures use different data formats, directly using data with inconsistent formats for data analysis increases the computational difficulty of data analysis, makes data format processing more challenging, and leads to low accuracy of data analysis results due to inconsistent data formats. At the same time, when dealing with large amounts of data, processing data with multiple different formats is very difficult. Therefore, there is a technical problem of low data processing efficiency. Summary of the Invention

[0005] This application provides data format conversion methods, apparatus, devices, storage media, and program products to achieve the technical effect of improving data processing efficiency.

[0006] In a first aspect, embodiments of this application provide a data format conversion method applied to a data acquisition system, the data acquisition system including a processing terminal, the method comprising:

[0007] The monitoring data to be processed is obtained through the processing terminal;

[0008] The processing terminal determines the target preset format conversion strategy corresponding to the target service from multiple preset format conversion strategies based on the target service corresponding to the monitoring data to be processed.

[0009] The processing terminal converts the format of the monitoring data to be processed according to the target preset format conversion strategy, and generates the target monitoring data.

[0010] In one possible implementation, the processing terminal converts the format of the monitoring data to be processed according to a preset format conversion strategy to generate the target monitoring data, including:

[0011] By traversing the data structure of the monitoring data to be processed through the processing terminal, the key-value pairs corresponding to the monitoring data to be processed are extracted according to the target preset format conversion strategy.

[0012] The processing terminal generates target monitoring data based on the target's preset format conversion strategy and key-value pairs.

[0013] In one possible implementation, the data acquisition system further includes a cloud server, a Kafka message queue subsystem, and a remote development machine, which acquires monitoring data to be processed through a processing terminal, including:

[0014] The producer module in the Kafka message queue subsystem collects monitoring data to be processed from the target node of the cloud server.

[0015] The producer module collects the monitoring data to be processed and stores it in the target topic of the Kafka message queue subsystem.

[0016] The consumer module in the Kafka message queue subsystem sends the monitoring data to be processed stored in the target topic to the remote development machine.

[0017] The monitoring data to be processed is sent to the processing terminal via a remote development machine, so that the processing terminal can obtain the monitoring data to be processed.

[0018] In one possible implementation, the monitoring data to be processed is sent to the processing terminal via a remote development machine, including:

[0019] When the amount of monitoring data to be processed stored on the remote development machine exceeds the preset data volume, all the monitoring data to be processed stored on the remote development machine will be sent to the processing terminal.

[0020] In one possible implementation, the processing terminal converts the format of the monitoring data to be processed according to a preset format conversion strategy to generate the target monitoring data, including:

[0021] The processing terminal performs data cleaning on the monitoring data to be processed, generating deduplicated monitoring data.

[0022] The processing terminal converts the deduplicated monitoring data according to the target's preset format conversion strategy to generate the target monitoring data.

[0023] In one possible implementation, the monitoring data to be processed includes at least one of the following: CPU utilization, memory utilization, and network inflow / outflow rates.

[0024] Secondly, embodiments of this application provide a data format conversion device applied to a data acquisition system, the data acquisition system including a processing terminal, the device including:

[0025] The acquisition module is used to acquire monitoring data to be processed through the processing terminal;

[0026] The first processing module is used to determine the target preset format conversion strategy corresponding to the target service from multiple preset format conversion strategies based on the target service corresponding to the monitoring data to be processed through the processing terminal;

[0027] The second processing module is used to convert the format of the monitoring data to be processed according to the target preset format conversion strategy through the processing terminal, and generate the target monitoring data.

[0028] In one possible implementation, the second processing module is further configured to:

[0029] By traversing the data structure of the monitoring data to be processed through the processing terminal, the key-value pairs corresponding to the monitoring data to be processed are extracted according to the target preset format conversion strategy.

[0030] The processing terminal generates target monitoring data based on the target's preset format conversion strategy and key-value pairs.

[0031] In one possible implementation, the data acquisition system further includes a cloud server, a Kafka message queue subsystem, and a remote development machine; the acquisition module is also used for:

[0032] The producer module in the Kafka message queue subsystem collects monitoring data to be processed from the target node of the cloud server.

[0033] The producer module collects the monitoring data to be processed and stores it in the target topic of the Kafka message queue subsystem.

[0034] The consumer module in the Kafka message queue subsystem sends the monitoring data to be processed stored in the target topic to the remote development machine.

[0035] The monitoring data to be processed is sent to the processing terminal via a remote development machine, so that the processing terminal can obtain the monitoring data to be processed.

[0036] In one possible implementation, the acquisition module is further configured to:

[0037] When the amount of monitoring data to be processed stored on the remote development machine exceeds the preset data volume, all the monitoring data to be processed stored on the remote development machine will be sent to the processing terminal.

[0038] In one possible implementation, the second processing module is further configured to:

[0039] The processing terminal performs data cleaning on the monitoring data to be processed, generating deduplicated monitoring data.

[0040] The processing terminal converts the deduplicated monitoring data according to the target's preset format conversion strategy to generate the target monitoring data.

[0041] In one possible implementation, the monitoring data to be processed includes at least one of the following: CPU utilization, memory utilization, and network inflow / outflow rates.

[0042] Thirdly, embodiments of this application provide a data format conversion device, including: a memory and a processor;

[0043] The memory stores the instructions that the computer executes;

[0044] The processor executes computer execution instructions stored in memory, causing the processor to perform the first aspect and / or various possible implementations of the first aspect as described above.

[0045] Fourthly, embodiments of this application provide a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the first aspect and / or various possible implementations of the first aspect.

[0046] Fifthly, embodiments of this application provide a computer program product, including a computer program that, when executed by a processor, implements the first aspect and / or various possible implementations of the first aspect.

[0047] This application provides a data format conversion method, apparatus, device, storage medium, and program product. It acquires monitoring data to be processed through a processing terminal of a data acquisition system; and based on the target service corresponding to the monitoring data to be processed, the processing terminal determines a target preset format conversion strategy corresponding to the target service from multiple preset format conversion strategies; and converts the monitoring data to be processed according to the target preset format conversion strategy to generate target monitoring data. This application utilizes targeted preset format conversion strategies to convert various types of monitoring data into the same format. Consistent data format improves data analysis efficiency during data analysis, thereby achieving the technical effect of improving data processing efficiency. Attached Figure Description

[0048] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0049] Figure 1 Flowchart of the data format conversion method provided in this application Figure 1 ;

[0050] Figure 2 Flowchart of the data format conversion method provided in this application Figure 2 ;

[0051] Figure 3 A system architecture diagram of a data processing system based on data format conversion is provided for embodiments of this application;

[0052] Figure 4 Flowchart of the data format conversion method provided in this application Figure 3 ;

[0053] Figure 5 A schematic diagram of the data format conversion device provided in this application;

[0054] Figure 6 Hardware structure diagram of the data format conversion device provided in this application.

[0055] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to particular embodiments. Detailed Implementation

[0056] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0057] In existing technologies, monitoring data is collected through data interfaces provided by various infrastructures, and unified data analysis is performed based on this monitoring data to obtain analysis results on the performance or operational stability of the corresponding infrastructures. However, due to the diverse data formats of infrastructures and the use of uniform data processing methods for these formats in existing technologies, the accuracy of data analysis results is low due to inconsistent data formats. Furthermore, the processing of large amounts of data is challenging, resulting in low data processing efficiency.

[0058] The data format conversion method provided in this application is applied to a data acquisition system, which includes a processing terminal. The processing terminal acquires monitoring data to be processed; based on the target service corresponding to the monitoring data to be processed, the processing terminal determines the target preset format conversion strategy corresponding to the target service from multiple preset format conversion strategies; and converts the monitoring data to be processed according to the target preset format conversion strategy to generate the target monitoring data. This application utilizes a targeted preset format conversion strategy to convert various types of monitoring data into the same format. Consistent data format improves the processing efficiency of data analysis and solves the technical problem of low data processing efficiency in existing technologies.

[0059] The technical solution of this application and how the technical solution of this application solves the above-mentioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will now be described with reference to the accompanying drawings.

[0060] Figure 1 Flowchart of the data format conversion method provided in this application Figure 1 The data format conversion method is applied to the data acquisition system, which includes a processing terminal. For example... Figure 1 As shown, the method includes:

[0061] S101. Obtain the monitoring data to be processed through the processing terminal.

[0062] Optionally, the monitoring data to be processed includes at least one of the following: Central Processing Unit (CPU) utilization, memory utilization, and network inflow / outflow rates.

[0063] It should be noted that this application does not limit the monitoring data to be processed; the monitoring data to be processed can be any monitoring data used to reflect the operational status of the infrastructure. For example, if the infrastructure is a server, the monitoring data to be processed can be: CPU utilization, memory utilization, disk usage, network traffic, network outflow and inflow rates, system load, number of processes, temperature monitoring, number of error logs, runtime, and service status.

[0064] In this embodiment, the monitoring data to be processed acquired by the processing terminal includes monitoring data in various formats; the format of the monitoring data to be processed is related to the device corresponding to the monitoring data to be processed; for example, if the acquired monitoring data to be processed is the memory utilization rate of a virtual machine, the data to be processed includes the memory utilization rate of the node corresponding to each virtual machine that needs to be collected, wherein the type of server to which each virtual machine is deployed and the category to which the virtual machine belongs will affect the memory utilization-related data finally obtained; thus, the format of the monitoring data to be processed acquired by the processing terminal is different.

[0065] S102. The processing terminal determines the target preset format conversion strategy corresponding to the target service from multiple preset format conversion strategies based on the target service corresponding to the monitoring data to be processed.

[0066] In this embodiment, each target service has a corresponding preset format conversion strategy, which can be used to convert the monitoring data to be processed into target monitoring data. If the target service corresponding to the prosecutor's office to be processed does not have a corresponding preset format conversion strategy, a new preset format conversion strategy needs to be designed for the new target service, and the existing preset format conversion strategy needs to be updated.

[0067] In this embodiment, multiple preset format conversion strategies can be stored in the processing terminal in the form of scripts or functions. When the processing terminal performs format processing on the monitoring data to be processed, it can run the corresponding script file or call the corresponding program function.

[0068] S103. The processing terminal converts the format of the monitoring data to be processed according to the target preset format conversion strategy to generate the target monitoring data.

[0069] Alternatively, one specific implementation method for generating target monitoring data can be:

[0070] By traversing the data structure of the monitoring data to be processed through the processing terminal, the key-value pairs corresponding to the monitoring data to be processed are extracted according to the target preset format conversion strategy.

[0071] The processing terminal generates target monitoring data based on the target's preset format conversion strategy and key-value pairs.

[0072] In this embodiment, the monitoring data to be processed obtained by the processing terminal contains multiple types of data items. If the data items required by the target preset format conversion strategy are consistent with the target monitoring data to be converted, then the data items need to be extracted from the original monitoring data to be processed. Generally, the corresponding data items are obtained by extracting key-value pairs.

[0073] For example, if the acquired monitoring data to be processed includes server CPU utilization, memory utilization, and disk usage, the target preset format conversion strategy needs to extract the key-value pairs of memory utilization and CPU utilization. Specifically: CPU utilization: key: `cpu_usage` value: `75%`; Memory utilization: key: `memory_usage` value: `60%`; Disk usage: key: `disk_usage` value: `80%`. Then the final extracted target monitoring data will be: CPU utilization: key: `cpu_usage` value: `75%`; Memory utilization: key: `memory_usage` value: `60%`.

[0074] Alternatively, another specific implementation method for generating target monitoring data can be:

[0075] The processing terminal performs data cleaning on the monitoring data to be processed, generating deduplicated monitoring data.

[0076] The processing terminal converts the deduplicated monitoring data according to the target's preset format conversion strategy to generate the target monitoring data.

[0077] In this embodiment, data cleaning includes: data deduplication and data filling, which fills in data missed during collection and removes duplicate monitoring data. This can reduce the complexity of the data before data format conversion, thereby reducing the difficulty of data format conversion and improving the efficiency of data format conversion.

[0078] The data format conversion method provided in this application involves acquiring monitoring data to be processed through the processing terminal of a data acquisition system; determining a target preset format conversion strategy corresponding to the target service from multiple preset format conversion strategies based on the target service corresponding to the monitoring data to be processed by the processing terminal; and converting the monitoring data to be processed according to the target preset format conversion strategy to generate target monitoring data. This application utilizes targeted preset format conversion strategies to convert various types of monitoring data into the same format, effectively improving the processing efficiency of data analysis when performing data analysis with consistent data format, thus achieving the technical effect of improving data processing efficiency.

[0079] Figure 2 Flowchart of the data format conversion method provided in this application Figure 2 This embodiment is in Figure 1 Based on the above embodiments shown, a detailed explanation is provided on how to acquire monitoring data to be processed through a processing terminal. The data acquisition system further includes: a cloud server, a Kafka message queue subsystem, and a remote development machine. For example... Figure 2As shown, the method includes:

[0080] S201. Collect monitoring data to be processed from the target node of the cloud server through the producer module of the Kafka message queue subsystem.

[0081] In this embodiment, the target node of the cloud server is the infrastructure node in the cloud computing network architecture, including but not limited to: computing resource node, storage service node, network service node, security service node, management and monitoring service node, backup and recovery service node.

[0082] For example, the target node of the cloud server can be a virtual private cloud service node used to provide network services. The monitoring data to be processed collected for the virtual private cloud service node includes, but is not limited to: network outflow and inflow rates, inbound and outbound traffic per unit time, network interface traffic and network interface status, and network bandwidth utilization.

[0083] S202. The producer module stores the collected monitoring data to be processed into the target topic in the Kafka message queue subsystem.

[0084] In this embodiment, Kafka refers to an open-source distributed stream processing platform used for processing data streams. The Kafka message queue subsystem includes a producer module, a consumer module, and a target topic; the monitoring data to be processed collected by the producer module is stored in the target topic, and the consumer module obtains the corresponding monitoring data to be processed by consuming the target topic.

[0085] Figure 3 This document provides a system architecture diagram for a data processing system based on data format conversion, as illustrated in an embodiment of this application. For example,... Figure 3 As shown, the data processing system provided in this embodiment includes: an infrastructure layer, a data acquisition layer, a data acquisition layer, and a data processing layer.

[0086] The infrastructure layer includes various types of infrastructure used to form a cloud computing network. The data acquisition layer is the producer module provided by the Kafka message queue subsystem in this application. The data acquisition layer consists of the target topic and consumer modules corresponding to the Kafka message queue subsystem in this application. The data processing layer is the processing terminal in this application, which includes a data format conversion module for format conversion and a data storage and analysis module for data storage and analysis. Data storage can utilize file caching or a database.

[0087] The data processing system also includes common components, including a logging system, configuration management, and script management. The logging system tracks collected monitoring data to be processed; configuration management allows for dynamic adjustment of relevant parameters, facilitating system migration; and script management stores commonly used test scripts for offline debugging and functional unit testing.

[0088] S203. Through the consumer module in the Kafka message queue subsystem, send the monitoring data to be processed stored in the target topic to the remote development machine.

[0089] In this embodiment, when the consumer module consumes the target topic, it can use data preprocessing to initially organize the acquired data to be processed. For example, regular expressions can be written into the consumer module to filter the data to be processed stored in the target topic, obtaining high-clarity data to facilitate subsequent data format conversion.

[0090] In this embodiment, in order to improve the efficiency of data processing, multiple consumer units can be set in the consumer module. The multiple consumer units can obtain the data to be processed from the target topic in a multi-threaded manner, thereby improving the efficiency of data transmission and further improving the efficiency of subsequent data processing.

[0091] S204. Send the monitoring data to be processed to the processing terminal via the remote development machine so that the processing terminal can obtain the monitoring data to be processed.

[0092] Alternatively, one specific implementation method for sending the monitoring data to be processed to the data processing terminal can be:

[0093] When the amount of monitoring data to be processed stored on the remote development machine exceeds the preset data volume, all the monitoring data to be processed stored on the remote development machine will be sent to the processing terminal.

[0094] For example, if the target topic is to collect the network inflow and outflow rates of a load balancer to ensure proper network traffic distribution, then during the data collection phase, a data collection agent is deployed on each load balancer node. The data collection agent periodically collects the network inflow and outflow rates. The producer module of the Kafka message queue subsystem collects the monitoring data to be processed collected by the data collection agent and sends the monitoring data to be processed to the target topic. The data format of the collected monitoring data to be processed can be: {"inbound_rate": 500.0, "outbound_rate": 450.0}, where inbound_rate represents the network inflow rate and outbound_rate represents the network outflow rate.

[0095] The Kafka message queue subsystem's consumer module is associated with the remote development machine; the consumer module consumes target topics and obtains monitoring data to be processed; the monitoring data to be processed is cached in the remote development machine; the format of the cached monitoring data to be processed is consistent with the format of the collected monitoring data to be processed.

[0096] When the number of pending monitoring data cached in the remote development machine exceeds the preset data volume of 100, the pending monitoring data can be sent to the processing terminal via a script file. The pending monitoring data is stored in the processing terminal as a file.

[0097] The processing terminal performs data extraction and format conversion on the acquired monitoring data, removing duplicate data, standardizing the data format, and adding timestamps. The processed data format can be: {"timestamp": "2024-05-24T10:00:00Z", "inbound_rate": 500.0, "outbound_rate": 450.0}; where timestamp represents the timestamp, inbound_rate represents the network inflow rate, and outbound_rate represents the network outflow rate.

[0098] After cleaning and format conversion, the data can be stored in the local database corresponding to the processing terminal. The stored data can then be used for data analysis and the generation of related reports, thereby intuitively and clearly reflecting the performance and stability of the cloud server.

[0099] In this embodiment, the Kafka message queue is used to collect and transmit the data to be processed tree. By leveraging Kafka's data stream processing capabilities, the efficiency of data transmission is improved. At the same time, by temporarily storing the collected monitoring data to be processed on a remote development machine, the storage and computing pressure on the local processing terminal is alleviated, thereby improving the efficiency of data processing.

[0100] In this embodiment, when the amount of monitoring data to be processed is large, the local processing terminal cannot process a large amount of data at the same time. By storing the data in a remote development machine and processing the data quantitatively, the pressure on the processing terminal can be alleviated, thereby improving the data processing efficiency.

[0101] Figure 4 Flowchart of the data format conversion method provided in this application Figure 3 .like Figure 4As shown, this application collects monitoring data to be processed corresponding to multiple infrastructures in the infrastructure layer through the producer module, and uploads the data to the target topic of the Kafka message queue subsystem. The consumer module consumes the target topic and uploads the acquired data to be processed to the remote development machine. When the data in the remote development machine exceeds the preset data volume, the data to be processed is pulled to the processing terminal for data cleaning, data storage and analysis.

[0102] For example, if the target topic is to collect the CPU utilization and memory utilization of a cloud server to ensure server performance and stability, then during the data collection phase, a data collection agent is deployed on each cloud server node; the data collection agent periodically collects the server's CPU utilization and memory utilization; the producer module of the Kafka message queue subsystem collects the monitoring data to be processed collected by the data collection agent and sends the monitoring data to be processed to the target topic; the data format of the collected monitoring data to be processed can be: {"cpu_usage": 45.0, "memory_usage":70.5}, where cpu_usage represents CPU utilization and memory_usage represents memory utilization.

[0103] The Kafka message queue subsystem's consumer module is associated with the remote development machine; the consumer module consumes target topics and obtains monitoring data to be processed; the monitoring data to be processed is cached in the remote development machine; the format of the cached monitoring data to be processed is consistent with the format of the collected monitoring data to be processed.

[0104] When the number of pending monitoring data cached in the remote development machine exceeds the preset data volume of 100, the pending monitoring data can be sent to the processing terminal via a script file. The pending monitoring data is stored in the processing terminal as a file.

[0105] The processing terminal performs data extraction and format conversion on the acquired monitoring data, removing duplicate data, standardizing the data format, and adding timestamps. The processed data format can be: {"timestamp": "2024-05-24T10:00:00Z", "cpu_usage": 45.0, "memory_usage": 70.5}; where timestamp represents the timestamp, cpu_usage represents CPU utilization, and memory_usage represents memory utilization.

[0106] After cleaning and format conversion, the data can be stored in the local database corresponding to the processing terminal. The stored data can then be used for data analysis and the generation of related reports, thereby intuitively and clearly reflecting the performance and stability of the cloud server.

[0107] In this embodiment, the Kafka message queue is used to collect and transmit the data to be processed tree. By leveraging Kafka's data stream processing capabilities, the efficiency of data transmission is improved. At the same time, by temporarily storing the collected monitoring data to be processed on a remote development machine, the storage and computing pressure on the local processing terminal is alleviated, thereby improving the efficiency of data processing.

[0108] Figure 5 This is a schematic diagram of the data format conversion device provided in this application, applied to a data acquisition system, which includes a processing terminal. Figure 5 As shown, the data format conversion device 50 provided in this embodiment includes:

[0109] The acquisition module 501 is used to acquire monitoring data to be processed through the processing terminal;

[0110] The first processing module 502 is used to determine the target preset format conversion strategy corresponding to the target service from multiple preset format conversion strategies based on the target service corresponding to the monitoring data to be processed through the processing terminal.

[0111] The second processing module 503 is used to convert the format of the monitoring data to be processed according to the target preset format conversion strategy through the processing terminal, and generate the target monitoring data.

[0112] In one possible implementation, the second processing module 503 is further configured to:

[0113] By traversing the data structure of the monitoring data to be processed through the processing terminal, the key-value pairs corresponding to the monitoring data to be processed are extracted according to the target preset format conversion strategy.

[0114] The processing terminal generates target monitoring data based on the target's preset format conversion strategy and key-value pairs.

[0115] In one possible implementation, the data acquisition system also includes a cloud server, a Kafka message queue subsystem, and a remote development machine. The acquisition module 501 is further used for:

[0116] The producer module in the Kafka message queue subsystem collects monitoring data to be processed from the target node of the cloud server.

[0117] The producer module collects the monitoring data to be processed and stores it in the target topic of the Kafka message queue subsystem.

[0118] The consumer module in the Kafka message queue subsystem sends the monitoring data to be processed stored in the target topic to the remote development machine.

[0119] The monitoring data to be processed is sent to the processing terminal via a remote development machine, so that the processing terminal can obtain the monitoring data to be processed.

[0120] In one possible implementation, the acquisition module 501 is also used for:

[0121] When the amount of monitoring data to be processed stored on the remote development machine exceeds the preset data volume, all the monitoring data to be processed stored on the remote development machine will be sent to the processing terminal.

[0122] In one possible implementation, the second processing module 503 is further configured to:

[0123] The processing terminal performs data cleaning on the monitoring data to be processed, generating deduplicated monitoring data.

[0124] The processing terminal converts the deduplicated monitoring data according to the target's preset format conversion strategy to generate the target monitoring data.

[0125] In one possible implementation, the monitoring data to be processed includes at least one of the following: CPU utilization, memory utilization, and network inflow / outflow rates.

[0126] The data format conversion device provided in this embodiment can execute the method provided in the above method embodiment. Its implementation principle and technical effect are similar, and will not be described in detail here.

[0127] Figure 6 A hardware structure diagram of the data format conversion device provided in this application. (See diagram below.) Figure 6 As shown, the data format conversion device 60 provided in this embodiment includes at least one processor 601 and a memory 602. Optionally, the device 60 further includes a communication component 603. The processor 601, memory 602, and communication component 603 are connected via a bus 604.

[0128] In a specific implementation, at least one processor 601 executes computer execution instructions stored in memory 602, causing at least one processor 601 to perform the above-described method.

[0129] The specific implementation process of processor 601 can be found in the above method embodiments, and its implementation principle and technical effect are similar. It will not be repeated here.

[0130] In the above embodiments, it should be understood that the processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in this invention can be directly implemented by a hardware processor, or implemented by a combination of hardware and software modules within the processor.

[0131] The memory may include random access memory (RAM) and may also include non-volatile memory (NVM), such as at least one disk storage device.

[0132] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, the buses shown in the accompanying drawings are not limited to a single bus or a single type of bus.

[0133] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described method.

[0134] This application also provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the above-described method.

[0135] The aforementioned readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The readable storage medium can be any available medium accessible to a general-purpose or special-purpose computer.

[0136] An exemplary readable storage medium is coupled to a processor, enabling the processor to read information from and write information to the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can reside in an Application Specific Integrated Circuit (ASIC). Alternatively, the processor and the readable storage medium can exist as discrete components in the device.

[0137] The division of units is merely a logical functional division; in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be indirect coupling or communication connection through some interfaces, devices, or units, and may be electrical, mechanical, or other forms.

[0138] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0139] In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0140] If a function is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0141] Those skilled in the art will understand that all or part of the steps of the above-described method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments; and the aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

[0142] Finally, it should be noted that other embodiments of the invention will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This invention is intended to cover any variations, uses, or adaptations of the invention that follow the general principles of the invention and include common knowledge or customary techniques in the art not disclosed herein, and is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of the invention is limited only by the appended claims.

Claims

1. A data format conversion method characterized by, The method is applied to a data collection system comprising a processing terminal, and comprises the following steps: acquiring, by the processing terminal, to-be-processed monitoring data; determining, by the processing terminal, a target preset format conversion strategy corresponding to a target service of the to-be-processed monitoring data from a plurality of preset format conversion strategies according to the target service; performing, by the processing terminal, format conversion on the to-be-processed monitoring data according to the target preset format conversion strategy to generate target monitoring data.

2. The method of claim 1, wherein, The step of performing, by the processing terminal, format conversion on the to-be-processed monitoring data according to the target preset format conversion strategy to generate target monitoring data comprises the following steps: extracting, by the processing terminal, a key-value pair corresponding to the to-be-processed monitoring data according to the target preset format conversion strategy by traversing a data structure of the to-be-processed monitoring data; generating, by the processing terminal, the target monitoring data according to the target preset format conversion strategy and the key-value pair.

3. The method according to claim 1 or 2, characterized in that, The data collection system further comprises a cloud server, a Kafka message queue subsystem, and a remote development machine, and the step of acquiring, by the processing terminal, to-be-processed monitoring data comprises the following steps: collecting, by a producer module in the Kafka message queue subsystem, the to-be-processed monitoring data from a target node of the cloud server; storing, by the producer module, the collected to-be-processed monitoring data to a target topic in the Kafka message queue subsystem; sending, by a consumer module in the Kafka message queue subsystem, the to-be-processed monitoring data stored in the target topic to the remote development machine; sending, by the remote development machine, the to-be-processed monitoring data to the processing terminal, so that the processing terminal acquires the to-be-processed monitoring data.

4. The method of claim 3, wherein, The step of sending, by the remote development machine, the to-be-processed monitoring data to the processing terminal comprises the following step: when the number of to-be-processed monitoring data stored in the remote development machine exceeds a preset data amount, sending, by the remote development machine, all the stored to-be-processed monitoring data to the processing terminal.

5. The method according to claim 1 or 2, characterized in that, The step of performing, by the processing terminal, format conversion on the to-be-processed monitoring data according to the target preset format conversion strategy to generate target monitoring data further comprises the following steps: performing, by the processing terminal, data cleaning on the to-be-processed monitoring data to generate de-duplicated monitoring data; performing, by the processing terminal, format conversion on the de-duplicated monitoring data according to the target preset format conversion strategy to generate the target monitoring data.

6. The method of claim 1 or 2, wherein, The to-be-processed monitoring data comprises at least one of the following: a central processing unit (CPU) utilization rate, a memory utilization rate, and a network inflow and outflow rate.

7. A data format conversion apparatus characterized by comprising: The method is applied to a data collection system comprising a processing terminal, and comprises the following steps: acquiring, by the processing terminal, to-be-processed monitoring data; determining, by the processing terminal, a target preset format conversion strategy corresponding to a target service of the to-be-processed monitoring data from a plurality of preset format conversion strategies according to the target service; The second processing module is configured to convert the format of the to-be-processed monitoring data according to the target preset format conversion strategy, and generate target monitoring data.

8. A data format conversion apparatus characterized by comprising: Comprising: a memory, a processor; the memory stores computer execution instructions; the processor executes the computer execution instructions stored in the memory, so that the processor executes the method in any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, The computer readable storage medium stores computer execution instructions, and the computer execution instructions are executed by the processor to implement the method in any one of claims 1-6.

10. A computer program product, characterised in that, The computer program is executed by the processor to implement the method in any one of claims 1-6.

Images