Data processing method and device, computer equipment and computer readable storage medium
The method addresses metadata storage inefficiencies in distributed cluster storage by deduplicating and structuring metadata in a target database, improving accuracy and efficiency in big data scenarios.
Patent Information
- Application Number
- CN202310658223.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-06-05
- Publication Date
- 2024-12-06
AI Technical Summary
Existing distributed cluster storage systems face challenges in quickly and accurately storing metadata information, leading to inefficiencies and instability in data management, particularly in big data scenarios like high-performance computing and video surveillance.
A data processing method involving deduplication of monitoring data in memory and cache, followed by storage in a target database using a configuration model and graph database, ensuring metadata meets storage requirements and is accurately stored.
Improves data storage accuracy and efficiency by deduplicating metadata twice, enhancing processing and storage efficiency through batch processing and structured database management.
Smart Images

Figure CN119094381A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of data processing technology, and in particular to a data processing method, device, computer equipment and computer-readable storage medium. Background Art
[0002] Distributed cluster storage is widely used in many scenarios of current big data (such as high performance, video surveillance, radio and television media, etc.). There is a huge amount of data in the distributed file system, and the premise of big data computing is to organize and store massive data. How to quickly and accurately store metadata information in a distributed system directly affects the performance of metadata operations. Based on this, it is urgent to propose a metadata path information storage method to facilitate the fast and accurate storage of metadata information. Summary of the invention
[0003] The present application provides a data processing method, apparatus, computer device and computer-readable storage medium, aiming to solve the problem of how to quickly and accurately store metadata information.
[0004] In one aspect, the present application provides a data processing method, the method comprising:
[0005] Acquire first monitoring data pushed in batches by the monitoring and alarm system through a preset target gateway, where the first monitoring data is determined by the monitoring and alarm system based on a pre-acquired task log;
[0006] Deduplication processing is performed on the first monitoring data in the memory of the target gateway to obtain deduplication initial metadata;
[0007] Based on the initial metadata, determining target metadata that meets target storage requirements;
[0008] The target metadata is stored in a preset target database.
[0009] In a possible implementation of the present application, determining target metadata that meets target storage requirements based on the initial metadata includes:
[0010] Determining whether the initial metadata exists in a preset cache;
[0011] If the initial metadata exists in the cache, determining that the initial metadata does not meet the target storage requirement;
[0012] If the initial metadata does not exist in the cache, it is determined that the initial metadata meets the target storage requirement, and the initial metadata is used as the target metadata.
[0013] In a possible implementation of the present application, the data in the cache is stored in a bit array structure, and the determining whether the initial metadata exists in the preset cache includes:
[0014] Obtaining an offset of the initial metadata in the bit array, and using the offset as an array subscript of the initial metadata;
[0015] Based on the array index, it is determined whether the initial metadata exists in a preset cache.
[0016] In a possible implementation of the present application, the preset target database includes a target configuration model, and storing the target metadata in the preset target database includes:
[0017] Taking each metadata in the target metadata as vertex data of a target vertex;
[0018] The vertex data is written into the target configuration model.
[0019] In a possible implementation of the present application, the target configuration model includes a time attribute. After writing the target vertex into the target configuration model of a preset target database, the method further includes:
[0020] Add an index to the time attribute.
[0021] In a possible implementation of the present application, after storing the target metadata in a preset target database, the method further includes:
[0022] Based on the target metadata and the preset relationship model, a task information graph is determined.
[0023] In a possible implementation of the present application, determining the task information graph based on the target metadata and a preset relationship model includes:
[0024] Encapsulating the target metadata into graph data of points and edges through a preset relational model;
[0025] The graphic data is written into a preset graphic database.
[0026] In another aspect, the present application provides a data processing device, the device comprising:
[0027] A first push unit, configured to obtain first monitoring data pushed in batches by the monitoring and alarm system through a preset target gateway, wherein the first monitoring data is determined by the monitoring and alarm system based on a pre-acquired task log;
[0028] A first deduplication unit, configured to perform deduplication processing on the first monitoring data in a memory in the target gateway to obtain deduplicated initial metadata;
[0029] A first determining unit, configured to determine target metadata that meets target storage requirements based on the initial metadata;
[0030] The first storage unit is used to store the target metadata in a preset target database.
[0031] In a possible implementation of the present application, the first determining unit specifically includes:
[0032] A second determining unit, configured to determine whether the initial metadata exists in a preset cache;
[0033] a third determining unit, configured to determine, if the initial metadata exists in the cache, that the initial metadata does not meet the target storage requirement;
[0034] The fourth determining unit is configured to determine, if the initial metadata does not exist in the cache, that the initial metadata meets the target storage requirement and use the initial metadata as the target metadata.
[0035] In a possible implementation of the present application, the data in the cache is stored in a bit array structure, and the second determining unit is specifically configured to:
[0036] Obtaining an offset of the initial metadata in the bit array, and using the offset as an array subscript of the initial metadata;
[0037] Based on the array index, it is determined whether the initial metadata exists in a preset cache.
[0038] In a possible implementation of the present application, the preset target database includes a target configuration model, and the first storage unit is specifically used to:
[0039] Taking each metadata in the target metadata as vertex data of a target vertex;
[0040] The vertex data is written into the target configuration model.
[0041] In a possible implementation of the present application, the target configuration model includes a time attribute, and after writing the target vertex into the target configuration model of a preset target database, the apparatus is further configured to:
[0042] Add an index to the time attribute.
[0043] In a possible implementation of the present application, after storing the target metadata in a preset target database, the device further includes:
[0044] The fifth determining unit is used to determine the task information graph based on the target metadata and a preset relationship model.
[0045] In a possible implementation of the present application, the fifth determining unit is specifically configured to:
[0046] Encapsulating the target metadata into graph data of points and edges through a preset relational model;
[0047] The graphic data is written into a preset graphic database.
[0048] On the other hand, the present application also provides a computer device, the computer device comprising:
[0049] one or more processors;
[0050] Memory; and
[0051] One or more application programs, wherein the one or more application programs are stored in the memory and configured to be executed by the processor to implement the data processing method.
[0052] On the other hand, the present application also provides a computer-readable storage medium on which a computer program is stored, and the computer program is loaded by a processor to execute the steps in the data processing method.
[0053] Beneficial effects:
[0054] The embodiment of the present application obtains the first monitoring data pushed by the monitoring alarm system in batches through a preset target gateway, and the first monitoring data is determined by the monitoring alarm system based on the pre-acquired task log; performs deduplication processing on the first monitoring data in the memory of the target gateway to obtain deduplicated initial metadata; determines target metadata that meets the target storage requirements based on the initial metadata; and stores the target metadata in a preset target database. In this way, before storing the metadata, the first monitoring data is deduplicated twice, respectively, which improves the accuracy of data storage. In addition, by processing data in batches, data processing efficiency is improved, and data storage efficiency is improved. BRIEF DESCRIPTION OF THE DRAWINGS
[0055] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.
[0056] Figure 1 is a scenario schematic diagram of a data processing system provided in an embodiment of the present application;
[0057] Figure 2 It is a schematic diagram of an embodiment of a data processing method provided in an embodiment of the present application;
[0058] Figure 3 is a schematic diagram of the structure of an embodiment of a data processing device provided in an embodiment of the present application;
[0059] Figure 4 It is a schematic diagram of the structure of an embodiment of a computer device provided in the embodiments of the present application. DETAILED DESCRIPTION
[0060] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of this application.
[0061] In the description of the present application, it should be understood that the terms "center", "longitudinal", "lateral", "length", "width", "thickness", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside" and the like indicate positions or positional relationships based on the positions or positional relationships shown in the accompanying drawings, which are only for the convenience of describing the present application and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present application. In addition, the terms "first" and "second" are only used for descriptive purposes and cannot be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, the features defined as "first" and "second" may explicitly or implicitly include one or more of the said features. In the description of the present application, the meaning of "multiple" is two or more, unless otherwise clearly and specifically defined.
[0062] In this application, the word "exemplary" is used to mean "used as an example, illustration, or description." Any embodiment described in this application as "exemplary" is not necessarily to be construed as being preferred or advantageous over other embodiments. The following description is given to enable any technician in the field to implement and use the present application. In the following description, details are listed for the purpose of explanation. It should be understood that a person of ordinary skill in the art can recognize that the present application can be implemented without using these specific details. In other instances, well-known structures and processes will not be elaborated in detail to avoid obscuring the description of the present application with unnecessary details. Therefore, the present application is not intended to be limited to the embodiments shown, but is consistent with the widest scope consistent with the principles and features disclosed in the present application.
[0063] At present, distributed cluster storage is widely used in many current big data scenarios (such as high performance, video surveillance, radio and television media resources, etc.). There is a huge amount of data in the distributed file system, and the premise of big data computing is to organize and store massive data. In related schemes, open source time series databases (abbreviated as db) are often used. One of the characteristics of this open source time series database is that it can automatically deduplicate monitoring data and only retain metadata, namely tagKey and tagValue. However, the inventor of the present application found that there are unstable factors in the open source time series database, which is manifested as the lack of queried metadata sometimes, and even the situation of service downtime. For this reason, the embodiments of the present application provide a data processing method, device, computer equipment and computer-readable storage medium, which are described in detail below.
[0064] like Figure 1 As shown, Figure 1 1 is a schematic diagram of a data processing system provided in an embodiment of the present application. The data processing system may include a computer device 100, in which a data processing device is integrated, such as Figure 1 A computer device 100 in FIG.
[0065] In the embodiment of the present application, the computer device 100 is mainly used to obtain the first monitoring data pushed in batches by the monitoring and alarm system through a preset target gateway, and the first monitoring data is determined by the monitoring and alarm system based on a pre-acquired task log; deduplicate the first monitoring data in the memory of the target gateway to obtain deduplicated initial metadata; determine target metadata that meets the target storage requirements based on the initial metadata; and store the target metadata in a preset target database.
[0066] In the embodiment of the present application, the computer device 100 may be a terminal or a server. When the computer device 100 is a server, it may be an independent server, or a server network or server cluster composed of servers. For example, the computer device 100 described in the embodiment of the present application includes but is not limited to a computer, a network host, a single network server, a plurality of network server sets or a cloud server constructed by a plurality of servers. The cloud server is constructed by a large number of computers or network servers based on cloud computing.
[0067] It is understandable that when the computer device 100 in the embodiment of the present application is a terminal, the terminal used may be a device including both receiving and transmitting hardware, that is, a device having receiving and transmitting hardware capable of performing two-way communication on a two-way communication link. Such a device may include: a cellular or other communication device having a single-line display or a multi-line display or a cellular or other communication device without a multi-line display. The specific computer device 100 may be a desktop terminal or a mobile terminal, and the computer device 100 may also be one of a mobile phone, a tablet computer, a laptop computer, a medical auxiliary instrument, etc.
[0068] Those skilled in the art will understand that Figure 1 The application environment shown in the figure is only one application scenario of the present application solution, and is not intended to limit the application scenario of the present application solution. Other application environments may also include Figure 1 More or less computer equipment as shown in Figure 1 Only one computer device is shown in the figure. It can be understood that the data processing system may also include one or more other computer devices, which are not specifically limited here.
[0069] In addition, if Figure 1 As shown, the data processing system may further include a memory 200 for storing data, such as first monitoring data pushed in batches by the monitoring alarm system and data processing data, such as data processing data when the data processing system is running.
[0070] It should be noted that Figure 1 The scenario diagram of the data processing system shown is merely an example. The data processing system and scenario described in the embodiments of the present application are intended to more clearly illustrate the technical solution of the embodiments of the present application, and do not constitute a limitation on the technical solution provided in the embodiments of the present application. A person of ordinary skill in the art will appreciate that with the evolution of the data processing system and the emergence of new business scenarios, the technical solution provided in the embodiments of the present application is equally applicable to similar technical problems.
[0071] Next, the data processing method provided by the embodiment of the present application is introduced.
[0072] In the data processing method of the present embodiment, a data processing device is used as an execution subject. For the sake of simplicity and ease of description, the execution subject will be omitted in the subsequent method embodiments. The data processing device is applied to a computer device. The method includes: obtaining first monitoring data pushed in batches by a monitoring and alarm system through a preset target gateway, and the first monitoring data is determined by the monitoring and alarm system based on a pre-acquired task log; deduplicating the first monitoring data in the memory of the target gateway to obtain deduplicated initial metadata; based on the initial metadata, determining target metadata that meets the target storage requirements; and storing the target metadata in a preset target database.
[0073] See also Figures 2 to 4 , Figure 2 This is a flow chart of an embodiment of a data processing method provided in an embodiment of the present application. The data processing method includes the following steps 201 to 204:
[0074] 201. Acquire first monitoring data pushed in batches by a monitoring and alarm system through a preset target gateway, where the first monitoring data is determined by the monitoring and alarm system based on a pre-acquired task log;
[0075] Among them, the first monitoring data is determined by the monitoring and alarm system based on the pre-acquired task log, and the specific determination process includes: collecting the task log of the target object; reading the data content in the task log, and using the data content as the first monitoring data, wherein the first monitoring data may include tagKey, tagValue and data value. Specifically, the above-mentioned task log of the target object can be implemented by using the preset Grafana Loki component. The Grafana Loki is a highly scalable and efficient log aggregation system designed for cloud-native environments. It can help users collect, archive, search and analyze large amounts of log data generated by the system. Compared with traditional centralized log storage solutions, Loki uses a lighter-weight approach to process log data, and can be queried in real time without preprocessing the data.
[0076] The above-mentioned target object is a system or application on a computer or server. For example, the target object is a scheduling platform, which can be understood as a tool platform. The task log is generated by the target object. The task log refers to the information that records the detailed operation process generated during the execution of the system or program. Specifically, the task log may include the following content: Timestamp: record the time point or duration of the event. Log level: set the corresponding log level for events of different importance, such as DEBUG, INFO, WARN, ERROR, FATAL, etc. Event description: simply and clearly describe what each event will do and the possible impact. Thread ID: in a multi-threaded environment, identify which thread generated the event. Identifier and tracking ID: used to identify multiple events of the same process or interaction to facilitate tracking of problems.
[0077] After the first monitoring data is collected, the first monitoring data can be written into the monitoring alarm system through the Grafana Loki component. Specifically, the monitoring alarm system can be Prometheus. Prometheus is an open source monitoring system and time series database that can help users record any number of indicators and perform visualization, alarm and other processing on these indicators. The core components of Prometheus may include the following: 1. Data model: The Prometheus data model is designed as multi-dimensional time series data, consisting of an indicator name, an optional key-value pair and a timestamp. Each indicator will have a floating point value. The Prometheus data model provides a flexible query language that can filter data according to different conditions. 2. Storage: Prometheus storage is particularly suitable for storing and querying large amounts of time series data. Due to the use of disk storage, calculation results and historical data can be persisted. Data storage usually uses adaptive compression technology to improve efficiency and reduce storage space usage. 3. Query: Prometheus uses the PromQL query language, which supports various functions and operators to query and classify, sample and aggregate the stored time series data. 4. Alerts: Prometheus has a flexible and customizable alert notification mechanism. Users can customize the conditions that trigger alerts and the people who receive alerts. At the same time, Prometheus supports a friendly alert interface and multiple alert notification methods, such as email, Slack, PagerDuty, etc.
[0078] Furthermore, after the first monitoring data is written into the monitoring alarm system, the target gateway preset in this embodiment will obtain the first monitoring data pushed in batches by the monitoring alarm system.
[0079] 202. Deduplication processing is performed on the first monitoring data in the memory of the target gateway to obtain deduplication initial metadata;
[0080] Among them, according to the above, the first monitoring data is pushed in batches by the monitoring alarm system, that is, the first monitoring data includes batch metadata. However, the inventor of the present application found that there may be duplicate data in each batch of metadata, and there may also be duplicate data between batch metadata of different batches. For this reason, deduplication can be performed twice. The first time is to deduplicate the first monitoring data in the memory of the target gateway to obtain the deduplicated initial metadata. Subsequently, when the target gateway interacts with the cache, the deduplicated initial metadata of the batch is deduplicated for the second time based on the deduplicated data of the previous batch. In this way, the storage speed and storage accuracy of the data can be effectively improved.
[0081] Among them, the target gateway is a preset gateway, which refers to an access point to an external network that is currently inaccessible to a network. On the Internet, a gateway usually refers to a server that forwards requests from a client to other servers, also known as an "interconnector" or "protocol converter." Specifically, it serves as a collection point between manufacturers and consumers of data usage, and performs many useful tasks, including: Data routing: The gateway routes requests to the correct backend server based on the URL, protocol type, and other identifier information in the request. Protocol conversion: When data needs to be exchanged between different protocols, the gateway can convert the data format and ensure its correct delivery. Authentication and security: The gateway can perform tasks such as authentication, authorization, and encryption and decryption to ensure data security. Cache and load balancing: The gateway can cache and load balance requests to improve performance and reliability.
[0082] As an optional implementation, deduplication in memory can be achieved using data structures such as hash tables or red-black trees. The specific method is: insert the elements to be deduplicated into a hash table or red-black tree one by one. Here we take a hash table as an example. When inserting an element, use the element itself as the key and specify a default value. When you insert the same element later, you will find that the key already exists in the hash table. At this time, you only need to ignore the corresponding value. Finally, traverse the hash table to get all non-duplicate elements.
[0083] 203. Determine target metadata that meets target storage requirements based on the initial metadata;
[0084] The target storage requirement means that metadata identical to the initial metadata does not exist in the preset cache.
[0085] As an optional implementation, determining the target metadata that meets the target storage requirements based on the initial metadata may include determining whether the initial metadata exists in a preset cache; if the initial metadata exists in the cache, determining that the initial metadata does not meet the target storage requirements; if the initial metadata does not exist in the cache, determining that the initial metadata meets the target storage requirements, and using the initial metadata as the target metadata.
[0086] Specifically, the cache can be Redis. Redis (Remote Dictionary Server) is an open source high-performance key-value storage database, also known as a data structure server. It supports multiple data structure types (such as strings, hashes, lists, sets, ordered sets, etc.) and multiple operation types (such as reading, writing, appending, increasing and decreasing, sorting, range query, etc.), and provides rich functions and operations under rich API interfaces. The characteristics of Redis include: Memory storage: Redis stores data in memory, so its reading speed is very fast. It can also store data on disk to ensure persistence. Diversified data structures and operations: Redis supports multiple data structure types (such as strings, hashes, lists, sets, ordered sets, etc.) and multiple operation types (such as reading, writing, appending, increasing and decreasing, sorting, range query, etc.). High concurrency: Redis is often used to solve high concurrent access problems because it can use a message-based approach to perform tasks, thereby enabling multiple clients to connect at the same time and process requests concurrently. High reliability: Redis ensures high data reliability through master-slave replication and data persistence. It supports different persistence methods, including RDB (snapshot files), AOF (append logs), etc. Simple: Redis is a very simple and easy-to-use database that can be easily installed and managed. Rich ecology: Redis has a fairly mature open source community and has derived many auxiliary tools, such as the redis-cli client and the Web operation interface RedisWeb. At the same time, it is also widely integrated with other technologies such as Kafka, MySQL, and Elaticsearch in the current technology ecology.
[0087] The data in the cache is stored in a bit array structure, and determining whether the initial metadata exists in the preset cache includes: obtaining an offset of the initial metadata in the bit array, and using the offset as an array subscript of the initial metadata; and determining whether the initial metadata exists in the preset cache based on the array subscript.
[0088] As an optional implementation, based on the array subscript, determining whether the initial metadata exists in the preset cache may specifically include: if the value corresponding to the array subscript is 1, it indicates that metadata identical to the initial metadata already exists in the cache; otherwise, it indicates that metadata identical to the initial metadata does not exist in the cache.
[0089] 204. Store the target metadata in a preset target database.
[0090] In this embodiment, the preset target database includes a target configuration model. Specifically, the target database is Nebula Graph, which is a distributed, graph database designed to solve the problem of large-scale data storage and query. It adopts a data structure based on the property graph model to support efficient storage and query of various types of data. Nebula Graph has the following characteristics: High performance: Through distributed architecture, data sharding, multi-copy mechanism and other methods, good scalability and high concurrency can be achieved. High reliability: A multi-copy mechanism is adopted to prevent single point failure and data loss, and backup and recovery mechanisms are provided. Flexible data model: Supports data model based on property graph, which can flexibly represent various types of data to meet the needs of different business scenarios. Rich query language: Supports Graph Query Language (GQL) similar to SQL, which can easily perform graph queries, and supports complex query operations and graph algorithms. Comprehensive security mechanism: Supports multiple security mechanisms such as adding new user accounts, role authorization, and data security management to ensure the security of graph data.
[0091] The target configuration model can be a Tag model, that is, a Tag model created in Nebula Graph. Specifically, the Tag model is a text classification model, which is usually used to automatically classify text tags. The idea is to convert the text into a vector representation, and then train and classify it based on a machine learning algorithm or a deep learning algorithm, and finally correspond the text to a predefined tag. Specifically, the Tag model usually includes the following steps: Word segmentation and text preprocessing: The text to be classified is preprocessed by word segmentation, stop word removal, etc., so as to better extract the feature information of the text. Feature extraction: The text is converted into a vector representation, and feature extraction can be performed using methods such as Bag of Words and TF-IDF. Training model: Based on the labeled training data, a machine learning algorithm or a deep learning algorithm is used to train the text classification model. Text classification: The text to be classified is input into the trained model and its corresponding label is output.
[0092] As an optional implementation, storing the target metadata in a preset target database includes: taking each metadata in the target metadata as vertex data of a target vertex; and writing the vertex data into the target configuration model.
[0093] Among them, taking each metadata in the target metadata as vertex data of a target vertex means that for each information in the target metadata, it can be regarded as a node in a graph database. For example, in a social network, a user can be regarded as a node, the user's attributes (such as age, gender, occupation) can be regarded as the attributes of the node (or called "attribute nodes"), and the relationship between users can be regarded as edges (or called "relationship nodes"). These nodes and edges constitute a graph. Writing the vertex data into the target configuration model means storing these nodes and edges in the target configuration model for subsequent queries and operations. Based on the target configuration model, dynamic management and query of different types of target data can be achieved. For example, in a social network, the target configuration model can be used to enter and modify user information, as well as query information such as a user's friend list.
[0094] The embodiment of the present application obtains the first monitoring data pushed by the monitoring alarm system in batches through a preset target gateway, and the first monitoring data is determined by the monitoring alarm system based on the pre-acquired task log; performs deduplication processing on the first monitoring data in the memory of the target gateway to obtain deduplicated initial metadata; determines target metadata that meets the target storage requirements based on the initial metadata; and stores the target metadata in a preset target database. In this way, before storing the metadata, the first monitoring data is deduplicated twice, respectively, which improves the accuracy of data storage. In addition, by processing data in batches, data processing efficiency is improved, and data storage efficiency is improved.
[0095] In some embodiments of the present application, the target configuration model includes a time attribute. After writing the target vertex into the target configuration model of a preset target database, the method further includes: adding an index to the time attribute.
[0096] The embodiment of the present application adds an index to the time attribute of the metadata, which means establishing a specific index structure in the database to quickly query and filter the metadata within a specified time range. Adding an index can greatly improve query efficiency and avoid scanning the entire table to obtain results that meet the requirements, thereby reducing system resource consumption, improving response speed and concurrency performance. Specifically, its time attribute indexes include B-Tree and Hash indexes. Among them, the B-Tree index is suitable for range query and sorting operations, but its write efficiency is low; the Hash index is suitable for exact matching and equal value queries, but does not support range query and sorting, so it needs to be selected according to actual needs. At the same time, adding an index will also take up additional storage space and cause certain update and maintenance costs. When selecting an index type, it is necessary to comprehensively consider factors such as data volume, query frequency, and data update (or write) frequency.
[0097] In some embodiments of the present application, after storing the target metadata in a preset target database, the method further includes: determining a task information graph based on the target metadata and a preset relationship model.
[0098] The preset relational model is a relational model (or "metadata model") established based on the relationship between these metadata, which is used to represent the interdependence between different indicators. For example, in a server monitoring environment, there may be a certain correlation between CPU utilization and memory capacity, that is, when CPU utilization is high, memory utilization may also be high. This is a relational model that can be abstracted into a graph structure.
[0099] As an optional implementation, determining the task information graph based on the target metadata and a preset relational model includes: encapsulating the target metadata into graph data of points and edges through a preset relational model; and writing the graph data into a preset graph database.
[0100] Specifically, through a preset relationship model, the target metadata is encapsulated into graph data of points and edges, and the graph data is written to the NebulaGraph graph database, thereby forming a complete task information graph. Each node in the graph represents a specific indicator, and each edge represents the relationship between different indicators. The embodiment of the present application can help operation and maintenance personnel understand the performance bottlenecks of the system, diagnose abnormal situations and respond quickly by performing complex network analysis and graph analysis on this graph.
[0101] In order to better implement the data processing method in the embodiment of the present application, based on the data processing method, a data processing device is also provided in the embodiment of the present application, such as Figure 3 As shown, the data processing device 300 includes:
[0102] A first push unit 301 is used to obtain first monitoring data pushed in batches by the monitoring alarm system through a preset target gateway, where the first monitoring data is determined by the monitoring alarm system based on a pre-acquired task log;
[0103] A first deduplication unit 302, configured to perform deduplication processing on the first monitoring data in the memory of the target gateway to obtain deduplicated initial metadata;
[0104] A first determining unit 303 is used to determine target metadata that meets target storage requirements based on the initial metadata;
[0105] The first storage unit 304 is used to store the target metadata in a preset target database.
[0106] In some embodiments of the present application, the first determining unit 303 specifically includes:
[0107] A second determining unit, configured to determine whether the initial metadata exists in a preset cache;
[0108] a third determining unit, configured to determine, if the initial metadata exists in the cache, that the initial metadata does not meet the target storage requirement;
[0109] The fourth determining unit is configured to determine, if the initial metadata does not exist in the cache, that the initial metadata meets the target storage requirement and use the initial metadata as the target metadata.
[0110] In some embodiments of the present application, the data in the cache is stored in a bit array structure, and the second determining unit is specifically configured to:
[0111] Obtaining an offset of the initial metadata in the bit array, and using the offset as an array subscript of the initial metadata;
[0112] Based on the array index, it is determined whether the initial metadata exists in a preset cache.
[0113] In some embodiments of the present application, the preset target database includes a target configuration model, and the first storage unit 304 is specifically used to:
[0114] Taking each metadata in the target metadata as vertex data of a target vertex;
[0115] The vertex data is written into the target configuration model.
[0116] In some embodiments of the present application, the target configuration model includes a time attribute, and after writing the target vertex into the target configuration model of a preset target database, the apparatus is further configured to:
[0117] Add an index to the time attribute.
[0118] In some embodiments of the present application, after storing the target metadata in a preset target database, the device further includes:
[0119] The fifth determining unit is used to determine the task information graph based on the target metadata and a preset relationship model.
[0120] In some embodiments of the present application, the fifth determining unit is specifically configured to:
[0121] Encapsulating the target metadata into graph data of points and edges through a preset relational model;
[0122] The graphic data is written into a preset graphic database.
[0123] The embodiment of the present application uses a first push unit 301, which is used to obtain the first monitoring data pushed in batches by the monitoring alarm system through a preset target gateway, and the first monitoring data is determined by the monitoring alarm system based on the pre-acquired task log; a first deduplication unit 302, which is used to deduplicate the first monitoring data in the memory of the target gateway to obtain the deduplicated initial metadata; a first determination unit 303, which is used to determine the target metadata that meets the target storage requirements based on the initial metadata; a first storage unit 304, which is used to store the target metadata in a preset target database. In this way, before storing the metadata, the first monitoring data is deduplicated twice, respectively, which improves the accuracy of data storage. In addition, by processing data in batches, the data processing efficiency is improved, and the data storage efficiency is improved.
[0124] In addition to the above-described data processing method and device, the embodiment of the present application further provides a computer device, which integrates any data processing device provided in the embodiment of the present application, and the computer device includes:
[0125] one or more processors;
[0126] Memory; and
[0127] One or more applications, wherein the one or more applications are stored in the memory and are configured to be executed by the processor to perform the operations of any method described in any of the above data processing method embodiments.
[0128] The present application also provides a computer device that integrates any data processing device provided in the present application. Figure 4 As shown, it shows a schematic diagram of the structure of the computer device involved in the embodiment of the present application, specifically:
[0129] The computer device may include components such as a processor 401 with one or more processing cores, a storage unit 402 with one or more computer-readable storage media, a power supply 403, and an input unit 404. Those skilled in the art will appreciate that Figure 4 The computer device structure shown in the figure does not constitute a limitation on the computer device, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently. Among them:
[0130] The processor 401 is the control center of the computer device. It uses various interfaces and lines to connect various parts of the entire computer device. By running or executing software programs and / or modules stored in the storage unit 402 and calling data stored in the storage unit 402, it executes various functions of the computer device and processes data, thereby monitoring the computer device as a whole. Optionally, the processor 401 may include one or more processing cores; preferably, the processor 401 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface, and application programs, and the modem processor mainly processes wireless communications. It is understandable that the above-mentioned modem processor may not be integrated into the processor 401.
[0131] The storage unit 402 can be used to store software programs and modules. The processor 401 executes various functional applications and data processing by running the software programs and modules stored in the storage unit 402. The storage unit 402 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area may store data created according to the use of the computer device, etc. In addition, the storage unit 402 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage devices. Accordingly, the storage unit 402 may also include a memory controller to provide the processor 401 with access to the storage unit 402.
[0132] The computer device also includes a power supply 403 for supplying power to various components. Preferably, the power supply 403 can be logically connected to the processor 401 through a power management system, so as to manage charging, discharging, and power consumption through the power management system. The power supply 403 can also include any components such as one or more DC or AC power supplies, recharging systems, power failure detection circuits, power converters or inverters, and power status indicators.
[0133] The computer device may further include an input unit 404, which may be used to receive input digital or character information and generate keyboard, mouse, joystick, optical or trackball signal input related to user settings and function control.
[0134] Although not shown, the computer device may also include a display unit, etc., which will not be described in detail here. Specifically, in the embodiment of the present application, the processor 401 in the computer device will load the executable file corresponding to the process of one or more application programs into the storage unit 402 according to the following instructions, and the processor 401 will run the application program stored in the storage unit 402, thereby realizing various functions, as follows:
[0135] The first monitoring data pushed in batches by the monitoring and alarm system is obtained through a preset target gateway, where the first monitoring data is determined by the monitoring and alarm system based on a pre-acquired task log; the first monitoring data is deduplicated in the memory of the target gateway to obtain deduplicated initial metadata; based on the initial metadata, target metadata that meets target storage requirements is determined; and the target metadata is stored in a preset target database.
[0136] The embodiment of the present application obtains the first monitoring data pushed by the monitoring alarm system in batches through a preset target gateway, and the first monitoring data is determined by the monitoring alarm system based on the pre-acquired task log; performs deduplication processing on the first monitoring data in the memory of the target gateway to obtain deduplicated initial metadata; determines target metadata that meets the target storage requirements based on the initial metadata; and stores the target metadata in a preset target database. In this way, before storing the metadata, the first monitoring data is deduplicated twice, respectively, which improves the accuracy of data storage. In addition, by processing data in batches, data processing efficiency is improved, and data storage efficiency is improved.
[0137] To this end, an embodiment of the present application provides a computer-readable storage medium, which may include: a read-only memory (ROM), a random access memory (RAM), a disk or an optical disk, etc. The computer-readable storage medium stores a plurality of instructions, which can be loaded by a processor to execute the steps in any data processing method provided in the embodiment of the present application. For example, the instruction can execute the following steps:
[0138] The first monitoring data pushed in batches by the monitoring and alarm system is obtained through a preset target gateway, where the first monitoring data is determined by the monitoring and alarm system based on a pre-acquired task log; the first monitoring data is deduplicated in the memory of the target gateway to obtain deduplicated initial metadata; based on the initial metadata, target metadata that meets target storage requirements is determined; and the target metadata is stored in a preset target database.
[0139] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.
[0140] The above is a detailed introduction to a data processing method, device, computer equipment and computer-readable storage medium provided in the embodiments of the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea; at the same time, for technical personnel in this field, according to the ideas of the present application, there will be changes in the specific implementation methods and application scopes. In summary, the content of this specification should not be understood as a limitation on the present application.
Claims
1. A data processing method, characterized in that: The method comprises: Acquire first monitoring data pushed in batches by the monitoring and alarm system through a preset target gateway, where the first monitoring data is determined by the monitoring and alarm system based on a pre-acquired task log; Deduplication processing is performed on the first monitoring data in the memory of the target gateway to obtain deduplication initial metadata; Based on the initial metadata, determining target metadata that meets target storage requirements; The target metadata is stored in a preset target database.
2. The data processing method according to claim 1, characterized in that: The determining, based on the initial metadata, target metadata that meets the target storage requirement includes: Determining whether the initial metadata exists in a preset cache; If the initial metadata exists in the cache, determining that the initial metadata does not meet the target storage requirement; If the initial metadata does not exist in the cache, it is determined that the initial metadata meets the target storage requirement, and the initial metadata is used as the target metadata.
3. The data processing method according to claim 2, characterized in that: The data in the cache is stored in a bit array structure, and the determining whether the initial metadata exists in the preset cache includes: Obtaining an offset of the initial metadata in the bit array, and using the offset as an array subscript of the initial metadata; Based on the array index, it is determined whether the initial metadata exists in a preset cache.
4. The data processing method according to claim 1, characterized in that: The preset target database includes a target configuration model, and storing the target metadata in the preset target database includes: Taking each metadata in the target metadata as vertex data of a target vertex; The vertex data is written into the target configuration model.
5. The data processing method according to claim 4, characterized in that: The target configuration model includes a time attribute. After writing the target vertex into the target configuration model of a preset target database, the method further includes: An index is added to the time attribute.
6. The data processing method according to claim 1, characterized in that: After storing the target metadata in a preset target database, the method further includes: Based on the target metadata and the preset relationship model, a task information graph is determined.
7. The data processing method according to claim 6, characterized in that: The determining of the task information graph based on the target metadata and the preset relationship model includes: Encapsulating the target metadata into graph data of points and edges through a preset relational model; The graphic data is written into a preset graphic database.
8. A data processing device, characterized in that: The device comprises: A first push unit, configured to obtain first monitoring data pushed in batches by the monitoring and alarm system through a preset target gateway, wherein the first monitoring data is determined by the monitoring and alarm system based on a pre-acquired task log; A first deduplication unit, configured to perform deduplication processing on the first monitoring data in a memory in the target gateway to obtain deduplicated initial metadata; A first determining unit, configured to determine target metadata that meets target storage requirements based on the initial metadata; The first storage unit is used to store the target metadata in a preset target database.
9. A computer device, characterized in that: The computer device comprises: one or more processors; Memory; and One or more application programs, wherein the one or more application programs are stored in the memory and configured to be executed by the processor to implement the data processing method according to any one of claims 1 to 7.
10. A computer-readable storage medium, characterized in that: A computer program is stored thereon, and the computer program is loaded by a processor to execute the steps in the data processing method according to any one of claims 1 to 7.
