Log management method and device based on swan gap distribution, terminal and medium

Abstract

The invention discloses a log management method and device based on swan gap distribution, a terminal and a medium. The method comprises the steps that log data generated in the operation process of multiple devices are collected; data analysis is conducted on the log data, the log data type of the log data is determined, and the log data type comprises the same log data of the multiple devices and specific log data of each device; according to the log data type, a storage scene of the log data is distributed, and the storage scene comprises a distributed storage system and an equipment local storage system. Therefore, the same log data and the specific log data of the log data generated by the multiple devices can be extracted and distributed to different storage scenes respectively, effective management of the log data is achieved, and the storage space needed by the log data is reduced.

Description

Technical Field

[0001] The present invention relates to the field of log management technology, and in particular to a log management method, device, terminal and medium based on Hongmeng distributed. Background Art

[0002] At present, the log data generated by system equipment often needs to be stored with the help of log storage servers. However, the log storage servers only store the collected log data from multiple devices or services directly in the file system according to date, device or service classification. It is impossible to integrate the log data of multiple devices or services, and it is impossible to identify and extract the log data of the same format from different devices or services for unified storage, resulting in a waste of storage space on existing log storage servers.

[0003] Therefore, the prior art still needs to be improved and enhanced. Summary of the invention

[0004] The technical problem to be solved by the present invention is that, in response to the above-mentioned defects of the prior art, a log management method, device, terminal and medium based on Hongmeng distribution are provided, aiming to solve the problem in the prior art that the log storage server cannot recognize and extract log data of the same format from different devices or services for unified storage, resulting in waste of storage space of the existing log storage server.

[0005] In order to solve the above technical problems, the technical solution adopted by the present invention is as follows:

[0006] In a first aspect, the present invention provides a log management method based on Hongmeng distributed, wherein the method comprises:

[0007] Collect log data generated by multiple devices during operation;

[0008] Performing data analysis on the log data to determine a log data type of the log data, where the log data type includes the same log data of the multiple devices and unique log data of each device;

[0009] According to the type of the log data, a storage scenario of the log data is allocated, and the storage scenario includes a distributed storage system and a device local storage system.

[0010] In one implementation, the collecting of log data generated by multiple devices during operation includes:

[0011] Monitor the cached data volume of log data generated by each device in real time;

[0012] When the cache data volume reaches a preset cache threshold, log data generated by each device during operation is collected based on a distributed soft bus.

[0013] In one implementation, the performing data analysis on the log data to determine the log data type of the log data, where the log data type includes the same log data of the multiple devices and the unique log data of each device, includes:

[0014] Splitting the log data to obtain data units of the log data;

[0015] Analyzing the data units using a data difference comparison algorithm to determine identical data units and unique data units;

[0016] The log data type of the log data is determined according to the same data unit and the unique data unit.

[0017] In one implementation, the splitting the log data to obtain data units of the log data includes:

[0018] Determining a log format of the log data, wherein the log format includes a text format, a timestamp format, a structured format, and a process format;

[0019] The log data is split according to the log format to obtain data units of the log data.

[0020] In one implementation, determining the log data type of the log data according to the same data unit and the unique data unit includes:

[0021] When the data unit is the same data unit, determining that the log data type of the log data is the same log data of the multiple devices;

[0022] When the data unit is the specific data unit, it is determined that the log data type of the log data is specific log data of each device.

[0023] In one implementation, the storage scenario of the log data is allocated according to the type of the log data, and the storage scenario includes a distributed storage system and a device local storage system, including:

[0024] Determine, according to the log data type, an application scenario of the log data type;

[0025] According to the application scenario, a storage scenario of the log data is allocated.

[0026] In one implementation, the storage scenario of the log data is allocated according to the type of the log data, and the storage scenario includes a distributed storage system and a device local storage system, and further includes:

[0027] When the log data type is the same log data of the multiple devices, determining that the storage scenario of the log data is a distributed storage system;

[0028] When the log data type is unique log data of each device, it is determined that the storage scenario of the log data is a local storage system of the device.

[0029] In a second aspect, an embodiment of the present invention further provides a log management device based on Hongmeng distributed, wherein the device includes:

[0030] The log data collection module is used to collect the log data generated by multiple devices during operation;

[0031] A log data type determination module, used to perform data analysis on the log data to determine the log data type of the log data, wherein the log data type includes the same log data of the multiple devices and the unique log data of each device;

[0032] A storage scenario allocation module is used to allocate storage scenarios for the log data according to the type of the log data, and the storage scenarios include a distributed storage system and a device local storage system.

[0033] In the third aspect, an embodiment of the present invention also provides a terminal, wherein the terminal includes a memory, a processor, and a log management program based on HarmonyOS distributed stored in the memory and runnable on the processor, and when the processor executes the log management program based on HarmonyOS distributed, it implements the steps of the log management method based on HarmonyOS distributed of any one of the above-mentioned schemes.

[0034] In the fourth aspect, an embodiment of the present invention also provides a computer-readable storage medium, wherein a log management program based on HarmonyOS distributed is stored on the computer-readable storage medium, and when the log management program based on HarmonyOS distributed is executed by a processor, the steps of the log management method based on HarmonyOS distributed described in any one of the above schemes are implemented.

[0035] Beneficial effect: Compared with the prior art, the present invention provides a log management method, device, terminal and medium based on Hongmeng distribution, which first collects the log data generated by multiple devices during operation. Then, the log data is analyzed to determine the log data type of the log data, and the log data type includes the same log data of the multiple devices and the unique log data of each device. Finally, according to the log data type, the storage scenario of the log data is allocated, and the storage scenario includes a distributed storage system and a device local storage system. Therefore, the present invention can extract the same log data and unique log data of the log data generated by multiple devices, and allocate them to different storage scenarios respectively, thereby realizing the effective management of the log data and reducing the storage space required for the log data. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 A flowchart of a specific implementation method of the log management method based on Hongmeng distribution provided in an embodiment of the present invention.

[0037] Figure 2 A functional schematic diagram of a log management device based on Hongmeng distribution provided in an embodiment of the present invention.

[0038] Figure 3 A functional block diagram of a terminal provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0039] In order to make the purpose, technical solution and effect of the present invention clearer and more specific, the present invention is further described in detail below with reference to the accompanying drawings and examples.

[0040] Those skilled in the art will appreciate that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. Unless specifically stated, the singular forms "one", "said", and "the" used herein may also include plural forms, and the term "comprising" used in the specification of the present invention refers to the presence of the features, integers, steps, operations, elements, and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or their groups. It should be understood that when an element is "connected" or "coupled" to another element, it may be directly connected or coupled to other elements, or there may also be intermediate elements. In addition, the "connection" or "coupling" used herein may include wireless connection or wireless coupling. The term "and / or" used herein includes all or any unit and all combinations of one or more associated listed items.

[0041] The present embodiment provides a log management method, device, terminal and medium based on Hongmeng distribution. When implemented specifically, the present embodiment first collects log data generated by multiple devices during operation. Then, the log data is analyzed to determine the log data type of the log data, and the log data type includes the same log data of the multiple devices and the unique log data of each device. Finally, according to the log data type, the storage scenario of the log data is allocated, and the storage scenario includes a distributed storage system and a device local storage system. Therefore, the present embodiment can extract the same log data and unique log data of the log data generated by multiple devices, and allocate them to different storage scenarios respectively, thereby realizing the effective management of the log data and reducing the storage space required for the log data.

[0042] Exemplary Methods

[0043] The log management method based on Hongmeng distribution in this embodiment can be applied to a terminal, which can be a preset mobile terminal such as a mobile phone, a tablet, a computer, or an intelligent device such as a smart TV. In this embodiment, the terminal can be an external device connected to the log management device based on Hongmeng distribution, or it can be a device built into the log management device based on Hongmeng distribution. Figure 1 As shown in , the log management method based on Hongmeng distributed in this embodiment includes the following steps:

[0044] Step S100: Collect log data generated by multiple devices during operation.

[0045] Since the device storage space is limited and most of the current log storage is remote log storage servers, additional device resources are required to assist in completing log storage. The remote log storage server only stores the collected log data from multiple devices or services directly in the file system according to date, device or service classification, and cannot integrate the log data of multiple devices or services, resulting in a waste of storage space of the log storage server. Therefore, this embodiment first needs to collect log data generated from multiple different devices during operation, and then further analyze and integrate various log data based on the Hongmeng distributed system.

[0046] In one implementation, step S100 of this embodiment specifically includes the following steps:

[0047] Step S101, real-time monitoring of the cached data volume of log data generated by each device;

[0048] Step S102: When the cache data amount reaches a preset cache threshold, log data generated by each device during operation is collected based on the distributed soft bus.

[0049] In order to reduce network overhead, reduce server load, ensure data integrity and optimize resource use, the present embodiment will monitor the cache data volume of the log data generated by each device in real time, and when the cache data volume reaches the preset cache threshold, collect the log data generated by each device during operation. Wherein, the preset cache threshold can be set by the staff according to the experience requirement. For example, each network transmission will have a certain overhead (such as establishing a connection, transmission control information, etc.), and setting a preset cache threshold to achieve batch transmission can significantly reduce these overheads, compared to sending immediately every time a log is generated, batch transmission can more efficiently utilize network bandwidth, especially in a high concurrency environment. For another example, if the preset cache threshold is not set in advance, as long as new log data is generated, it will be sent to the server immediately, causing the server to frequently receive and process a large number of small data packets, which will increase the CPU and I / O load of the server. On the contrary, setting a preset cache threshold in the present embodiment to achieve batch transmission can reduce the number of times the server processes requests, help smooth network traffic, and avoid network congestion or server overload caused by a large amount of sudden log data.

[0050] At the same time, this embodiment is based on a distributed soft bus to collect log data generated by each device during operation. Hongmeng distributed technology is a system architecture that aims to achieve a seamless collaborative experience across devices, and the distributed soft bus is a specific implementation of Hongmeng distributed technology. The distributed soft bus has the characteristics of low latency (by optimizing the network transmission protocol to achieve low-latency communication between devices), high bandwidth (supporting high-speed data transmission, suitable for scenarios with large amounts of data exchange) and supporting multi-hop networking (supporting multi-hop networking between devices, expanding network coverage). Compared with traditional distributed storage, which pays more attention to the scale, performance and reliability of data, Hongmeng distributed storage emphasizes the collaborative experience across devices and the consistency and convenience of user data. In terms of data access, Hongmeng distributed storage realizes seamless switching between devices, and users can directly access data stored on other devices from any terminal, while traditional distributed storage is usually oriented to server-side applications, and users have less direct access. In terms of application scenarios, traditional distributed storage is often used in large-scale data processing scenarios such as data centers and cloud computing, while Hongmeng distributed storage mainly serves the smart device ecosystem, such as mobile phones, tablets, smart homes, etc., to enhance users' cross-device usage experience. In terms of technical implementation, Hongmeng uses a distributed soft bus and a unified device management system to achieve a lower latency and more efficient data sharing mechanism, which is different from traditional distributed storage that relies on network protocols and complex middleware.

[0051] Specifically, the distributed soft bus will also use a unified log format to collect log data to ensure that all devices and services follow the same format (such as JSON format), so that the generated logs all use a unified format, which facilitates the analysis and processing of log data in subsequent processes of this embodiment.

[0052] Further, after the present embodiment collects the log data generated by each device during operation based on the distributed soft bus, the log data will also be sent to the central processing node, and the central processing node will then uniformly complete the subsequent data analysis process of the log data. Optionally, the distributed soft bus can use a log collection tool (such as Filebeat, Fluentd) to send the log data to the central processing node. At the same time, the central processing node can use tools such as Kibana for unified monitoring and analysis.

[0053] Step S200: performing data analysis on the log data to determine a log data type of the log data, where the log data type includes the same log data of the multiple devices and unique log data of each device.

[0054] Since the present embodiment allocates different storage scenarios of log data according to the log data type, the present embodiment needs to perform data analysis on the log data first to determine the log data type of the log data.

[0055] In one implementation, step S200 of this embodiment specifically includes the following steps:

[0056] Step S201, splitting the log data to obtain data units of the log data;

[0057] Step S202: Analyze the data units using a data difference comparison algorithm to determine identical data units and unique data units;

[0058] Step S203: Determine the log data type of the log data according to the same data unit and the unique data unit.

[0059] In specific implementation, this embodiment needs to split the collected log data to obtain finer-grained data units, thereby helping to better manage and analyze log information.

[0060] In one implementation, step S201 of this embodiment specifically includes the following steps:

[0061] Step S2011, determining the log format of the log data, wherein the log format includes a text format, a timestamp format, a structured format and a process format;

[0062] Step S2012: split the log data according to the log format to obtain data units of the log data.

[0063] In general, the log format of log data includes text format, timestamp format, structured format and process format, and different log formats have different splitting modes. In order to improve the splitting efficiency of log data, the present embodiment first needs to determine the log format of log data, then, split the log data according to the current log format, such as, when determining that the current log format is text format (i.e., each log record is a line of text), adopt the splitting mode of reading and processing each log record line by line. When determining that the current log format is timestamp format (i.e., recorded by time), according to the timestamp field in the log, the log data is split by time period (such as every hour, every day). When determining that the current log format is structured format (such as JSON format), by parsing the structured fields in the log data, it is split into multiple independent data units. When determining that the current log format is process format (i.e., log data generated by multiple devices or multiple processes), according to the device ID or process ID field in the log, the log data is split by device or process.

[0064] It can be seen that the method of splitting log data according to the current log format in this embodiment can more accurately and quickly obtain the data unit of the current log data. Among them, the data unit is the smallest data unit, which specifically includes information such as level, information, process name, log field, etc.

[0065] Next, this embodiment uses the data units obtained by analyzing the data difference comparison algorithm to determine the same data units and unique data units. Among them, the data difference comparison algorithm includes data difference analysis, similarity measurement and difference detection algorithm. Data difference analysis is to identify the differences between two or more groups of data through statistical methods. Similarity measurement is to use distance or similarity indicators (such as Euclidean distance, cosine similarity, etc.) to quantify the differences between data. Difference detection algorithm, that is, change point detection algorithm used for time series data analysis, is used to identify abnormal changes in data streams.

[0066] In this embodiment, data difference analysis is mainly used. For example, the data units with the same name (such as level, information) generated by each device are compared one by one. When the data units of all devices are the same, the data units are determined to be the same data units. The data units with the same name (such as log fields, process names) generated by each device are compared one by one. When the data units of all devices are different, the data units are determined to be unique data units. Finally, this embodiment can determine the log data type of log data based on the same data units and unique data units.

[0067] In one implementation, step S203 of this embodiment specifically includes the following steps:

[0068] Step S2031: when the data unit is the same data unit, determining that the log data type of the log data is the same log data of the multiple devices;

[0069] Step S2032: When the data unit is the specific data unit, determine that the log data type of the log data is specific log data of each device.

[0070] That is to say, when the data units split out in this embodiment present the same data units on multiple devices, it can be determined that the log data type of the log data is the same log data of multiple devices. When the data units split out present unique data units on multiple devices, it can be determined that the log data type of the log data is unique log data of each device.

[0071] Step S300: Allocate storage scenarios for the log data according to the type of the log data, where the storage scenarios include a distributed storage system and a device local storage system.

[0072] Specifically, this embodiment can reasonably distribute the obtained log data types to a distributed storage system or a local storage system of a device, so as to achieve effective management of the log data and reduce the storage space required for the log data.

[0073] In one implementation, step S300 of this embodiment specifically includes the following steps:

[0074] Step S301: Determine the application scenario of the log data type according to the log data type;

[0075] Step S302: Allocate a storage scenario for the log data according to the application scenario.

[0076] Specifically, the application scenarios corresponding to log data of different log data types are also different. This embodiment first determines the application scenario of the log data type and then allocates the storage scenario of the log data, which can improve the subsequent retrieval efficiency of the stored log data.

[0077] Generally speaking, application scenarios are mainly obtained through comprehensive analysis of data volume, access frequency, security and compliance, cost, and specific application requirements.

[0078] In terms of data volume, if the amount of log data is very large, and the log files generated every day reach the GB or even TB level, then it is more appropriate to use a distributed storage system, which can provide better scalability and fault tolerance. If the amount of log data is small and there is no need to store a large amount of historical data for a long time, then the local storage system of the device is usually sufficient. In terms of access frequency, if log data needs to be frequently queried and analyzed, such as real-time monitoring or troubleshooting, distributed storage systems can usually provide higher read and write performance and lower latency. If log data is mainly used for archiving or auditing and is not frequently accessed, then the local storage system of the device may be more economical. In terms of security and compliance, if the log data contains sensitive information and needs to meet strict compliance requirements (such as GDPR, HIPAA, etc.), distributed storage systems usually provide stronger security and access control mechanisms. If it is relatively less sensitive data, then the local storage system of the device may be secure enough, especially through appropriate encryption and access control measures. In terms of cost, the initial investment and maintenance cost of the distributed storage system is high, but as the amount of data grows, the unit storage cost may decrease. If the budget is limited, you can choose the local storage system of the device. If the budget allows and large-scale data needs to be processed, a distributed storage system is a better choice. In terms of specific application requirements, if real-time log analysis is required, a distributed storage system combined with a stream processing platform (such as Kafka, Flink) can provide powerful real-time processing capabilities. If only regular offline analysis is required, then the device local storage system plus batch processing tools (such as Hadoop, Spark) are sufficient.

[0079] Therefore, this embodiment first obtains the application scenario of the log data type by comprehensively analyzing the data volume, access frequency, security and compliance, cost and specific application requirements, and then further allocates the storage scenario of the log data.

[0080] In one implementation, step S300 of this embodiment further includes the following steps:

[0081] Step S303: When the log data type is the same log data of the multiple devices, determining that the storage scenario of the log data is a distributed storage system;

[0082] Step S304: When the log data type is unique log data of each device, determine that the storage scenario of the log data is a local storage system of the device.

[0083] Specifically, the storage scenarios in this embodiment include a distributed storage system and a device local storage system.

[0084] When the log data type is the same log data of multiple devices, this embodiment will send the same log data to a distributed storage system for unified storage, such as Hongmeng's distributed file system or distributed database. After this embodiment sends the same log data to a distributed storage system for unified storage, it can deduplicate the duplicated identical log data, achieving the beneficial technical effect of saving storage space and reducing storage overhead.

[0085] When the log data type is unique log data of each device, this embodiment will send the unique log data back to the corresponding device, for example, stored in a local file system (such as SQLite), a local database, or memory storage of the local device.

[0086] Optionally, this embodiment also regularly monitors and maintains the distributed storage system and the device local storage system. For example, monitoring tools (such as Prometheus, Grafana) are used to monitor the health status of the Elasticsearch cluster and local storage to ensure that the subsequently generated log data can continue to be distributed there.

[0087] In summary, this embodiment provides a log management method based on Hongmeng distribution. When it is implemented specifically, this embodiment first collects the log data generated by multiple devices during operation. Then, the log data is analyzed to determine the log data type of the log data, and the log data type includes the same log data of the multiple devices and the unique log data of each device. Finally, according to the log data type, the storage scenario of the log data is allocated, and the storage scenario includes a distributed storage system and a device local storage system. Therefore, this embodiment can extract the same log data and unique log data of the log data generated by multiple devices, and allocate them to different storage scenarios respectively, thereby realizing the effective management of the log data and reducing the storage space required for the log data.

[0088] Exemplary Devices

[0089] Based on the above embodiments, the present invention also provides a log management device based on Hongmeng distributed system, such as Figure 2As shown in , the device includes: a log data collection module 10, a log data type determination module 20, and a storage scenario allocation module 30. Specifically, the log data collection module 10 is used to collect log data generated by multiple devices during operation. The log data type determination module 20 is used to perform data analysis on the log data to determine the log data type of the log data, and the log data type includes the same log data of the multiple devices and the unique log data of each device. The storage scenario allocation module 30 is used to allocate the storage scenario of the log data according to the log data type, and the storage scenario includes a distributed storage system and a device local storage system.

[0090] In one implementation, the log data collection module 10 includes:

[0091] A cache data volume determination unit, used to monitor in real time the cache data volume of log data generated by each device;

[0092] The log data collection unit is used to collect the log data generated by each device during operation based on the distributed soft bus when the cache data amount reaches a preset cache threshold.

[0093] In one implementation, the log data type determination module 20 includes:

[0094] A data unit acquisition unit, used to split the log data to obtain data units of the log data;

[0095] A data unit analysis unit, used to analyze the data units using a data difference comparison algorithm to determine identical data units and unique data units;

[0096] The log data type determining unit is used to determine the log data type of the log data according to the same data unit and the unique data unit.

[0097] In one implementation, the data unit acquisition unit includes:

[0098] A log format determination subunit, used to determine the log format of the log data, wherein the log format includes a text format, a timestamp format, a structured format and a process format;

[0099] The data unit determination subunit is used to split the log data according to the log format to obtain data units of the log data.

[0100] In one implementation, the log data type determination unit includes:

[0101] A first log data type determination subunit, configured to determine, when the data unit is the same data unit, that the log data type of the log data is the same log data of the plurality of devices;

[0102] The second log data type determination subunit is used to determine, when the data unit is the specific data unit, that the log data type of the log data is specific log data of each device.

[0103] In one implementation, the storage scene allocation module 30 includes:

[0104] An application scenario determining unit, configured to determine an application scenario of the log data type according to the log data type;

[0105] A storage scenario allocation unit is used to allocate the storage scenario of the log data according to the application scenario.

[0106] In one implementation, the storage scene allocation module 30 further includes:

[0107] A first storage scenario allocation unit, configured to determine that the storage scenario of the log data is a distributed storage system when the log data type is the same log data of the multiple devices;

[0108] The second storage scenario allocation unit is configured to determine that the storage scenario of the log data is a local storage system of the device when the type of the log data is the unique log data of each device.

[0109] The working principles of each module in the Hongmeng distributed log management device of this embodiment are the same as the principles of each step in the above method embodiment, and will not be repeated here.

[0110] Based on the above embodiment, the present invention further provides a terminal, the principle block diagram of the terminal can be as follows: Figure 3 The terminal may include one or more processors 100 ( Figure 3 Only one is shown), a memory 101 and a computer program 102 stored in the memory 101 and executable on one or more processors 100, for example, a log management program based on Hongmeng distribution. When one or more processors 100 execute the computer program 102, the various steps in the embodiment of the log management method based on Hongmeng distribution can be implemented. Alternatively, when one or more processors 100 execute the computer program 102, the functions of each module / unit in the embodiment of the log management device based on Hongmeng distribution can be implemented, which is not limited here.

[0111] In one embodiment, the processor 100 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0112] In one embodiment, the memory 101 may be an internal storage unit of an electronic device, such as a hard disk or memory of the electronic device. The memory 101 may also be an external storage device of the electronic device, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the electronic device. Further, the memory 101 may also include both an internal storage unit of the electronic device and an external storage device. The memory 101 is used to store computer programs and other programs and data required by the terminal. The memory 101 may also be used to temporarily store data that has been output or is to be output.

[0113] Those skilled in the art will understand that Figure 3 The principle block diagram shown in the figure is only a block diagram of a partial structure related to the scheme of the present invention, and does not constitute a limitation on the terminal to which the scheme of the present invention is applied. The specific terminal may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0114] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, operating database or other media used in the embodiments provided by the present invention may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double operational data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0115] In summary, the present invention discloses a log management method, device, terminal and medium based on Hongmeng distribution, the method comprising: collecting log data generated by multiple devices during operation; performing data analysis on the log data to determine the log data type of the log data, the log data type comprising the same log data of the multiple devices and the unique log data of each device; according to the log data type, allocating the storage scenario of the log data, the storage scenario comprising a distributed storage system and a device local storage system. Therefore, the present invention can extract the same log data and unique log data of the log data generated by multiple devices, and allocate them to different storage scenarios respectively, thereby realizing the effective management of the log data and reducing the storage space required for the log data.

[0116] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A log management method based on Hongmeng distribution, characterized in that: The method comprises: Collect log data generated by multiple devices during operation; Performing data analysis on the log data to determine a log data type of the log data, where the log data type includes the same log data of the multiple devices and unique log data of each device; According to the type of the log data, a storage scenario of the log data is allocated, and the storage scenario includes a distributed storage system and a device local storage system.

2. According to the Hongmeng distributed log management method according to claim 1, it is characterized in that: The log data generated by multiple devices during operation is collected, including: Monitor the cached data volume of log data generated by each device in real time; When the cache data volume reaches a preset cache threshold, log data generated by each device during operation is collected based on a distributed soft bus.

3. According to the Hongmeng distributed log management method according to claim 1, it is characterized in that: The performing data analysis on the log data to determine the log data type of the log data, wherein the log data type includes the same log data of the multiple devices and the unique log data of each device, includes: Splitting the log data to obtain data units of the log data; Analyzing the data units using a data difference comparison algorithm to determine identical data units and unique data units; The log data type of the log data is determined according to the same data unit and the unique data unit.

4. According to the Hongmeng distributed log management method according to claim 3, it is characterized in that: The step of splitting the log data to obtain data units of the log data includes: Determining a log format of the log data, wherein the log format includes a text format, a timestamp format, a structured format, and a process format; The log data is split according to the log format to obtain data units of the log data.

5. According to the Hongmeng distributed log management method according to claim 3, it is characterized in that: The determining the log data type of the log data according to the same data unit and the unique data unit includes: When the data unit is the same data unit, determining that the log data type of the log data is the same log data of the multiple devices; When the data unit is the specific data unit, it is determined that the log data type of the log data is specific log data of each device.

6. The log management method based on Hongmeng distribution according to claim 1 is characterized in that: The storage scenario of the log data is allocated according to the log data type, and the storage scenario includes a distributed storage system and a device local storage system, including: Determine, according to the log data type, an application scenario of the log data type; According to the application scenario, a storage scenario of the log data is allocated.

7. The log management method based on Hongmeng distribution according to claim 6 is characterized in that: The allocating storage scenarios of the log data according to the log data type, wherein the storage scenarios include a distributed storage system and a device local storage system, further comprising: When the log data type is the same log data of the multiple devices, determining that the storage scenario of the log data is a distributed storage system; When the log data type is unique log data of each device, it is determined that the storage scenario of the log data is a local storage system of the device.

8. A log management device based on Hongmeng distributed system, characterized in that: The device comprises: The log data collection module is used to collect the log data generated by multiple devices during operation; A log data type determination module, used to perform data analysis on the log data to determine the log data type of the log data, wherein the log data type includes the same log data of the multiple devices and the unique log data of each device; A storage scenario allocation module is used to allocate storage scenarios for the log data according to the type of the log data, and the storage scenarios include a distributed storage system and a device local storage system.

9. A terminal, characterized in that: The terminal includes a memory, a processor, and a log management program based on HarmonyOS distributed stored in the memory and executable on the processor. When the processor executes the log management program based on HarmonyOS distributed, the steps of the log management method based on HarmonyOS distributed as described in any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a log management program based on HarmonyOS distribution. When the log management program based on HarmonyOS distribution is executed by the processor, the steps of the log management method based on HarmonyOS distribution as described in any one of claims 1-7 are implemented.

Images