Voice platform full-link log management analysis method and system

Abstract

The invention discloses a voice platform full-link log management analysis method and system, and relates to the technical field of information, and the method comprises the steps: obtaining log data; storing the log data by using a log storage engine; analyzing the log data through a machine learning algorithm; visually displaying the log data; and log security control. According to the method, a distributed architecture is adopted, and a collector and a storage engine are deployed at a log generation source, so that comprehensive collection and storage of full-link log data are realized, real-time monitoring and analysis are ensured, and data collection efficiency and data comprehensiveness are improved; by utilizing the efficient storage engine, the system can process a large amount of log data, and the query efficiency is improved; by introducing a machine learning algorithm, intelligent analysis and mining of log data are realized, and potential problems and optimization points in the log data are found.

Description

Technical Field

[0001] The present invention relates to the field of information technology, and in particular to a full-link log management and analysis method and system for a voice platform. Background Art

[0002] With the rapid development of information technology, especially the rise of cloud computing, big data, and artificial intelligence, voice communication platforms have received unprecedented attention. As an important form of modern communication, the stability and user experience of voice platforms are crucial. In this context, log management and analysis technology has become the key to improving the service quality of voice platforms. Traditional log management methods mainly focus on log collection and simple analysis, but in large-scale and complex voice platforms, these methods often cannot meet the needs of efficient management, in-depth analysis, and visual display of log data.

[0003] Existing technologies have many deficiencies in the full-link log management and analysis of voice platforms. First, log data collection is not comprehensive and it is difficult to cover all links of the full link. Second, log analysis is inefficient and it is impossible to track the source of the problem in real time. In addition, the security and compliance issues of log data have not received sufficient attention. More importantly, existing technologies lack intelligent analysis and mining of log data, making it difficult to discover potential problems and optimization points. These deficiencies seriously restrict the service quality and operation and maintenance efficiency of the voice platform. Summary of the invention

[0004] In view of the above-mentioned problems, the present invention is proposed.

[0005] Therefore, the technical problem solved by the present invention is: the existing voice platform full-link log management and analysis method has the problems of incomplete log data collection, low analysis efficiency, low security and compliance, and how to achieve efficient, comprehensive management and in-depth analysis of the voice platform full-link log.

[0006] To solve the above technical problems, the present invention provides the following technical solutions: a full-link log management and analysis method for a voice platform, comprising: obtaining log data; storing log data using a log storage engine; analyzing log data through a machine learning algorithm; visualizing log data; and controlling log security.

[0007] As a preferred solution of the voice platform full-link log management and analysis method described in the present invention, wherein: the acquisition of log data includes: defining a unified log format; defining log information collection rules; and deploying a log collector.

[0008] As a preferred solution of the full-link log management and analysis method of the voice platform described in the present invention, the deployment of the log collector includes: configuring the log collector; deploying the configured log collector to each log generation source of the distributed system to obtain log data.

[0009] As a preferred solution of the full-link log management and analysis method of the voice platform described in the present invention, the log storage engine includes: clustered deployment of the log storage engine; adding an index strategy; implementing shard replication; and performing data writing and querying.

[0010] As a preferred solution of the voice platform full-link log management and analysis method described in the present invention, the storing of log data includes: compressing the log data by a compression algorithm; encrypting the compressed data by an encryption algorithm; and writing the encrypted data storage into a storage engine.

[0011] As a preferred solution of the full-link log management and analysis method of the voice platform described in the present invention, the analysis of log data through a machine learning algorithm includes: preprocessing the log data; model training of a suitable machine learning algorithm; real-time analysis and prediction of log data; batch analysis and prediction of historical data; anomaly detection and early warning of log data; trend prediction of log data and optimization of the system.

[0012] As a preferred solution of the voice platform full-link log management and analysis method described in the present invention, the log security control includes: access control of the log system; authority management of system users; and data encryption of log data.

[0013] Another object of the present invention is to provide a full-link log management and analysis system for a voice platform, which can ensure the integrity and real-time performance of full-link log data by constructing a distributed log collection and analysis system and adopting a distributed log collection architecture and machine learning algorithms, thereby realizing intelligent analysis and mining of log data, and solving the problem of incomplete log data collection and difficulty in discovering potential problems and optimization points in current technologies.

[0014] As a preferred solution of the voice platform full-link log management and analysis system described in the present invention, it includes: a log data collection module, a log data storage module, a log data analysis module, a log data visualization module, and a security control module; the log data collection module is used to obtain log data; the log data storage module is used to store log data using a log storage engine; the log data analysis module is used to analyze log data through a machine learning algorithm; the log data visualization module is used for log data visualization display; and the security control module is used for log security control.

[0015] A computer device includes a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement a method for full-link log management and analysis of a voice platform.

[0016] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of a full-link log management and analysis method for a voice platform.

[0017] Beneficial effects of the present invention: The voice platform full-link log management and analysis method provided by the present invention adopts a distributed architecture, deploys the collector and the storage engine at the log generation source, realizes the comprehensive collection and storage of the full-link log data, ensures real-time monitoring and analysis, and improves the efficiency and comprehensiveness of data collection; through distributed collection, not only the risk of single point failure is reduced, but also the overall stability and performance of the system are improved; by using an efficient storage engine, the system can process a large amount of log data, and improves query efficiency; through data compression and encryption technology, the data is compressed in the data storage stage to save data space while also improving storage efficiency. The use of data encryption technology not only guarantees data security, but also improves the security and reliability of the system; by introducing machine learning algorithms, intelligent analysis and mining of log data are realized, and potential problems and optimization points in log data are found; through real-time analysis of data, historical data analysis, anomaly detection, and trend prediction, data analysis is made more in-depth and comprehensive, which can help operation and maintenance personnel fully understand the system status; through access control, authority management and data encryption, the security and compliance of log data in the process of collection, storage, analysis and transmission are ensured. The present invention achieves better results in terms of efficiency, reliability and security of log management analysis. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0019] Figure 1 An overall flow chart of a full-link log management and analysis method for a voice platform provided in the first embodiment of the present invention.

[0020] Figure 2 An overall flow chart of a voice platform full-link log management and analysis system provided for the third embodiment of the present invention. DETAILED DESCRIPTION

[0021] In order to make the above-mentioned purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are described in detail below in conjunction with the drawings of the specification. Obviously, the described embodiments are part of the embodiments of the present invention, but not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary persons in the art without creative work should fall within the scope of protection of the present invention.

[0022] Example 1, reference Figure 1 , is an embodiment of the present invention, and provides a voice platform full-link log management and analysis method, including:

[0023] S1: Get log data.

[0024] In an embodiment of the present application, obtaining log data includes: defining a unified log format; defining log information collection rules; and deploying a log collector.

[0025] Furthermore, the log format is a structured way of recording log information, which specifies the fields contained in the log and the order and format of the fields;

[0026] In the full-link log management and analysis method of the voice platform, the defined log format includes log level, timestamp, business module identifier, operation type, request ID, user ID, parameter information, and exception information;

[0027] Among them, log level: indicates the importance or urgency of the log, such as INFO (information), WARN (warning), ERROR (error), etc.

[0028] Timestamp: records the time when the log is generated, usually in the ISO 8601 standard format, such as "YYYY-MM-DDTHH:MM:SS.sssZ";

[0029] Business module identifier: used to distinguish the business module or component to which the log belongs, such as "auth" (authentication module), "voice" (voice module), etc.

[0030] Operation type: indicates the operation type recorded in the log, such as "login", "call", "record", etc.

[0031] Request ID: A unique ID used to identify a request, which facilitates tracking requests in a distributed system;

[0032] User ID: records the user ID that performs the operation, used to associate user behavior with logs;

[0033] Parameter information: record the parameter information required for the operation, such as the number of the called party, the file path of the recording, etc.

[0034] Exception information: If an operation fails, the exception information is recorded, including the exception type, exception message, and stack trace.

[0035] It should be noted that the log information collection rules are the rules that guide the collector to collect log information. They specify the scope, method, frequency, etc. of the log collection process;

[0036] In the full-link log management and analysis method of the voice platform, the defined collection rules include collection scope, collection frequency, collection method, filtering conditions, and storage location;

[0037] Among them, the collection scope: clearly define the log types that need to be collected, such as system logs, business logs, exception logs, etc.;

[0038] Collection frequency: Determine the frequency of log collection based on business needs and data volume, such as real-time collection, collection every minute, collection every hour, etc.;

[0039] Collection method: Determine the log collection method, such as through log files, network transmission, message queues, etc.;

[0040] Filter conditions: Set filter conditions to exclude log information that does not need to be collected, such as collecting only ERROR-level logs;

[0041] Storage location: Specify the storage location of the collected log information, such as the local file system, remote server, cloud storage, etc.

[0042] In an embodiment of the present application, deploying a log collector includes: configuring the log collector; and deploying the configured log collector to each log generation source of the distributed system to obtain log data.

[0043] Furthermore, configuring a log collector is to select a suitable log collector based on the business scale and needs, configure the collector to read the defined log format and collection rules, and collect log data from the log generation source.

[0044] It should be noted that the log collector may be a Logstash collector, a Filebeat collector, or other collectors suitable for log collection.

[0045] In the embodiment of the present application, the log collector selected is the Logstash collector. When using the Logstash collector, the collection rules are defined by writing a Logstash configuration file. The Logstash configuration file includes an input plug-in, a filter plug-in, and an output plug-in, etc. The configured Logstash collector is deployed to each log generation source to obtain log data.

[0046] In an optional embodiment, the log collector selected is the Filebeat collector. When using the Filebeat collector, it is necessary to download the Filebeat binary file at each log generation source, and adjust the file appropriately according to the system configuration, and then create or edit the Filebeat configuration file, specify the log file or directory read by Filebeat, specify the destination to which the log data will be sent, define the collection rules, set the log format, deploy the Filebeat collector at each log generation source, test the configuration and start it.

[0047] S2: Use the log storage engine to store log data.

[0048] In an embodiment of the present application, the log storage engine includes: clustered deployment of the log storage engine; adding an index strategy; implementing shard replication; and performing data writing and querying.

[0049] In an embodiment of the present application, storing log data includes: compressing the log data using a compression algorithm; encrypting the compressed data using an encryption algorithm; and writing the encrypted data storage into a storage engine.

[0050] Furthermore, log data storage uses an efficient log storage engine to support the rapid writing and query of large-scale log data. At the same time, data compression and encryption technology are used in combination to ensure the security and storage efficiency of log data.

[0051] The specific steps of storing log data are: first compressing the acquired log data, then encrypting the compressed data, and then storing the encrypted data in the container through the log storage engine;

[0052] When reading log data, the encrypted data is first decrypted and then the decrypted data is decompressed to restore the original data.

[0053] It should also be noted that clustered deployment of the log storage engine is achieved by adding more nodes to improve the scalability and fault tolerance of the system;

[0054] Adding an index strategy is to create a suitable index strategy for log data to improve query efficiency. You can optimize the index configuration based on factors such as the field of log data, query frequency, and query mode.

[0055] Shard replication is achieved by the log storage engine dividing data into multiple shards and replicating each shard to multiple nodes to improve data availability and fault tolerance. By properly configuring the number of shards and the replication factor, the read and write performance and storage cost can be balanced.

[0056] Data writing and querying uses the interface provided by the log storage engine to write log data into the engine and perform fast query and analysis operations.

[0057] It should be noted that the log storage engine may be Elasticsearch, Graylog, or other suitable log storage engines.

[0058] In the embodiment of the present application, the log storage engine uses Elasticsearch, which is a Lucene-based search engine that can be expanded to thousands of nodes and PB-level data, supports query and full-text search, and can provide near real-time search capabilities.

[0059] In an optional embodiment, the log storage engine uses Splunk, which is a powerful log management, search and analysis platform that supports complex multi-field searches and time range queries, can monitor and analyze log data in real time, and can index log data and support large-scale data processing and visualization.

[0060] It should also be noted that the data compression algorithm can select an efficient compression algorithm such as Snappy, LZ4, etc. to compress the log data. Before the log data is written into the storage engine, the corresponding compression algorithm can be used to compress the log data. When reading the log data, the compressed data is decompressed to restore the original data.

[0061] The data encryption algorithm can select symmetric encryption algorithms such as AES to encrypt log data.

[0062] S3: Analyze log data through machine learning algorithms.

[0063] In an embodiment of the present application, analyzing log data through a machine learning algorithm includes: preprocessing log data; model training of a suitable machine learning algorithm; real-time analysis and prediction of log data; batch analysis and prediction of historical data; anomaly detection and early warning of log data; trend prediction of log data and optimization of the system.

[0064] Furthermore, the machine learning algorithm may be a classification algorithm, a clustering algorithm, or other machine learning algorithms suitable for processing the corresponding log data, depending on the specific situation.

[0065] In the embodiments of the present application, the machine learning algorithm uses a combination of multiple algorithms for processing according to the specific situation;

[0066] Specifically, classification algorithm: by introducing logistic regression algorithm to classify log data, such as distinguishing normal logs from abnormal logs;

[0067] Clustering algorithm: K-means algorithm is used to cluster similar log data into clusters to find potential problems or patterns;

[0068] Association rule mining algorithm: Apriori algorithm is used to discover the association between log data, such as whether there is a correlation between certain operations;

[0069] Time series analysis algorithm: LSTM algorithm is used to analyze the changing trend of log data over time to predict future log conditions;

[0070] Anomaly detection algorithm: The Isolation Forest algorithm is used to detect anomalies in log data or data that deviates from normal patterns.

[0071] In an optional embodiment, the machine learning algorithm realizes intelligent analysis and mining of log data by introducing multiple algorithms such as decision tree, DBSCAN, FP-Growth, ARIMA, LOF, etc., to discover potential problems and optimization points.

[0072] It should also be noted that data preprocessing includes cleaning the log data, removing invalid or redundant information, formatting the log data so that it meets the input requirements of the machine learning algorithm, and extracting features from the log data.

[0073] It should also be noted that model training for a suitable machine learning algorithm includes selecting a suitable machine learning algorithm, building a model based on the characteristics of the log data, training the model using historical log data so that the model can learn the inherent laws and patterns of the log data, and verifying and tuning the trained model.

[0074] It should also be noted that real-time analysis and prediction of log data includes inputting the real-time generated log data into a trained model for real-time analysis and prediction. Based on the analysis results, potential problems or anomalies are discovered in a timely manner, and corresponding alarms or notifications are triggered.

[0075] It should also be noted that batch analysis and forecasting of historical data includes batch analysis of historical data to discover long-term trends or patterns.

[0076] It should also be noted that anomaly detection and early warning of log data includes using anomaly detection algorithms to monitor log data in real time, promptly discover anomalies or data that deviates from normal patterns, and trigger corresponding early warning mechanisms based on the results of anomaly detection, such as sending emails, text messages or messages to notify relevant personnel for processing.

[0077] It should also be noted that trend prediction of log data and optimization of the system include using time series algorithms to predict the trend of log data to predict future log conditions, and optimizing and adjusting the system based on the prediction results.

[0078] S4: Log data visualization

[0079] Furthermore, the visual log display methods include but are not limited to time series diagrams, topology diagrams, and heat maps.

[0080] It should be noted that log visualization is achieved by introducing JavaScript libraries and frameworks, including D3.js, AntV, ECharts, and React frameworks.

[0081] S5: Log security control

[0082] In the embodiment of the present application, log security control includes: access control to the log system; authority management of system users; and data encryption of log data.

[0083] Furthermore, access control to the log system includes identity authentication, which is to authenticate the users who access the log system to ensure that only authorized users can access the log data. Access control can use a variety of identity authentication methods such as username and password, digital certificates, and biometrics;

[0084] Configure access policies based on user roles and permissions. When a user is an administrator, he or she has access to all log data.

[0085] When the user is a common user, the permission allows access to log data related to the user.

[0086] Session management monitors and manages user sessions to ensure that users cannot continue to access the log system after the session ends or times out.

[0087] It should be noted that the authority management of system users includes authority division, which is finely divided according to the user's responsibilities and needs. The authority may include reading, writing, deleting, etc., to ensure that users can only perform operations consistent with their own authority;

[0088] Review permissions, and regularly review and adjust user permissions to ensure that user permissions match their responsibilities;

[0089] Record permission logs and user access and operation logs to log data for auditing and tracing.

[0090] It should also be noted that data encryption for log data includes storage encryption, which performs symmetric encryption and asymmetric encryption on the data stored in the log system to ensure that even if the data is stolen, it cannot be easily encrypted;

[0091] Data transmission encryption uses SSL / TLS encryption technology during the transmission of log data to ensure that the data is not stolen or tampered with during transmission.

[0092] Example 2 is an embodiment of the present invention, which provides a full-link log management and analysis method for a voice platform. In order to verify the beneficial effects of the present invention, scientific demonstration is carried out through economic benefit calculation and simulation experiments.

[0093] In this embodiment, a large-scale voice platform is used as a test object. The platform processes millions of voice calls every day and generates a large amount of log data.

[0094] First, a unified log format is defined and detailed log information collection rules are formulated. As a result, multiple log collectors are deployed. These collectors are configured at various key nodes of the platform to collect log data. To ensure the comprehensiveness and accuracy of the data, the collectors are tested and calibrated.

[0095] Secondly, the collected log data is stored in the server using the clustered log storage engine. After the data is stored, the collected log data is used to perform model training using the machine learning algorithm.

[0096] Finally, the trained machine learning algorithm model is used to analyze and predict the log data generated by the voice platform in real time, perform batch analysis and prediction on historical data, perform anomaly detection and trend prediction, and visualize the log data using visualization tools.

[0097] Through experiments, it is found that the method of the present invention is superior to traditional methods in terms of log collection efficiency, data storage efficiency, and analysis accuracy. In particular, in terms of log collection efficiency, the method of the present invention is 14% higher than the traditional method, which significantly improves the speed and quality of data collection. In terms of data storage efficiency, the method of the present invention also shows better performance, which is mainly due to the clustered storage and optimization strategy adopted by the present invention.

[0098] The improvement in analysis accuracy is mainly due to the application of machine learning algorithms, which can more accurately identify patterns and anomalies in log data.

[0099] In general, the method of the present invention shows significant advantages in improving the overall performance of full-link log management and analysis of the voice platform, especially in terms of efficiency, accuracy and security.

[0100] Example 3, reference Figure 2, as an embodiment of the present invention, provides a voice platform full-link log management and analysis system, including a log data collection module, a log data storage module, a log data analysis module, a log data visualization module, and a security control module.

[0101] The log data collection module is used to obtain log data; the log data storage module is used to store log data using a log storage engine; the log data analysis module is used to analyze log data through a machine learning algorithm; the log data visualization module is used to visualize log data; and the security control module is used for log security control.

[0102] If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the methods of each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0103] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in conjunction with such instruction execution systems, devices or apparatuses. For the purposes of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in conjunction with such instruction execution systems, devices or apparatuses.

[0104] More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or more wires (electronic device), a portable computer disk case (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be a paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, deciphering, or processing in another suitable manner as necessary, and then stored in a computer memory.

[0105] It should be understood that the various parts of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc. It should be noted that the above embodiments are only used to illustrate the technical solution of the present invention and are not limited. Although the present invention is described in detail with reference to the preferred embodiments, it should be understood by those skilled in the art that the technical solution of the present invention can be modified or replaced by equivalents without departing from the spirit and scope of the technical solution of the present invention, which should be included in the scope of the claims of the present invention.

[0106] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the present invention, which should all be included in the scope of the claims of the present invention.

Claims

1. A voice platform full-link log management and analysis method, characterized in that: include: Get log data; Use the log storage engine to store log data; Analyze log data through machine learning algorithms; Visual display of log data; Log security control.

2. The voice platform full-link log management and analysis method according to claim 1, characterized in that: The obtaining of log data includes: Define unified log format; Define log information collection rules; Deploy the log collector.

3. The voice platform full-link log management and analysis method according to claim 2, characterized in that: The deployment log collector includes: Configure the log collector; Deploy the configured log collector to each log generation source in the distributed system to obtain log data.

4. The voice platform full-link log management and analysis method according to claim 3, characterized in that: The log storage engine includes: Clustered deployment of log storage engine; Add index strategy; Implement shard replication; Write and query data.

5. The voice platform full-link log management and analysis method according to claim 4, characterized in that: The stored log data includes: Log data is compressed using a compression algorithm; The compressed data is encrypted using an encryption algorithm; Write encrypted data to the storage engine.

6. The voice platform full-link log management and analysis method according to claim 5, characterized in that: Analyzing log data by machine learning algorithm includes: Preprocess log data; Perform model training on suitable machine learning algorithms; Real-time analysis and prediction of log data; Batch analysis and prediction of historical data; Perform anomaly detection and early warning on log data; Predict trends in log data and optimize the system.

7. The voice platform full-link log management and analysis method according to claim 6, characterized in that: The log security control includes: Implement access control on the log system; Manage the permissions of system users; Encrypt log data.

8. A system using the voice platform full-link log management and analysis method as described in any one of claims 1 to 7, characterized in that: It includes log data collection module, log data storage module, log data analysis module, log data visualization module and security control module; The log data collection module is used to obtain log data; The log data storage module is used to store log data using a log storage engine; The log data analysis module is used to analyze log data through a machine learning algorithm; The log data visualization module is used for log data visualization; The security control module is used for log security control.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the voice platform full-link log management and analysis method described in any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the voice platform full-link log management and analysis method described in any one of claims 1 to 7 are implemented.

Images