Data Processing Method, Apparatus, Electronic Device and Computer Storage Medium

Abstract

The embodiments of the present application provide a data processing method, device, electronic device and computer storage medium, which relate to the field of data processing technology. The method includes: an Internet system obtains log data to be processed, and generates an update log template for the initial log template of the target Internet system based on the log data. If at least one update log template is generated, the at least one update log template is converted into at least one corresponding first feature vector and transmitted to a data center; the target log template transmitted by the data center is received, and the initial log template is updated based on the target log template. The data processing method provided by the present application, in the face of a large amount of log data, the Internet system first extracts the update log template from the log data and then sends it to the data center. The data center generates the target log template based on the update log template and then sends it to each Internet system, which can effectively reduce the load of the data center.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and more specifically, to a data processing method, device, electronic device, and computer storage medium. Background Art

[0002] With the widespread use of log analysis in consoles, network applications, storage systems, concurrent systems, and general systems, log files play a vital role in monitoring network conditions, detecting hardware failures, and protecting software security. By analyzing system logs, which record dynamic information during program runtime, operations and maintenance personnel can effectively analyze and reproduce errors, thereby fixing system errors and improving system reliability.

[0003] However, the number of logs generated every day in multiple systems is huge, and all of them need to be transmitted to the data center for aggregation. The data center uniformly analyzes and applies all log data. The amount of log data that the data center needs to analyze is large, and the load is heavy. Summary of the Invention

[0004] The present invention aims to solve the problem of a large number of logs and heavy log data load in data centers. The technical solution is as follows:

[0005] In a first aspect, the present application provides a data processing method, which is applied to any target Internet system among at least two Internet systems, comprising:

[0006] Obtaining log data to be processed, and generating an updated log template for an initial log template of the target Internet system based on the log data; wherein the initial log template is used to parse the log data to be processed;

[0007] If at least one update log template is generated, converting the at least one update log template into corresponding at least one first feature vector and transmitting the first feature vector to the data center;

[0008] Receive the target log template transmitted by the data center, and update the initial log template based on the target log template; wherein the target log template is generated by the data center by aggregating the first feature vectors sent by various Internet systems.

[0009] In an optional embodiment of the first aspect, generating an updated log template for an initial log template of the target Internet system according to the log data includes:

[0010] Matching the log data with an initial log template of the target Internet system to obtain a matching result;

[0011] If the matching result is a matching failure, an update log template is generated based on the log data.

[0012] In an optional embodiment of the first aspect, the method further includes:

[0013] If the matching result is a successful match, the index value corresponding to the log template matching the log data is updated; wherein the index value points to the log data matched by the corresponding log template.

[0014] In a second aspect, the present application provides a data processing method, applied to a data center, comprising:

[0015] Receiving first feature vectors respectively transmitted by at least two Internet systems; for any target Internet system among the at least two Internet systems, the first feature vector is converted from an update log template; the update log template is generated by the target Internet system based on log data to be processed for an initial log template of the target Internet system;

[0016] aggregating the at least two first feature vectors to obtain a second feature vector, and generating a target log template based on the second feature vector;

[0017] The target log template is sent to each Internet system respectively, so that each Internet system updates its own initial log template respectively.

[0018] In an optional embodiment of the second aspect, aggregating the at least two first eigenvectors to obtain a second eigenvector includes:

[0019] Aggregating the at least two first eigenvectors based on initial weights of the respective first eigenvectors to obtain an initial second eigenvector;

[0020] Determining first difference information between each first eigenvector and the initial second eigenvector; determining second difference information based on each first difference information;

[0021] If the second difference information meets a preset condition, the initial second eigenvector is used as the second eigenvector; the preset condition includes that the second difference information is less than or equal to a preset threshold.

[0022] In an optional embodiment of the second aspect, the method further includes:

[0023] If the second difference information does not meet the preset conditions, repeatedly adjust the initial weights of the respective first eigenvectors; obtain an updated initial second eigenvector based on the respective first eigenvectors after weight adjustment, until the second difference information corresponding to the updated initial second eigenvector and the respective first eigenvectors meets the preset conditions, and use the updated initial second eigenvector as the second eigenvector.

[0024] In an optional embodiment of the second aspect, the method further includes:

[0025] Determining a communication mode based on the plurality of Internet systems and data centers to be processed; the communication mode including star, mesh or chain;

[0026] The target log template is transmitted to the multiple Internet systems and the data center based on the communication method.

[0027] In a third aspect, a data processing device is provided, which is applied to any target Internet system among at least two Internet systems, and includes:

[0028] an acquisition module, configured to acquire log data to be processed and generate an updated log template for an initial log template of the target Internet system based on the log data; wherein the initial log template is used to parse the log data to be processed;

[0029] a transmission module, configured to, if at least one update log template is generated, convert the at least one update log template into at least one corresponding first feature vector and transmit the first feature vector to a data center;

[0030] An update module is used to receive the target log template transmitted by the data center and update the initial log template based on the target log template; wherein, the target log template is generated by the data center by aggregating the first feature vectors sent by various Internet systems.

[0031] In a fourth aspect, a data processing device is provided, which is applied to a data center and includes:

[0032] a receiving module configured to receive first feature vectors respectively transmitted by at least two Internet systems; for any target Internet system among the at least two Internet systems, the first feature vector is converted from an update log template; the update log template is generated by the target Internet system based on the log data to be processed for the initial log template of the target Internet system;

[0033] an aggregation module, configured to aggregate the at least two first feature vectors to obtain a second feature vector, and generate a target log template based on the second feature vector;

[0034] The sending module is used to send the target log template to each Internet system respectively, so that each Internet system can update its own initial log template respectively.

[0035] In a fifth aspect, an electronic device is provided, comprising a memory, a processor, and a computer program stored on the memory, characterized in that the processor executes the computer program to implement the steps of the method described in any one of the first aspect or the second aspect of the present application.

[0036] In a sixth aspect, a computer-readable storage medium is provided, on which a computer program is stored, characterized in that when the computer program is executed by a processor, it implements the data processing method shown in any one of the first aspect or the second aspect of the present application.

[0037] The beneficial effects of the technical solution provided by the embodiments of the present application are:

[0038] The data processing method provided in the present application first generates an update log template based on the log data to be processed by any target Internet system in two or more Internet systems, and then sends the update log template to a data center. The data center then generates a target log template based on the update log template sent by each target Internet system, and sends the target log template to each target Internet system, so that each target Internet system parses the log data to be processed based on the target log template, which can effectively reduce the load of the data center.

[0039] In addition, each Internet system extracts the update log template from the log data and converts it into the first feature vector, and transmits the vector to the data center, which can effectively reduce the amount of data transmitted and reduce the transmission pressure.

[0040] Furthermore, by aggregating the first eigenvectors transmitted by two or more Internet systems in a data center to obtain a second eigenvector, and adjusting the weights of each first eigenvector until the difference between the second eigenvector and each first eigenvector meets the preset conditions, an updated second eigenvector is obtained and a target log template is generated and passed to each Internet system, the aggregation result of the data center can be closer to the optimal aggregation result, making the above data processing method more effective. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for describing the embodiments of the present application.

[0042] Figure 1 A schematic diagram of an application scenario of a data processing method provided in an embodiment of the present application;

[0043] Figure 2 A flowchart of a data processing method applied to any target Internet system of at least two Internet systems provided in an embodiment of the present application;

[0044] Figure 3 A flowchart of a data processing method applied to a data center provided in an embodiment of the present application;

[0045] Figure 4 A flowchart illustrating an example of a data processing method provided in an embodiment of the present application;

[0046] Figure 5 A schematic structural diagram of a data processing device provided in an embodiment of the present application and applied to any target Internet system of at least two Internet systems;

[0047] Figure 6 A schematic diagram of the structure of a data processing device for use in a data center provided in an embodiment of the present application;

[0048] Figure 7 A schematic diagram of the structure of an electronic device applicable to a data processing method provided in an embodiment of the present application. DETAILED DESCRIPTION

[0049] The following describes the embodiments of the present application in conjunction with the accompanying drawings. It should be understood that the embodiments described below in conjunction with the accompanying drawings are exemplary descriptions for explaining the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions of the embodiments of the present application.

[0050] Those skilled in the art will understand that, unless otherwise stated, the singular forms "a", "an", "said", and "the" used herein may also include plural forms. It should be further understood that the terms "including" and "comprising" used in the embodiments of the present application mean that the corresponding features can be implemented as the features, information, data, steps, operations, and components presented, but do not exclude implementation as other features, information, data, steps, operations, components, and / or combinations thereof supported by the present technical field. It should be understood that when we say that an element is "connected" or "coupled" to another element, the element can be directly connected or coupled to the other element, or it can refer to the element and the other element establishing a connection relationship through an intermediate element. In addition, the "connection" or "coupling" used here can include wireless connection or wireless coupling. The term "and / or" used here indicates at least one of the items defined by the term, for example, "A and / or B" can be implemented as "A", or as "B", or as "A and B".

[0051] In order to make the objectives, technical solutions and advantages of this application clearer, the implementation methods of this application will be further described in detail below with reference to the accompanying drawings.

[0052] In existing technologies, various Internet systems transmit all log data to a data center for aggregation. The data center then extracts log templates, processes semantic features, aggregates semantic features, and performs other steps on the log data. This not only puts a certain amount of pressure on data transmission, but also places high computing requirements on the data center.

[0053] In response to at least one technical problem or area that needs improvement in the relevant technology, the present application proposes a data processing method, device, electronic device and computer storage medium. The data processing method provided by this solution first generates an update log template based on the log data to be processed by any target Internet system in two or more Internet systems, and then sends the update log template to the data center. The data center then generates a target log template based on the update log template sent by each target Internet system, and sends the target log template to each target Internet system respectively, so that each target Internet system can parse the log data to be processed based on the target log template, which can effectively reduce the load of the data center.

[0054] In addition, each Internet system extracts the update log template from the log data and converts it into the first feature vector, and transmits the vector to the data center, which can effectively reduce the amount of data transmitted and reduce the transmission pressure.

[0055] Furthermore, by aggregating the first eigenvectors transmitted by two or more Internet systems in a data center to obtain a second eigenvector, and adjusting the weights of each first eigenvector until the difference between the second eigenvector and each first eigenvector meets the preset conditions, an updated second eigenvector is obtained and a target log template is generated and passed to each Internet system, the aggregation result of the data center can be closer to the optimal aggregation result, making the above data processing method more effective.

[0056] The following describes several exemplary embodiments to illustrate the technical solutions of the embodiments of the present application and the technical effects produced by the technical solutions of the present application. It should be noted that the following embodiments can refer to, draw on, or combine with each other, and the same terms, similar features, and similar implementation steps in different embodiments will not be repeated.

[0057] Figure 1 Schematic diagram of an application scenario of the data processing method provided in an embodiment of the present application, wherein the application environment may include two or more Internet systems 101 and data centers 102, and each Internet system and data center may transmit data through the network.

[0058] Specifically, in any target Internet system among two or more Internet systems 101, log data to be processed is obtained, and an update log template for the initial log template of the target Internet system is generated based on the log data; wherein, the initial log template is used to parse the log data to be processed; if at least one update log template is generated, the at least one update log template is converted into at least one corresponding first feature vector and transmitted to the data center; the target log template transmitted by the data center is received, and the initial log template is updated based on the target log template; wherein, the target log template is generated by the data center by aggregating the first feature vectors sent by each Internet system.

[0059] In the above application scenario, the above data processing method is used in the Internet system 101 to extract log templates from log data, vectorize the log templates, and send them to the data center. In other embodiments, the log templates transmitted from the Internet system can also be aggregated in the data center to determine the second feature vector and the target log template. The above Internet system 101 and data center 102 can be deployed on a server or on some terminal that supports the above data processing method. The above application scenario is only an example and does not limit the application scenario of the data processing method of this application.

[0060] Those skilled in the art will understand that a server may include a server installed with a computer capable of processing database operations. A server may be an independent physical server, a server cluster or distributed system consisting of multiple physical servers, or a cloud server or server cluster that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms.

[0061] The terminal can be a smart phone (such as an Android phone, an iOS phone, etc.), a tablet computer, a laptop computer, a digital broadcast receiver, a MID (Mobile Internet Devices), a PDA (Personal Digital Assistant), a desktop computer, a smart home appliance, a vehicle-mounted terminal (such as a vehicle-mounted navigation terminal, a vehicle-mounted computer, etc.), a smart speaker, a smart watch, etc. The terminal and the server can be directly or indirectly connected via wired or wireless communication, but are not limited to this. The embodiments of the present invention can be applied to various scenarios, including but not limited to cloud technology, artificial intelligence, smart transportation, assisted driving, etc. The specific requirements can also be determined based on the actual application scenario requirements and are not limited here.

[0062] In some possible implementations, taking the execution subject as any target Internet system in at least two Internet systems as an example, the embodiment of the present application provides a data processing method, such as Figure 2 As shown, the following steps may be included:

[0063] Herein, at least two Internet systems refer to two or more Internet systems.

[0064] S201 , obtaining log data to be processed, and generating an updated log template for an initial log template of the target Internet system according to the log data.

[0065] The initial log template is used to parse the log data to be processed. The initial log template can be a log template obtained by processing historical log data.

[0066] Among them, Internet systems may include IT systems.

[0067] The method for generating the update log template can be selected according to the log parsing application, which may include: troubleshooting, performance optimization, security auditing, business analysis, and operation and maintenance management.

[0068] Specifically, the internet system collects logs and obtains raw log data. This raw log data includes the system's operating status, time information, key descriptions, and other important information used to record the internet system's operation. After log collection is completed, the raw log data is filtered according to a preset filtering method to obtain filtered log data for processing. The preset filtering method includes deleting duplicate information and removing data that meets preset useless information.

[0069] Specifically, the steps of generating an update log template based on log data can be: comparing the log data to be processed with the initial log template. If each log data can determine a matching template from the initial log template, then no update log template is generated; if there is log data that does not match the initial log template, then an update log template is generated based on the log data.

[0070] S202: If at least one update log template is generated, convert the at least one update log template into corresponding at least one first feature vector and transmit the first feature vector to a data center.

[0071] Specifically, if at least one update log template is generated, a preset vectorization method is used to convert the update log template into a first feature vector based on the update log template. There can be multiple update log templates and multiple first feature vectors, with each update log template and first feature vector corresponding one to the other. If multiple update log templates exist, each of the multiple update log templates has a corresponding first feature vector.

[0072] In the specific implementation process, the preset vectorization methods may include the BERT (Bidirectional Encoder Representations from Transformers) pre-trained model and the Transformer model; among them, the BERT pre-trained model can be used to generate high-quality word vectors or perform fine-tuning on downstream tasks; the Transformer model consists of two parts: an encoder and a decoder. The encoder is responsible for converting the input sequence into a series of feature representations, while the decoder uses these feature representations to generate the output sequence.

[0073] During implementation, the vectorized results are transmitted to the data center. For example, if a log entry is directly uploaded using existing methods, dozens of characters would be required. In actual production environments, daily log data can reach hundreds of GB or even TB levels. After template extraction, each template corresponds to several log entries, and the data is compressed to a few megabytes or even smaller. Vectorization reduces the data volume to megabytes, increasing data transmission and storage capacity by over a thousand times, significantly reducing transmission and storage pressures.

[0074] S203: Receive the target log template transmitted by the data center, and update the initial log template based on the target log template.

[0075] The target log template is generated by the data center by aggregating first feature vectors sent by various Internet systems.

[0076] After each Internet system obtains the target log template transmitted by the data center and updates its own log template, the template of the data center and the template of each Internet system can remain consistent.

[0077] Specifically, after receiving the target log template, the target log template is compared with the initial log template already existing in the Internet system, and the new template or updated template is determined based on the comparison result. After the log template is updated, the updated log template can be used to parse the log data and perform related applications.

[0078] In some possible implementations, generating an updated log template for the initial log template of the target Internet system based on the log data in S201 includes:

[0079] (1) matching the log data with an initial log template of the target Internet system to obtain a matching result;

[0080] (2) If the matching result is a matching failure, an update log template is generated based on the log data.

[0081] Specifically, each log data item is matched against the initial log template, obtaining a matching result. If the matching result is unsuccessful, it is considered that the current log template can be updated, and an updated log template is generated. Subsequent log data from the log data that generated the updated log template is also matched against the updated log template generated by the log data. Matching log data with existing log data can effectively reduce the amount of log data and eliminate the steps of generating log templates from useless log data.

[0082] In the specific implementation process, the Spell method (an online streaming log parsing method based on the longest common subsequence) can be used to extract log templates. Among them, LCS (longest common subsequence) is a subsequence of two or more known sequences, and it is the longest subsequence that meets this condition. LCSobject (Longest Common Subsequence Object) is an object that stores structured log templates and related information; LCSseq is the longest common subsequence of multiple logs; LCSMap (Longest Common Subsequence Map) is a list of all Longest Common Subsequence objects. Each time a new log data is processed, the LCSMap is searched for an object that matches the log data. If no object matches the current log data, a new log template is extracted from the log data center.

[0083] In some possible implementations, the data processing method further includes:

[0084] (1) If the matching result is a successful match, the index value corresponding to the log template matching the log data is updated; wherein the index value points to the log data matched by the corresponding log template.

[0085] Among them, the matching methods may include: Spell (online streaming log parsing method based on the longest common subsequence), DeepLog (deep log), LogAnomaly (log anomaly detection), and LogRobust (log robustness).

[0086] In the specific implementation process, an online streaming log parsing method based on the longest common subsequence can be used to extract log templates. Every time a new log data is processed, the object matching the log data is searched in the LCSMap. If there is an object whose longest common subsequence with the log data is greater than a preset threshold, the object is considered to match the log data; when there is a longest common subsequence object that matches the log data, the index value corresponding to the LCSobject is updated; wherein the index value can be used to find the log data according to the log template.

[0087] During the specific implementation process, if there is a need to search for log data corresponding to a log template, the log data corresponding to the log template can be queried according to the above index value. There can be multiple log data corresponding to each log template.

[0088] In some possible implementations, taking the execution subject as a data center as an example, the embodiment of the present application provides a data processing method, such as Figure 2 As shown, the following steps may be included:

[0089] S301: Receive first feature vectors transmitted respectively by at least two Internet systems.

[0090] Among them, for any target Internet system among two or more Internet systems, the first feature vector is obtained by transforming the update log template; the update log template is generated by the target Internet system based on the log data to be processed based on the initial log template of the target Internet system.

[0091] S302: Aggregate the at least two first feature vectors to obtain a second feature vector, and generate a target log template based on the second feature vector.

[0092] There may be one or more second eigenvectors, which are the result of aggregating at least two first eigenvectors and represent the commonalities and differences between the first eigenvectors.

[0093] The method of aggregating the at least two first eigenvectors may include a DTW (Dynamic Time Warping) method.

[0094] In the specific implementation process, the aggregation steps can be: calculating the distances between all first eigenvectors according to the DTW algorithm and constructing a distance matrix; inputting the distance matrix into the clustering algorithm to obtain the category to which each sample point is divided; aligning all samples in the same category using the DTW algorithm; for each category, calculating the average distance between all samples as the representative time series of the category.

[0095] Specifically, the step of generating a target log template based on the second feature vector may include: using a BERT pre-trained model and a Transformer model to map the second feature vector to the log template to obtain the target log template.

[0096] S303: Send the target log template to each Internet system respectively, so that each Internet system updates its own initial log template respectively.

[0097] Specifically, the obtained target log template is sent to each Internet system to realize template sharing between the data center and the Internet system, so that each Internet system can use the shared template to perform log data analysis and related applications.

[0098] In some possible implementations, aggregating the at least two first eigenvectors in S302 to obtain a second eigenvector includes:

[0099] (1) aggregating the at least two first eigenvectors based on the initial weights of the first eigenvectors to obtain an initial second eigenvector;

[0100] (2) determining first difference information between each first eigenvector and the initial second eigenvector; and determining second difference information based on each first difference information;

[0101] (3) If the second difference information meets the preset conditions, the initial second eigenvector is used as the second eigenvector.

[0102] The preset condition includes that the second difference information is less than or equal to a preset threshold.

[0103] Among them, the first difference information can represent the difference between the results of semantic aggregation of each Internet system and the results of semantic aggregation of the data center; there can be multiple first difference information, and in the specific implementation process, the first difference information can be one-to-one corresponding to the first feature vector.

[0104] In some possible implementations, the data processing method further includes:

[0105] (1) If the second difference information does not meet the preset conditions, repeatedly adjust the initial weights of the first eigenvectors; obtain updated initial second eigenvectors based on the first eigenvectors after weight adjustment, until the second difference information corresponding to the updated initial second eigenvector and the first eigenvectors meets the preset conditions, and use the updated initial second eigenvector as the second eigenvector.

[0106] Specifically, the second difference information can be expressed using the following formula:

[0107]

[0108] Among them, α i The weight of the difference between the aggregation results of the log template of each Internet system and the aggregation results of the data center log template is used. γ is an adjustable threshold (the default is 1 / N, where N is the number of tasks). L i The difference between the aggregation results of the log template of each Internet system and the aggregation results of the data center log template.

[0109] The first difference information can be expressed using the following cross entropy loss function:

[0110]

[0111] Among them, p(x j ) is the difference between the aggregation result of the Internet system log template and the optimal aggregation result, q(x j ) is the difference between the log template aggregation result of the data center and the optimal aggregation result; wherein the optimal aggregation result may include: the feature vector obtained after all the log data to be processed is transmitted to the data center for processing.

[0112] In the above process, if the second difference information does not meet the preset condition, the weight of the difference between the aggregation result of the log template of each Internet system and the aggregation result of the log template of the data center is adjusted according to the above formula (1) until the second difference information meets the preset condition. The preset condition may include that the value of the above second difference information formula is less than a preset threshold.

[0113] In the specific implementation process, the GridSearchCV (grid search) method can be used to adjust the above weights, that is, within the specified parameter range, the parameters are adjusted in sequence according to the step size, and the adjusted parameters are used to train the learner to find the parameters with the highest accuracy on the validation set from all the parameters, that is, the weights in the method of this application; wherein, after each parameter adjustment, the parameters are passed to each Internet system for parameter synchronization and gradient update, and recalculated based on the adjusted parameters.

[0114] In some possible implementations, the data processing method further includes:

[0115] (1) Determine the communication method based on multiple Internet systems and data centers to be processed;

[0116] (2) Transmitting the target log template to the multiple Internet systems and the data center based on the communication method.

[0117] The communication mode may include star type, mesh type or chain type, etc.

[0118] Specifically, the communication mode of the shared log template is determined according to the actual conditions of multiple Internet systems and data centers.

[0119] In specific implementations, star-shaped communication is used for scenarios involving a unified data center, such as a carrier's business support network operations management system, an industrial big data platform, or a financial bank's unified data management platform or operations and maintenance system. Internet systems that allow for the selection of shared or shared objects use a mesh-shaped communication approach. Chain-shaped communication can be used if individual node failures render central node results unavailable. Choosing these modes across different data centers or internet systems allows for log data-based anomaly monitoring, log pattern management, detection strategy management, alarm forwarding strategies, and fault self-healing management.

[0120] In the above embodiment, any target Internet system in two or more Internet systems first generates an update log template based on the log data to be processed, and then sends the update log template to the data center. The data center then generates a target log template based on the update log template sent by each target Internet system, and sends the target log template to each target Internet system, so that each target Internet system parses the log data to be processed based on the target log template, which can effectively reduce the load of the data center.

[0121] In addition, each Internet system extracts the update log template from the log data and converts it into the first feature vector, and transmits the vector to the data center, which can effectively reduce the amount of data transmitted and reduce the transmission pressure.

[0122] Furthermore, by aggregating the first eigenvectors transmitted by two or more Internet systems in a data center to obtain a second eigenvector, and adjusting the weights of each first eigenvector until the difference between the second eigenvector and each first eigenvector meets the preset conditions, an updated second eigenvector is obtained and a target log template is generated and passed to each Internet system, the aggregation result of the data center can be closer to the optimal aggregation result, making the above data processing method more effective.

[0123] In order to more clearly understand the data processing method described in the embodiments of the present application, it will be further elaborated below with reference to examples.

[0124] In one example, the data processing method of the present application, such as Figure 4 As shown, this may include:

[0125] In any target Internet system among two or more Internet systems, log data to be processed is obtained (i.e., log collection as shown in the figure), and an update log template for the initial log template of the target Internet system is generated based on the log data (i.e., log template extraction as shown in the figure); wherein, the initial log template is used to parse the log data to be processed; if at least one update log template is generated, the at least one update log template is respectively converted into at least one corresponding first feature vector (i.e., Internet system semantic feature processing as shown in the figure) and transmitted to the data center; the data center receives the first feature vectors respectively transmitted by two or more Internet systems; based on the initial weights of each first feature vector, at least two first feature vectors are aggregated to obtain an initial second feature vector (i.e., data center semantic feature aggregation as shown in the figure); the relationship between each first feature vector and the initial second feature vector is determined first difference information; determining the second difference information based on each first difference information; if the second difference information does not meet the preset conditions, repeatedly adjusting the initial weights of each first eigenvector; obtaining an updated initial second eigenvector based on each first eigenvector after adjusting the weights, until the second difference information corresponding to the updated initial second eigenvector and each first eigenvector meets the preset conditions, and using the updated initial second eigenvector as the second eigenvector (i.e., the iterative optimization global optimum shown in the figure); generating a target log template based on the second eigenvector (i.e., the log restoration shown in the figure); sending the target log template to each Internet system respectively; each Internet system receives the target log template transmitted by the data center, and updates the initial log template based on the target log template, so as to realize the parsing and application of log data (i.e., the log parsing application shown in the figure).

[0126] The above-mentioned data processing method generates an update log template based on the log data to be processed by any target Internet system in two or more Internet systems, and then sends the update log template to the data center. The data center then generates a target log template based on the update log template sent by each target Internet system, and sends the target log template to each target Internet system respectively, so that each target Internet system parses the log data to be processed based on the target log template, which can effectively reduce the load of the data center.

[0127] In addition, each Internet system extracts the update log template from the log data and converts it into the first feature vector, and transmits the vector to the data center, which can effectively reduce the amount of data transmitted and reduce the transmission pressure.

[0128] Furthermore, by aggregating the first eigenvectors transmitted by two or more Internet systems in a data center to obtain a second eigenvector, and adjusting the weights of each first eigenvector until the difference between the second eigenvector and each first eigenvector meets the preset conditions, an updated second eigenvector is obtained and a target log template is generated and passed to each Internet system, the aggregation result of the data center can be closer to the optimal aggregation result, making the above data processing method more effective.

[0129] The embodiment of the present application provides a data processing device, which is applied to any target Internet system in at least two Internet systems, such as Figure 5 As shown, the apparatus 50 may include: an acquisition module 501, a transmission module 502, and an update module 503, wherein:

[0130] An acquisition module 501 is configured to acquire log data to be processed and generate an updated log template for an initial log template of the target Internet system based on the log data; wherein the initial log template is used to parse the log data to be processed;

[0131] A transmission module 502 is configured to, if at least one update log template is generated, convert the at least one update log template into at least one corresponding first feature vector and transmit the first feature vector to a data center;

[0132] The updating module 503 is used to receive the target log template transmitted by the data center and update the initial log template based on the target log template; wherein, the target log template is generated by the data center by aggregating the first feature vectors sent by various Internet systems.

[0133] The embodiment of the present application provides a data processing device, which is applied to a data center, such as Figure 6 As shown, the device 60 may include: a receiving module 601, an aggregation module 602 and a sending module 603, wherein:

[0134] Receiving module 601, configured to receive first feature vectors transmitted respectively by at least two Internet systems; the first feature vectors are obtained by transforming an update log template generated based on log data to be processed;

[0135] Aggregation module 602, configured to aggregate the at least two first feature vectors to obtain a second feature vector, and generate a target log template based on the second feature vector;

[0136] The sending module 603 is configured to send the target log template to each Internet system respectively, so that the Internet system updates its own initial log template.

[0137] The data processing device provided in the present application generates an update log template based on the log data to be processed by any target Internet system in two or more Internet systems, and then sends the update log template to a data center. The data center then generates a target log template based on the update log template sent by each target Internet system, and sends the target log template to each target Internet system, so that each target Internet system can parse the log data to be processed based on the target log template, which can effectively reduce the load of the data center.

[0138] In addition, each Internet system extracts the update log template from the log data and converts it into the first feature vector, and transmits the vector to the data center, which can effectively reduce the amount of data transmitted and reduce the transmission pressure.

[0139] Furthermore, by aggregating the first eigenvectors transmitted by two or more Internet systems in a data center to obtain a second eigenvector, and adjusting the weights of each first eigenvector until the difference between the second eigenvector and each first eigenvector meets the preset conditions, an updated second eigenvector is obtained and a target log template is generated and passed to each Internet system, the aggregation result of the data center can be closer to the optimal aggregation result, making the above data processing method more effective.

[0140] In an embodiment of the present application, an electronic device is provided, including a memory, a processor, and a computer program stored in the memory. The processor executes the above-mentioned computer program to implement the steps of the data processing method. Compared with the related art, it can achieve: collecting log data and extracting log templates in the Internet system, vectorizing the log templates and then passing them to the data center, which can effectively reduce the amount of transmitted data.

[0141] In an alternative embodiment, an electronic device is provided, such as Figure 7 As shown, Figure 7 The electronic device 7000 shown includes: a processor 7001 and a memory 7003. The processor 7001 and the memory 7003 are connected, for example, via a bus 7002. Optionally, the electronic device 7000 may further include a transceiver 7004, which may be used for data exchange between the electronic device and other electronic devices, such as data transmission and / or data reception. It should be noted that in actual applications, the number of transceivers 7004 is not limited to one, and the structure of the electronic device 7000 does not constitute a limitation on the embodiments of the present application.

[0142] Processor 7001 can be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It can implement or execute the various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of this application. Processor 7001 can also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, etc.

[0143] Bus 7002 may include a path for transmitting information between the above components. Bus 7002 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus. Bus 7002 may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 7 Only one thick line is used in the diagram, but this does not mean that there is only one bus or one type of bus.

[0144] The memory 7003 can be a ROM (Read Only Memory) or other types of static storage devices that can store static information and instructions, a RAM (Random Access Memory) or other types of dynamic storage devices that can store information and instructions, or an EEPROM (Electrically Erasable Programmable Read Only Memory), a CD-ROM (Compact Disc Read Only Memory) or other optical disk storage, optical disk storage (including compact discs, laser discs, optical discs, digital versatile discs, Blu-ray discs, etc.), magnetic disk storage media, other magnetic storage devices, or any other medium that can be used to carry or store computer programs and can be read by a computer, without limitation here.

[0145] The memory 7003 is used to store the computer program for executing the embodiments of the present application, and the execution is controlled by the processor 7001. The processor 7001 is used to execute the computer program stored in the memory 7003 to implement the steps shown in the above method embodiments.

[0146] The electronic devices include but are not limited to: terminals and servers to which the above-mentioned data processing method can be applied.

[0147] An embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the steps and corresponding contents of the aforementioned method embodiment can be implemented.

[0148] The terms "first," "second," "third," "fourth," "1," "2," and the like (if any) in the specification and claims of this application and the accompanying drawings are used to distinguish similar objects and are not necessarily used to describe a particular order or sequential sequence. It should be understood that the terms used in this manner are interchangeable where appropriate, so that the embodiments of the application described herein can be implemented in an order other than that shown or described in the drawings.

[0149] It should be understood that, although each operation step is indicated by arrows in the flowchart of the embodiment of the present application, the order of implementation of these steps is not limited to the order indicated by the arrows. Unless otherwise clearly stated herein, in some implementation scenarios of the embodiment of the present application, the implementation steps in each flowchart can be performed in other orders according to demand. In addition, some or all of the steps in each flowchart can include multiple sub-steps or multiple stages based on actual implementation scenarios. Some or all of these sub-steps or stages can be executed at the same time, and each sub-step or stage in these sub-steps or stages can also be executed at different times respectively. Under different scenarios at the execution time, the execution order of these sub-steps or stages can be flexibly configured according to demand, and the embodiment of the present application does not limit this.

[0150] The above description is only an optional implementation method for some implementation scenarios of this application. It should be pointed out that for ordinary technicians in this technical field, without departing from the technical concept of the solution of this application, the use of other similar implementation methods based on the technical ideas of this application also falls within the protection scope of the embodiments of this application.

Claims

1. A data processing method, characterized in that: Applicable to any target Internet system among at least two Internet systems, including: Obtaining log data to be processed, and generating an updated log template for an initial log template of the target Internet system based on the log data; wherein the initial log template is used to parse the log data to be processed; If at least one update log template is generated, converting the at least one update log template into corresponding at least one first feature vector and transmitting the first feature vector to the data center; Receive the target log template transmitted by the data center, and update the initial log template based on the target log template; wherein the target log template is generated by the data center by aggregating the first feature vectors sent by various Internet systems.

2. The data processing method according to claim 1, wherein: Generating an updated log template for the initial log template of the target Internet system according to the log data includes: Matching the log data with an initial log template of the target Internet system to obtain a matching result; If the matching result is a matching failure, an update log template is generated based on the log data.

3. The data processing method according to claim 2, characterized in that: The method further comprises: If the matching result is a successful match, the index value corresponding to the log template matching the log data is updated; wherein the index value points to the log data matched by the corresponding log template.

4. A data processing method, characterized in that: Applications in data centers include: Receiving first feature vectors respectively transmitted by at least two Internet systems; for any target Internet system among the at least two Internet systems, the first feature vector is converted from an update log template; the update log template is generated by the target Internet system based on log data to be processed for an initial log template of the target Internet system; aggregating the at least two first feature vectors to obtain a second feature vector, and generating a target log template based on the second feature vector; The target log template is sent to each Internet system respectively, so that each Internet system updates its own initial log template respectively.

5. The data processing method according to claim 4, characterized in that: The step of aggregating the at least two first eigenvectors to obtain a second eigenvector includes: Aggregating the at least two first eigenvectors based on initial weights of the respective first eigenvectors to obtain an initial second eigenvector; Determining first difference information between each first eigenvector and the initial second eigenvector; determining second difference information based on each first difference information; If the second difference information meets a preset condition, the initial second eigenvector is used as the second eigenvector; the preset condition includes that the second difference information is less than or equal to a preset threshold.

6. The data processing method according to claim 5, characterized in that: The method further comprises: If the second difference information does not meet the preset conditions, repeatedly adjust the initial weights of the respective first eigenvectors; obtain an updated initial second eigenvector based on the respective first eigenvectors after weight adjustment, until the second difference information corresponding to the updated initial second eigenvector and the respective first eigenvectors meets the preset conditions, and use the updated initial second eigenvector as the second eigenvector.

7. The data processing method according to claim 4, characterized in that: The method further comprises: Determining a communication mode based on the plurality of Internet systems and data centers to be processed; the communication mode includes a star mode, a mesh mode, or a chain mode; The target log template is transmitted to the multiple Internet systems and the data center based on the communication method.

8. A data processing device, characterized in that: Applicable to any target Internet system among at least two Internet systems, including: an acquisition module, configured to acquire log data to be processed and generate an updated log template for an initial log template of the target Internet system based on the log data; wherein the initial log template is used to parse the log data to be processed; a transmission module, configured to, if at least one update log template is generated, convert the at least one update log template into at least one corresponding first feature vector and transmit the first feature vector to a data center; An update module is used to receive the target log template transmitted by the data center and update the initial log template based on the target log template; wherein, the target log template is generated by the data center by aggregating the first feature vectors sent by various Internet systems.

9. A data processing device, characterized in that: Applications in data centers include: a receiving module configured to receive first feature vectors transmitted respectively by at least two Internet systems; for any target Internet system among the at least two Internet systems, the first feature vector is converted from an update log template; the update log template is generated by the target Internet system based on the log data to be processed for the initial log template of the target Internet system; an aggregation module, configured to aggregate the at least two first feature vectors to obtain a second feature vector, and generate a target log template based on the second feature vector; The sending module is used to send the target log template to each Internet system respectively, so that each Internet system can update its own initial log template respectively.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory, wherein: The processor executes the computer program to implement the steps of the method according to any one of claims 1 to 3 or 4 to 7.

11. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the data processing method according to any one of claims 1 to 3 or 4 to 7 are implemented.

Images