Log data processing method and device, electronic equipment and storage medium

Abstract

The application relates to the technical field of data processing and artificial intelligence, and discloses a log data processing method and device, an electronic device and a storage medium, which comprise the following steps: obtaining initial log data to be processed, performing word segmentation processing on the initial log data, and obtaining a log word sequence; extracting an initial text weight feature of the log word sequence, combining a preset industry-specific dictionary, performing weighting processing on the initial text weight feature, and obtaining a log feature vector; inputting the log feature vector into a pre-trained clustering prediction model for processing, obtaining a belonging probability distribution of the initial log data to multiple business clustering clusters, and determining a multi-semantics clustering result of the initial log data according to the belonging probability distribution. The application can be applied to a log data analysis scene of a business system of financial insurance and medical health, can improve the fineness of complex log feature and multi-dimensional business clustering cluster matching, and can further improve the multi-semantics log clustering accuracy.

Description

Technical Field

[0001] This application relates to the fields of data processing and artificial intelligence technology, and can be applied to the field of log data analysis in financial insurance and healthcare business systems. In particular, it relates to a log processing method, apparatus, electronic device and storage medium. Background Technology

[0002] In financial insurance and healthcare systems, massive amounts of heterogeneous log data with specific business characteristics are typically generated. Accurate classification and clustering of this log data is fundamental to transforming unstructured text into structured data and subsequently conducting underlying system analysis. Currently, the industry typically processes this type of log data using conventional word frequency feature extraction combined with hard clustering algorithms, or directly introduces large-scale distributed computing frameworks for large-scale data parsing and analysis.

[0003] The inventors realized that when faced with highly specialized and information-intensive industry business data, existing solutions often only extract the surface features of general text. This approach is difficult to effectively analyze the complex business states and multiple semantics intertwined in a single log, resulting in low clustering accuracy of mixed logs containing specific industry features, which cannot meet the needs of complex business systems for high-precision analysis of underlying data. Summary of the Invention

[0004] This invention provides a log data processing method, apparatus, electronic device, and storage medium to solve the technical problem of low accuracy in multi-semantic log clustering in existing log data processing solutions.

[0005] To achieve the above objectives, a first aspect of this application proposes a log data processing method, the method comprising: Obtain the initial log data to be processed, and perform word segmentation on the initial log data to obtain a log word sequence; Extract the initial text weight features of the log word sequence, and combine them with a preset industry-specific dictionary to perform weighted processing on the initial text weight features to obtain the log feature vector; The log feature vector is input into a pre-trained clustering prediction model for processing to obtain the probability distribution of the initial log data belonging to multiple business clusters. Based on the attribution probability distribution, the multi-semantic clustering result of the initial log data is determined.

[0006] To achieve the above objectives, a second aspect of this application provides a log data processing apparatus, the apparatus comprising: The sequence acquisition module is used to acquire the initial log data to be processed and to perform word segmentation on the initial log data to obtain a log word sequence; The feature weighting extraction module is used to extract the initial text weight features of the log word sequence, and to perform weighting processing on the initial text weight features in combination with a preset industry-specific dictionary to obtain the log feature vector; The clustering processing module is used to input the log feature vector into a pre-trained clustering prediction model for processing, so as to obtain the probability distribution of the initial log data belonging to multiple business clusters; The multi-semantic determination module is used to determine the multi-semantic clustering result of the initial log data based on the attribution probability distribution.

[0007] To achieve the above objectives, a third aspect of this application provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the method described in the first aspect.

[0008] To achieve the above objectives, a fourth aspect of the present application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the method of the first aspect described above.

[0009] The above-mentioned log data processing method, device, electronic device, and storage medium can acquire the initial log data to be processed, perform word segmentation on the initial log data to obtain a log word sequence; extract the initial text weight features of the log word sequence, and perform weighted processing on the initial text weight features in combination with a preset industry-specific dictionary to obtain a log feature vector; input the log feature vector into a pre-trained clustering prediction model for processing to obtain the probability distribution of the initial log data belonging to multiple business clusters; and determine the multi-semantic clustering result of the initial log data based on the probability distribution. In this application, addressing the analysis needs of log data containing complex business states and professional terminology in business scenarios such as finance, insurance, and healthcare, the approach first strengthens industry-specific keyword features by combining a pre-defined industry-specific dictionary with initial text weight features, thereby reducing interference from general terms and generating a more accurate log feature vector. Then, this log feature vector is input into a clustering prediction model to output a probability distribution. By using the soft boundary of the probability distribution to overcome the limitations of traditional single hard clustering, a single log entry can be reasonably mapped to multiple relevant business dimensions based on the probability distribution. This effectively improves the precision and objectivity of matching complex log features with multi-dimensional business clusters, solving the technical problem of low accuracy in multi-semantic log clustering in existing log data processing solutions. Attached Figure Description

[0010] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0011] Figure 1 This is a schematic diagram of an application environment for a log data processing method according to an embodiment of the present invention; Figure 2 This is a flowchart of the log data processing method provided in the embodiments of this application; Figure 3 This is another flowchart of the log data processing method provided in the embodiments of this application; Figure 4 This is another flowchart of the log data processing method provided in the embodiments of this application; Figure 5 This is another flowchart of the log data processing method provided in the embodiments of this application; Figure 6 This is another flowchart of the log data processing method provided in the embodiments of this application; Figure 7 This is another flowchart of the log data processing method provided in the embodiments of this application; Figure 8 This is another flowchart of the log data processing method provided in the embodiments of this application; Figure 9 This is a schematic diagram of the log data processing device provided in the embodiments of this application; Figure 10 This is a schematic diagram of the structure of a computer device according to an embodiment of the present invention; Figure 11 This is another structural schematic diagram of a computer device according to one embodiment of the present invention. Detailed Implementation

[0012] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0013] It should be noted that although functional modules are divided in the device schematic diagram and a logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first," "second," etc., in the specification, claims, and the aforementioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

[0014] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0015] In financial insurance and healthcare systems, massive amounts of heterogeneous log data with specific business characteristics are typically generated. Accurate classification and clustering of this log data is fundamental to transforming unstructured text into structured data and subsequently conducting underlying system analysis. Currently, the industry typically processes this type of log data using conventional word frequency feature extraction combined with hard clustering algorithms, or directly introduces large-scale distributed computing frameworks for large-scale data parsing and analysis.

[0016] The inventors realized that when faced with highly specialized and information-intensive industry business data, existing solutions often only extract the surface features of general text. This approach is difficult to effectively analyze the complex business states and multiple semantics intertwined in a single log, resulting in low clustering accuracy of mixed logs containing specific industry features, which cannot meet the needs of complex business systems for high-precision analysis of underlying data.

[0017] Based on this, embodiments of this application provide a log data processing method, apparatus, device, and storage medium, aiming to solve the technical problem that existing financial data processing solutions suffer from low accuracy in matching heterogeneous data and high costs for manual intervention due to a lack of semantic understanding and fault tolerance mechanisms.

[0018] The log data processing method, apparatus, device, and storage medium provided in this application are specifically described through the following embodiments. First, the log data processing method in this application is described.

[0019] The log data processing method provided in this application embodiment can be applied to, for example, Figure 1In this application environment, the client communicates with the server via a network. The server can obtain the initial log data to be processed, perform word segmentation on the initial log data to obtain a log word sequence; extract the initial text weight features of the log word sequence, and perform weighted processing on the initial text weight features in combination with a preset industry-specific dictionary to obtain a log feature vector; input the log feature vector into a pre-trained clustering prediction model for processing to obtain the probability distribution of the initial log data belonging to multiple business clusters; determine the multi-semantic clustering result of the initial log data based on the probability distribution, and feed back the multi-semantic clustering result or interactive prompts generated based on the multi-semantic clustering result to the client. This application addresses the analysis needs of log data containing complex business states and professional terminology in business scenarios such as finance, insurance, and healthcare. First, it uses a weighted approach combining a pre-defined industry-specific dictionary with initial text weight features to accurately strengthen industry-specific keyword features and reduce interference from general terms, thereby generating a more accurate log feature vector. Then, this log feature vector is input into a clustering prediction model to output a probability distribution. By using the soft boundary of the probability distribution, it overcomes the limitations of traditional single hard clustering, enabling a single log entry to be reasonably mapped to multiple relevant business dimensions based on the probability distribution. This effectively improves the precision and objectivity of matching complex log features with multi-dimensional business clusters, solving the technical problem of low accuracy in multi-semantic log clustering in existing log data processing solutions. The client can be, but is not limited to, various personal computers, laptops, smartphones, tablets, and portable wearable devices. The server can be implemented using a standalone server or a server cluster consisting of multiple servers. The following detailed description of specific embodiments further illustrates this application.

[0020] Figure 2 This is an optional flowchart of the log data processing method provided in the embodiments of this application. Figure 2 The method may include, but is not limited to, steps S201 to S204.

[0021] Step S201: Obtain the initial log data to be processed, and perform word segmentation on the initial log data to obtain the log word sequence.

[0022] Step S202: Extract the initial text weight features of the log word sequence, and combine them with a preset industry-specific dictionary to perform weighted processing on the initial text weight features to obtain the log feature vector.

[0023] Step S203: Input the log feature vector into the pre-trained clustering prediction model for processing to obtain the probability distribution of the initial log data belonging to multiple business clusters.

[0024] Step S204: Determine the multi-semantic clustering result of the initial log data based on the attribution probability distribution.

[0025] In step S201 of some embodiments, initial log data refers to raw text records collected directly from the underlying layers of various business systems without structured parsing. For example, in a healthcare scenario, initial log data may be raw service call logs with timestamps and system error codes recorded by a hospital information system; in a financial insurance scenario, it may be unformatted transaction logs generated by a core transaction system. This initial log data typically exists in the form of long strings, containing rich business operation status and system operation details, but due to its unstructured nature, it is difficult to directly use for numerical calculations and model analysis. To transform unstructured text into computationally usable basic units, the initial log data needs to be segmented. Specifically, a segmentation algorithm is used to divide the continuous text string into individual lexical units with independent semantic functions. Assuming the total number of words obtained after segmentation is n, the generated log word sequence can be used with a set... To express mathematically, where This represents the i-th discrete log word in the sequence. Through word segmentation, the originally unordered long text is broken down into an ordered sequence of words, providing a data foundation for the subsequent extraction of the weight features of each word.

[0026] Please see Figure 3 In some embodiments, step S201 may include, but is not limited to, steps S301 to S303.

[0027] Step S301: Perform structure parsing on the initial log data to extract the unstructured text content from the initial log data.

[0028] Step S302: According to the preset entity recognition rules, the sensitive data in the unstructured text content is masked and replaced to obtain standardized text content.

[0029] Step S303: Perform word segmentation on the standardized text content and filter out preset stop words to obtain a log word sequence.

[0030] In step S301 of some embodiments, structural parsing refers to the process of stripping and disassembling the overall information of the log according to the preset generation template or data recording specification of the log; unstructured text content refers to the free text body of variable length, written in a mixture of natural language or code variables, remaining after removing fixed format fields such as timestamps, basic message headers, and communication protocol identifiers from the log records. Considering that the initial log data generated at the underlying level often exists in a semi-structured form, in order to focus on the core semantic information, it is necessary to first use regular expressions or preset parsing scripts to perform structural parsing on the initial log data to be processed. Through this parsing operation, the formatted packet headers and encapsulation structures that do not substantially help subsequent semantic clustering are discarded, thereby accurately extracting the unstructured text content that truly reflects the business operation status and abnormal operation details, thus locking in target data for subsequent in-depth text semantic mining.

[0031] In step S302 of some embodiments, the preset entity recognition rules refer to a set of matching logic pre-defined based on industry security compliance requirements, used to locate specific format strings in text; sensitive data refers to core elements involving user privacy or confidentiality, such as user bank accounts and transaction passwords in financial insurance scenarios, or patient medical insurance numbers and ID card numbers in medical and health scenarios; standardized text content refers to intermediate text that has been cleaned of the aforementioned sensitive privacy information and possesses unified specifications and security compliance. After extracting the unstructured text content in step S301, in order to meet the compliance requirements of highly privacy-sensitive industries such as finance and healthcare, it is necessary to call the preset entity recognition rules to perform a global scan of the above text content. Once sensitive data conforming to the above rules is identified in the unstructured text content, a mask replacement process is performed, for example, replacing the plaintext ID card number with a specific security placeholder "". "or a unified semantic tag"<ID_CARD> "Through this mask replacement, without disrupting the original log context syntax structure and business semantic coherence, the security risks of sensitive data leakage can be completely eliminated, thereby outputting secure and standardized text content."

[0032] In step S303 of some embodiments, stop words refer to redundant characters or words that frequently appear in natural language text but do not contribute to expressing core business semantics, such as common meaningless function words, conjunctions, and special punctuation marks. For standardized text content that has undergone security desensitization, a word segmentation engine is first used to segment it from a continuous long string into multiple discrete word units. Since the word set obtained from the initial segmentation inevitably contains a large number of noisy words with no actual analytical value, a preset stop word list is further introduced for comparison and cleaning, filtering out the matched stop words from the segmentation results. Assuming that the initial segmentation yields m words, and after stop word filtering, n words with actual semantic value are retained, these n retained words are finally arranged in their original order to generate a highly condensed and non-redundant log word sequence. .

[0033] Through steps S301 to S303 described above, this embodiment first performs structural parsing on the initial log data to be processed, stripping away the external encapsulation format and extracting the unstructured text content. Next, entity recognition rules are used to mask and replace sensitive data, maintaining the basic syntactic structure and contextual relationships of the log text while mitigating compliance risks associated with sensitive privacy information. Finally, word segmentation and stop word filtering remove redundant characters and interference from the text, yielding the corresponding log word sequence. These preprocessing steps effectively remove invalid information and reduce noise interference, providing standardized and compliant foundational data for subsequent text feature extraction and multi-semantic clustering analysis.

[0034] In step S202 of some embodiments, the initial text weight features are used to characterize the basic importance of each word in the log word sequence based on its frequency and distribution in the text's own context, reflecting the importance of the words at the general text level; the industry-specific dictionary is a pre-defined professional terminology database for a specific business domain, which includes keywords with clear business orientation and their corresponding enhanced weight coefficients. First, each log word in the aforementioned log word sequence W... Feature extraction is performed, and the initial text weight features of each word are calculated, denoted here as... Since basic feature extraction often fails to distinguish between general vocabulary and specialized terminology in specific business contexts, a pre-defined industry-specific dictionary is introduced to weight the initial text features. Adjustments are made. Specifically, the words in the log word sequence are compared with industry-specific dictionaries. If a log word matches a professional term in the dictionary (e.g., "medical insurance settlement interface" in the medical field, or "reversal timeout" in the financial insurance field), the corresponding weighting coefficient configured in the dictionary is obtained. If no match is found, the default baseline coefficient is used. Then, the initial text weight features are... with weighting coefficients Weighted calculations are performed to enhance the feature representation of professional terms with business value and suppress the weights of generic conjunctions or meaningless characters, thereby obtaining the weighted feature values. Finally, based on a specific dimension of the log word sequence, the weighted feature values ​​of each word are... By combining these features, a log feature vector is constructed to characterize the overall semantics and business characteristics of the initial log data, denoted as... .

[0035] Please see Figure 4 In some embodiments, step S202 may include, but is not limited to, steps S401 to S404.

[0036] Step S401: Calculate the initial text weight value of each log word in the log word sequence.

[0037] Step S402: Match and analyze the log words with the preset industry-specific dictionary. If the match is successful, obtain the corresponding specific weight coefficient.

[0038] Step S403: The initial text weight value and the specific weight coefficient are fused and calculated to obtain the weighted feature value.

[0039] Step S404: Construct a log feature vector based on the weighted feature values ​​of each log word in the log word sequence.

[0040] In step S401 of some embodiments, the generated log word sequence is scanned based on preset feature calculation rules, and the weight value of each log word at the basic text level is calculated and recorded as the initial text weight value. Specifically, this embodiment uses the Term Frequency-Inverse Document Frequency (TF-IDF) algorithm for quantization. First, the total number of words in the current initial log data is counted, and the Term Frequency (TF) of the current log word in that log is calculated—the ratio of the number of times the word appears to the total number of words. Simultaneously, the total number of historical log samples and the total number of historical logs containing the specific log word are obtained. Based on this, the Inverse Document Frequency (IDF) is calculated. The calculation logic is to divide the total number of log samples by the number of logs containing the word and take the logarithm of the quotient. Finally, the TF value and the IDF value are multiplied to obtain the initial text weight value of the log word. This step objectively quantifies the semantic basis of words from a general textual statistics perspective, providing a data foundation for subsequent weighted industry dictionary calculations.

[0041] In step S402 of some embodiments, the industry-specific dictionary is a high-value professional terminology database pre-built based on specific business domains (such as highly specialized business scenarios like finance and insurance, and healthcare). It contains a large number of professional terms with clear business orientation, fault indication, or abnormal state description, and pre-configures corresponding enhanced weight coefficients for these terms. In practice, each term in the log word sequence is sequentially compared with the entries in the industry-specific dictionary using string comparison or semantic alignment matching. If a log word successfully matches a professional term in the dictionary, the corresponding industry-specific weight coefficient configured in that dictionary is extracted. This coefficient is usually set to a value greater than 1; if it is not matched, the exclusive weight coefficient of the log term is set to the corresponding default baseline value, such as 1.0.

[0042] In step S403 of some embodiments, the initial text weight value obtained in step S401 is... The specific weighting coefficients determined in step S402 Perform multiplication numerical fusion operations to calculate The weighted eigenvalues ​​are then obtained and denoted as the weighted eigenvalues. This fusion computing logic can specifically amplify the features of professional terms that hit industry-specific dictionaries, while reducing the proportion of features of general, meaningless words, thereby objectively reflecting the relative importance of core words in specific business scenarios at the mathematical level.

[0043] In step S404 of some embodiments, the corresponding weighted feature values ​​are adjusted according to the original order of each log term in the log term sequence. Vectorization and concatenation are performed to construct the log feature vector. For example, the weighted feature values ​​of each word are combined sequentially to generate a high-dimensional log feature vector that represents the semantics of the initial log entry. Through the above steps S401 to S404, this embodiment first calculates the initial text weight value of the log words, then obtains the exclusive weight coefficient by matching with the industry-specific dictionary, and merges the two to calculate the weighted feature value. Finally, a log feature vector is constructed based on the weighted feature value. This can combine general text features and industry-specific dictionary information to generate a log feature vector that reflects the semantics of a specific business scenario, providing objective high-dimensional input data for the subsequent clustering prediction model analysis and processing.

[0044] Please see Figure 5 In some embodiments, the method provided in this application may also include, but is not limited to, steps S501 to S503.

[0045] Step S501: Extract unregistered words from historical business logs and calculate the co-occurrence frequency between unregistered words and existing industry feature words in the industry-specific dictionary.

[0046] Step S502: If the co-occurrence frequency is greater than the preset co-occurrence threshold, then the unregistered words are marked as candidate industry words.

[0047] Step S503: In response to receiving the review and confirmation instruction for candidate industry terms in the interactive interface, dynamically write the candidate industry terms and the corresponding weight coefficients assigned manually into the industry-specific dictionary.

[0048] In step S501 of some embodiments, "unregistered words" refers to discrete words that appear in the word segmentation results of historical business logs but have not yet been included in the industry-specific dictionary; co-occurrence frequency refers to the statistical frequency of unregistered words and existing industry feature words in the dictionary appearing together in the same context window or fixed text segment. Specifically, for unregistered words in historical business logs that have not been successfully matched by the industry-specific dictionary, the number of times they co-occur with existing industry feature words in the dictionary in the same context is counted, and the corresponding co-occurrence frequency is calculated through counting logic. For example, for newly added business terms or unrecorded words appearing in historical financial logs, the frequency of their co-occurrence with existing industry feature words "Timeout" or "Transfer" is counted, and the corresponding co-occurrence frequency value is quantified.

[0049] In step S502 of some embodiments, the co-occurrence threshold refers to a pre-set numerical limit used to determine whether an unregistered term possesses industry-specific attributes; candidate industry terms refer to unregistered terms that have passed co-occurrence screening and are awaiting manual review and confirmation. Specifically, the co-occurrence frequency calculated in step S501 is compared with the preset co-occurrence threshold. If the co-occurrence frequency exceeds the threshold, it indicates that the unregistered term is related to existing core business terms, and it is then screened and marked as a candidate industry term; if it does not exceed the threshold, it is not marked or filtered.

[0050] In step S503 of some embodiments, the review and confirmation instruction refers to the confirmation signal issued by the operator after reviewing the candidate industry terms through the interactive interface; the corresponding weight coefficient refers to the enhanced weight value manually assigned to the candidate industry term. Specifically, the marked candidate industry terms are displayed in the interactive interface. After receiving the review and confirmation instruction input by the manager and the corresponding weight coefficient manually assigned to the term, a dynamic writing operation is performed to supplement and update the candidate industry term and its exclusive weight coefficient into the industry-specific dictionary. For example, the candidate term is manually reviewed in the interactive interface, and after confirmation, it is written into the dictionary along with the assigned exclusive weight coefficient, realizing the dynamic expansion and updating of the dictionary.

[0051] Through steps S501 to S503 described above, this embodiment first extracts unregistered words from historical business logs and calculates their co-occurrence frequency with existing industry feature words. Then, it filters candidate industry words by comparing them with a preset co-occurrence threshold. Finally, in response to the review and confirmation instruction on the interactive interface, the candidate industry words and their corresponding manually assigned weight coefficients are dynamically written into the industry-specific dictionary. This process can dynamically expand the content of the industry-specific dictionary through co-occurrence analysis and manual review mechanisms, providing a continuously updated basic vocabulary for subsequent feature matching and weight calculation.

[0052] In step S203 of some embodiments, the pre-trained clustering prediction model refers to a machine learning model that, after iterative training and convergence using a large number of historical log samples, possesses the ability to classify and infer unknown feature vectors; business clusters refer to various log status categories divided at the business logic level, such as preset "payment gateway abnormal cluster" or "account frozen status cluster"; the attribution probability distribution quantifies the likelihood of the current initial log data belonging to each business cluster in the form of a numerical set. Based on the model and business classification criteria defined above, the log feature vector X constructed in the aforementioned steps is used as the input data of the pre-trained clustering prediction model. After receiving the log feature vector X, the model calculates the degree of association between the feature vector and the pre-set K business clusters through its internal parameter operation logic, and converts this degree of association into a probability distribution output. Specifically, this process outputs an attribution probability distribution with dimension K, denoted as... In this distribution, This represents the probability that the initial log data will be assigned to the k-th business cluster, and satisfies... Therefore, the output of the clustering prediction model is no longer an absolutely unique single classification label, but a soft probability set covering multiple business dimensions.

[0053] Please see Figure 6 In some embodiments, the pre-training process of the clustering prediction model in step S203 may include, but is not limited to, steps S601 to S605.

[0054] Step S601: Obtain the historical log sample set and extract the sample feature vector of each historical log sample in the historical log sample set.

[0055] Step S602: Obtain the distribution parameters configured for multiple business clusters in the initial clustering model.

[0056] Step S603: Based on the current distribution parameters, calculate the association probability value of each sample feature vector mapped to each business cluster.

[0057] Step S604: Iteratively update the distribution parameters of each business cluster based on the association probability value corresponding to each sample feature vector.

[0058] Step S605: In response to the updated distribution parameters satisfying the preset convergence condition, the converged initial clustering model is determined as the pre-trained clustering prediction model.

[0059] In some embodiments, steps S601 to S605 firstly involve obtaining a historical log sample set and extracting the sample feature vector of each historical log sample in the historical log sample set. The sample feature vector is obtained by weighting the initial text weight features of historical log samples with an industry-specific dictionary. Next, in step S602, the distribution parameters configured for multiple business clusters in the initial clustering model are obtained, such as the weights, mean vectors, or covariance matrices of each cluster. The initial clustering model in this application uses a Gaussian Mixture Model (GMM) to achieve probabilistic soft clustering. Before obtaining the distribution parameters, the model needs to automatically determine the number K of the multidimensional business clusters. Specifically, the total number of historical log samples currently input for training is counted, the square root of this total number of samples is calculated, and then the square root value is compared with a constant 2, and the larger of the two values ​​is automatically determined as the number K of business clusters. After determining the number K, the initial weights, mean vectors, and covariance matrices configured for these K business clusters in the initial clustering model are further obtained. Then, in step S603, based on the current distribution parameters, the association probability value of each sample feature vector mapped to each business cluster is calculated. This is used to characterize the probability that the feature vector of the i-th sample belongs to the j-th business cluster, so as to quantify the degree of association between the sample and each cluster; then in step S604, based on the association probability value corresponding to each sample feature vector... The distribution parameters of each business cluster are updated through iterative calculations, allowing the distribution parameters of the clustering model to continuously approximate the true distribution of the sample data. Specifically, the Expectation-Maximization (EM) algorithm can be used for iterative parameter optimization. In the expectation step (E-step), based on the current distribution parameters, the posterior probability of each sample feature vector generated by each business cluster (i.e., each Gaussian component) is calculated. In the maximization step (M-step), the posterior probabilities calculated in the E-step are used as weights to re-estimate and update the mean vector, covariance matrix, and mixture weight coefficients of each business cluster. Through the alternating iteration of the EM algorithm, the distribution parameters of the clustering model continuously approximate the true distribution of the sample data until the model converges. Finally, in step S605, in response to the updated distribution parameters satisfying the preset convergence condition, i.e., the preset threshold or iteration limit for determining that the model parameters tend to stabilize, the converged initial clustering model is determined as the pre-trained clustering prediction model.

[0060] Through steps S601 to S605 described above, this embodiment first obtains the sample feature vectors of the historical log sample set, then obtains the distribution parameters of the initial clustering model and calculates the association probability value, then iteratively updates the distribution parameters based on the association probability value, and finally determines the pre-trained clustering prediction model when the convergence condition is met. This process can optimize the distribution parameters of the clustering model through iterative training, providing a trained prediction model to support subsequent log clustering analysis.

[0061] In step S204 of some embodiments, the multi-semantic clustering result refers to the final determination result that the initial log data simultaneously contains or indicates multiple concurrent business states. It is mainly used to overcome the processing limitations of conventional hard clustering, which forcibly restricts a log entry to a single category. To obtain the aforementioned accurate multi-semantic clustering result, after obtaining the aforementioned attribution probability distribution... Then, a comprehensive judgment needs to be made based on the specific numerical values ​​of the distribution. For example, by analyzing the various probability values. Based on the differences in size and distribution, several business clusters with prominent probability values ​​are selected. In the healthcare registration and settlement scenario, if an abnormal log simultaneously implies both "registration interface error" and "insufficient out-of-pocket balance," since step S203 outputs a probability distribution, this log will exhibit relatively high probability values ​​in both corresponding business clusters. Therefore, based on the attribution probability distribution, the initial log data can be reasonably mapped to the identified multiple relevant business clusters, thus forming the final multi-semantic clustering result and objectively restoring the multiple concurrent attributes of complex log data in actual business operations.

[0062] Please see Figure 7In some embodiments, step S204 may include, but is not limited to, steps S701 to S703.

[0063] Step S701: Obtain the probability value corresponding to each business cluster in the attribution probability distribution.

[0064] Step S702: Compare each probability value with a preset attribution confidence threshold, and determine the business clusters whose corresponding probability values ​​are greater than or equal to the attribution confidence threshold as the target clusters.

[0065] Step S703: The set containing at least one target cluster is determined as the multi-semantic clustering result; wherein, in response to the number of target clusters being greater than or equal to two, the initial log data is determined to be a multi-semantic mixed log containing multiple business states.

[0066] In step S701 of some embodiments, the attribution probability distribution refers to the set of probability distributions calculated by the clustering prediction model, used to characterize the initial log data belonging to each business cluster; the probability value refers to the specific attribution probability value of the log data corresponding to each business cluster. Specifically, based on the output of the aforementioned clustering prediction model, the probability value corresponding to each business cluster in the attribution probability distribution is obtained. , is used to characterize the probability that the current log data belongs to the kth business cluster.

[0067] In step S702 of some embodiments, the attribution confidence threshold refers to a pre-set probability value limit used to filter valid attribution categories; the target cluster refers to a business cluster that meets the confidence requirement after comparison and filtering. Specifically, the probability values ​​obtained in step S701 are... Each is compared with the preset attribution confidence threshold. Perform a size comparison and assign the probability value to the corresponding attribute with a probability value greater than or equal to the attribute confidence threshold. The business clusters are selected and identified as the target clusters.

[0068] In step S703 of some embodiments, the multi-semantic clustering result refers to a set consisting of at least one target cluster; the multi-semantic hybrid log refers to log data that simultaneously contains multiple different business states. Specifically, the set containing at least one target cluster is determined as the multi-semantic clustering result, and when the number of target clusters is greater than or equal to two, the initial log data is determined to be a multi-semantic hybrid log that simultaneously contains multiple business states.

[0069] Through steps S701 to S703 described above, this embodiment first obtains the probability values ​​corresponding to each business cluster in the attribution probability distribution. Then, by comparing these probabilities with an attribution confidence threshold, business clusters with probability values ​​greater than or equal to the attribution confidence threshold are identified as target clusters. Finally, the set containing at least one target cluster is determined as the multi-semantic clustering result. When the number of target clusters is greater than or equal to two, the initial log data is determined to be a multi-semantic mixed log. This process can accurately identify and determine the multiple business states contained in the log data, providing objective and reliable classification results for log analysis in complex business scenarios.

[0070] Please see Figure 8 In some embodiments, after step S204, steps S801 to S802 may be included, but are not limited to.

[0071] Step S801: In response to detecting that the multi-semantic clustering results contain abnormal clusters that characterize abnormal system operation, corresponding hierarchical alarm information is generated based on the business dimension corresponding to the abnormal clusters and the probability value in the probability distribution of their affiliation.

[0072] Step S802: Distribute the hierarchical alarm information and initial log data to the corresponding business monitoring terminals according to the preset monitoring routing rules.

[0073] In step S801 of some embodiments, the abnormal cluster refers to the target cluster in the multi-semantic clustering results used to characterize the failure or abnormal state of the business process; the business dimension refers to the specific business type or functional module attribute pointed to by the abnormal cluster; and the hierarchical alarm information refers to alarm prompts with different urgency levels or grades generated based on the business dimension and the probability value in the attribution probability distribution. Specifically, the existence of abnormal clusters characterizing system malfunctions is detected in the multi-semantic clustering results. If they exist, the business dimension corresponding to the abnormal cluster and the probability value in the attribution probability distribution are combined, with the probability value used to quantify the degree of abnormal deviation or confidence level, for comprehensive calculation and evaluation to generate corresponding hierarchical alarm information.

[0074] In step S802 of some embodiments, the monitoring routing rule refers to a pre-configured target distribution strategy used to guide the flow of alarm information and log data; the business monitoring terminal refers to a terminal device or processing node used to receive alarm information and perform operation and maintenance display. Specifically, the hierarchical alarm information containing specific levels and the corresponding initial log data generated in step S801 are matched and distributed according to the preset monitoring routing rule, and the relevant data is transmitted to the corresponding business monitoring terminal for operation and maintenance personnel to perform subsequent anomaly handling and status monitoring.

[0075] To further reduce the deployment cost of heavy computing frameworks and improve real-time monitoring capabilities, the monitoring routing rules and terminal distribution mechanism in this embodiment rely on a lightweight hierarchical data storage architecture. Specifically, the system caches frequently triggered hierarchical alarm information and high-confidence clustering results in a Redis in-memory database to ensure an ultra-low response time of less than 10 milliseconds, meeting the real-time triggering requirements of multi-dimensional alarms. Simultaneously, the full attribution probability distribution data of multi-semantic clustering results and complete initial log data are persistently routed and stored in relational databases such as MySQL. This design, while ensuring alarm timeliness, provides complete data support for the subsequent generation of root cause analysis reports containing probability weights by the business monitoring terminals, achieving a balance between operational costs and computational efficiency.

[0076] Through the aforementioned steps S801 to S802, this embodiment first generates corresponding hierarchical alarm information based on the business dimension and probability value when an abnormal cluster is detected. Then, it distributes this information, along with initial log data, to the corresponding business monitoring terminal according to preset monitoring routing rules. This process can achieve accurate hierarchical alarms and targeted routing distribution for complex abnormal states, providing efficient and accurate fault response for system operation and maintenance.

[0077] Through the steps S201 to S204 described above, this application embodiment addresses the analysis needs of log data containing complex business states and professional terms in business scenarios such as finance, insurance, and healthcare. First, by combining a preset industry-specific dictionary with initial text weight features, it can accurately strengthen industry-specific keyword features to reduce the interference of general terms, thereby generating a more accurate log feature vector. Then, the log feature vector is input into a clustering prediction model to output a probability distribution. By using the soft boundary of the probability distribution to break through the limitations of traditional single hard clustering, a single log can be reasonably mapped to multiple related business dimensions based on the probability distribution. This effectively improves the precision and objectivity of matching complex log features with multi-dimensional business clusters, and can solve the technical problem of low accuracy in multi-semantic log clustering in existing log data processing solutions.

[0078] Please see Figure 9 This application also provides a log data processing apparatus that can implement the above-described log data processing method. The apparatus includes: The sequence acquisition module is used to acquire the initial log data to be processed and to perform word segmentation on the initial log data to obtain the log word sequence; The feature weighting extraction module is used to extract the initial text weight features of the log word sequence, and to perform weighting processing on the initial text weight features in combination with a preset industry-specific dictionary to obtain the log feature vector; The clustering processing module is used to input log feature vectors into a pre-trained clustering prediction model for processing, and to obtain the probability distribution of the initial log data belonging to multiple business clusters. The multi-semantic determination module is used to determine the multi-semantic clustering results of the initial log data based on the attribution probability distribution.

[0079] In some embodiments, the feature weighted extraction module is specifically used for: Calculate the initial text weight value for each log word in the log word sequence; The log words are matched and analyzed with the preset industry-specific dictionary. If the match is successful, the corresponding exclusive weight coefficient is obtained. The initial text weight values ​​are fused with the specific weight coefficients to obtain the weighted feature values; Log feature vectors are constructed based on the weighted feature values ​​of each log word in the log word sequence.

[0080] In some embodiments, the clustering processing module is further configured to: Obtain a historical log sample set and extract the sample feature vector of each historical log sample in the historical log sample set; wherein, the sample feature vector is obtained by weighting the initial text weight features of the historical log samples with an industry-specific dictionary; Obtain the distribution parameters configured for multiple business clusters in the initial clustering model; Based on the current distribution parameters, calculate the association probability value of each sample feature vector to each business cluster. Based on the correlation probability value corresponding to each sample feature vector, the distribution parameters of each business cluster are iteratively updated; In response to the updated distribution parameters satisfying the preset convergence condition, the initial clustering model after convergence is determined as the pre-trained clustering prediction model.

[0081] In some embodiments, the multiple semantic determination module is specifically used for: Obtain the probability value corresponding to each business cluster in the attribution probability distribution; Each probability value is compared with a preset attribution confidence threshold, and the business clusters whose corresponding probability values ​​are greater than or equal to the attribution confidence threshold are determined as the target clusters. A set containing at least one target cluster is identified as a multi-semantic clustering result; wherein, in response to the number of target clusters being greater than or equal to two, the initial log data is determined to be a multi-semantic mixed log containing multiple business states.

[0082] In some embodiments, the sequence acquisition module is specifically used for: The initial log data is parsed to extract the unstructured text content. Based on preset entity recognition rules, sensitive data in unstructured text content is masked and replaced to obtain standardized text content. Standardized text content is segmented into words and filtered for preset stop words to obtain a log word sequence.

[0083] In some embodiments, the feature weighted extraction module is further used for: Extract unregistered words from historical business logs and calculate the co-occurrence frequency between unregistered words and existing industry-specific words in the industry-specific dictionary; If the co-occurrence frequency is greater than the preset co-occurrence threshold, then the unregistered words will be marked as candidate industry words. In response to receiving a review and confirmation instruction for candidate industry terms in the interactive interface, the candidate industry terms and their corresponding manually assigned weight coefficients are dynamically written into the industry-specific dictionary.

[0084] In some embodiments, the multiple semantic determination module is further configured to: In response to the detection of abnormal clusters in the multi-semantic clustering results that represent abnormal system operation, corresponding hierarchical alarm information is generated based on the business dimension corresponding to the abnormal cluster and the probability value in the probability distribution of the affiliation. The tiered alarm information and initial log data are distributed to the corresponding business monitoring terminals according to the preset monitoring routing rules.

[0085] This invention provides a log data processing device. First, by combining a preset industry-specific dictionary with initial text weight features, it strengthens industry-specific keyword features to reduce interference from general vocabulary, thereby generating a more accurate log feature vector. Then, it inputs this log feature vector into a clustering prediction model to output an attribution probability distribution. By using the soft boundary of the probability distribution, it breaks through the limitations of traditional single hard clustering, enabling a single log entry to be reasonably mapped to multiple related business dimensions based on the probability distribution. This effectively improves the precision and objectivity of matching complex log features with multi-dimensional business clusters, and solves the technical problem of low accuracy in multi-semantic log clustering in existing log data processing solutions.

[0086] Specific limitations regarding the log data processing device can be found in the limitations of the log data processing method described above, and will not be repeated here. Each module in the aforementioned log data processing device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in the computer device in hardware form, or stored in the memory of the computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0087] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 10 As shown, the computer device includes a processor, memory, network interface, and database connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile and / or volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and database. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The network interface is used to communicate with external clients via a network connection. When the computer program is executed by the processor, it implements the functions or steps of a log data processing method on the server side.

[0088] In one embodiment, a computer device is provided, which may be a client, and its internal structure diagram may be as follows: Figure 11 As shown, the computer device includes a processor, memory, network interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The network interface is used to communicate with an external server via a network connection. When executed by the processor, the computer program implements a client-side function or step based on a log data processing method.

[0089] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to perform the following steps: Obtain the initial log data to be processed, and perform word segmentation on the initial log data to obtain the log word sequence; Extract the initial text weight features from the log word sequence, and then weight the initial text weight features using a pre-defined industry-specific dictionary to obtain the log feature vector; The log feature vectors are input into a pre-trained clustering prediction model for processing to obtain the probability distribution of the initial log data belonging to multiple business clusters. Based on the attribution probability distribution, determine the multi-semantic clustering results of the initial log data.

[0090] It should be noted that the functions or steps that can be implemented by the computer-readable storage medium or computer device described above can be referred to the relevant descriptions on the server side and client side in the foregoing method embodiments. To avoid repetition, they will not be described one by one here.

[0091] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0092] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is used as an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0093] It should be noted that any AI models, software tools, or components not belonging to this company appearing in the embodiments of this application are merely illustrative examples and do not represent actual use. All user personal information involved in the embodiments of this application has been authorized (with the knowledge and consent) by the relevant parties or has been fully authorized by all parties, and the executing entity may obtain it through various legal and compliant means. The collection, storage, use, processing, transmission, provision, and disclosure of the information, data, and signals involved all comply with relevant laws and regulations and do not violate public order and good morals.

[0094] The above-described embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.

Claims

1. A log data processing method, characterized in that, include: Obtain the initial log data to be processed, and perform word segmentation on the initial log data to obtain a log word sequence; Extract the initial text weight features of the log word sequence, and combine them with a preset industry-specific dictionary to perform weighted processing on the initial text weight features to obtain the log feature vector; The log feature vector is input into a pre-trained clustering prediction model for processing to obtain the probability distribution of the initial log data belonging to multiple business clusters. Based on the attribution probability distribution, the multi-semantic clustering result of the initial log data is determined.

2. The log data processing method according to claim 1, characterized in that, The initial text weight features of the log word sequence are extracted, and the initial text weight features are weighted in conjunction with a preset industry-specific dictionary to obtain a log feature vector, including: Calculate the initial text weight value for each log word in the log word sequence; The log terms are matched and analyzed with the preset industry-specific dictionary. If the match is successful, the corresponding specific weight coefficient is obtained. The initial text weight value and the specific weight coefficient are fused together to obtain the weighted feature value; The log feature vector is constructed based on the weighted feature values ​​of each log word in the log word sequence.

3. The log data processing method according to claim 1, characterized in that, The pre-training process of the clustering prediction model includes: Obtain a historical log sample set and extract the sample feature vector of each historical log sample in the historical log sample set; wherein, the sample feature vector is obtained by weighting the initial text weight features of the historical log sample with the industry-specific dictionary; Obtain the distribution parameters configured for multiple business clusters in the initial clustering model; Based on the current distribution parameters, calculate the association probability value of each sample feature vector to each business cluster. Based on the association probability value corresponding to each of the sample feature vectors, the distribution parameters of each of the business clusters are iteratively updated; In response to the update of the distribution parameters satisfying the preset convergence condition, the converged initial clustering model is determined as the pre-trained clustering prediction model.

4. The log data processing method according to claim 1, characterized in that, The step of determining the multi-semantic clustering result of the initial log data based on the attribution probability distribution includes: Obtain the probability value corresponding to each of the business clusters in the attribution probability distribution; Each probability value is compared with a preset attribution confidence threshold, and the business clusters whose corresponding probability values ​​are greater than or equal to the attribution confidence threshold are determined as target clusters. The set containing at least one of the target clusters is determined as the multi-semantic clustering result; wherein, in response to the number of target clusters being greater than or equal to two, the initial log data is determined to be a multi-semantic mixed log containing multiple business states simultaneously.

5. The log data processing method according to claim 1, characterized in that, The initial log data is segmented to obtain a log word sequence, including: The initial log data is parsed to extract the unstructured text content. According to the preset entity recognition rules, the sensitive data in the unstructured text content is masked and replaced to obtain standardized text content; The standardized text content is segmented into words, and preset stop words are filtered out to obtain the log word sequence.

6. The log data processing method according to claim 1, characterized in that, The method further includes: Extract unregistered words from historical business logs and calculate the co-occurrence frequency between the unregistered words and existing industry-specific words in the industry-specific dictionary; If the co-occurrence frequency is greater than a preset co-occurrence threshold, the unregistered words are marked as candidate industry words. In response to receiving a review and confirmation instruction for the candidate industry terms in the interactive interface, the candidate industry terms and their corresponding manually assigned weight coefficients are dynamically written into the industry-specific dictionary.

7. The log data processing method according to claim 1, characterized in that, After determining the multi-semantic clustering result of the initial log data, the method further includes: In response to the detection that the multi-semantic clustering result contains an abnormal cluster that represents an abnormal system operation, a corresponding hierarchical alarm message is generated based on the business dimension corresponding to the abnormal cluster and the probability value in the attribution probability distribution. The tiered alarm information and the initial log data are distributed to the corresponding business monitoring terminals according to the preset monitoring routing rules.

8. A log data processing device, characterized in that, The device includes: The sequence acquisition module is used to acquire the initial log data to be processed and to perform word segmentation on the initial log data to obtain a log word sequence; The feature weighting extraction module is used to extract the initial text weight features of the log word sequence, and to perform weighting processing on the initial text weight features in combination with a preset industry-specific dictionary to obtain the log feature vector; The clustering processing module is used to input the log feature vector into a pre-trained clustering prediction model for processing, so as to obtain the probability distribution of the initial log data belonging to multiple business clusters; The multi-semantic determination module is used to determine the multi-semantic clustering result of the initial log data based on the attribution probability distribution.

9. An electronic device, characterized in that, The electronic device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the log data processing method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the log data processing method according to any one of claims 1 to 7.

Images