Intelligent analysis method and system for unstructured text data based on large model

Abstract

The present application relates to the technical field of data processing, and more particularly to a method and system for intelligent analysis of unstructured text data based on a large model. The method comprises: obtaining unstructured text data to be analyzed, and performing cleaning and word segmentation processing to obtain a plurality of candidate words; calculating the global regular interference degree of each candidate word; based on the global regular interference degree, calculating the fault semantic specificity factor of each candidate word; determining the initial node weight of the candidate word in the TextRank algorithm; running the TextRank algorithm based on the initial node weight to extract an implicit fault chain summary; inputting the implicit fault chain summary into a large model, instructing the large model to perform fault evolution analysis and output maintenance suggestions. The present application optimizes keyword extraction through interference degree suppression, semantic factor enhancement and dynamic entropy analysis, thereby more accurately identifying fault words and improving the quality of large model input and the reliability of fault analysis.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data processing, and in particular to a method and system for intelligent analysis of unstructured text data based on a large model. BACKGROUND

[0002] In the operation and maintenance management of modern industry, especially coal mine equipment, the unstructured text data generated by front-line workers, such as equipment maintenance logs, shift handover records, and on-site inspection descriptions, contains key information about the evolution of equipment hidden faults. In order to mine key information about equipment faults, the industry usually uses large model technology for deep semantic analysis.

[0003] However, due to the characteristics of non-standard expression and high noise density of log data, directly inputting the original data into the large model will cause context window overflow or key information to be submerged. Therefore, the TextRank algorithm is usually used to extract keywords and summaries from the log as pre-input for the large model. The TextRank algorithm is based on graph sorting principles and can extract core words from text unsupervisedly.

[0004] However, the traditional TextRank algorithm mainly relies on the co-occurrence frequency and connectivity between words when calculating node weights. In unstructured text data such as equipment maintenance logs, there are a large number of high-frequency regular maintenance terms, such as normal inspection, equipment operation, and no abnormalities. Regular words have extremely high frequency and widely co-occur with various equipment words, resulting in regular words occupying extremely high weights in the word co-occurrence graph of the TextRank algorithm, and thus being misjudged as core keywords by the traditional TextRank algorithm. On the contrary, words that truly indicate hidden fault chains, such as microseisms and slight abnormal sounds, are submerged by high-frequency regular words and cannot be effectively extracted, resulting in invalid information being input into the large model, which prevents the large model from reasoning about the evolution of equipment faults and affects the identification of key fault information. SUMMARY

[0005] To solve the problem that the traditional TextRank algorithm is easily covered by high-frequency regular words, which prevents the subsequent large model from reasoning about the evolution of equipment faults and affects the accuracy of fault chain mining, the present application provides a method and system for intelligent analysis of unstructured text data based on a large model.

[0006] In the first aspect, the present application provides a method for intelligent analysis of unstructured text data based on a large model, which adopts the following technical solution:

[0007] The method comprises the following steps: obtaining unstructured text data to be analyzed, and performing cleaning and word segmentation processing to obtain a plurality of candidate words, and dividing the cleaned unstructured text data into a plurality of time windows; calculating a global regular interference degree of each candidate word, which is positively correlated with the total frequency of occurrence of each candidate word in all time windows and negatively correlated with the standard deviation of the frequency of occurrence of each candidate word in all time windows; based on the global regular interference degree, calculating a fault semantic specificity factor of each candidate word, which is positively correlated with the character length of each candidate word and negatively correlated with the global regular interference degree and the distribution entropy of adjacent nodes of each candidate word in a word co-occurrence network; normalizing the fault semantic specificity factor, and taking the normalized result as the initial weight of the corresponding candidate word in the TextRank algorithm; running the TextRank algorithm based on the initial weight of the node to extract an implicit fault chain summary; inputting the implicit fault chain summary into a large model to instruct the large model to perform fault evolution analysis and output a maintenance suggestion.

[0008] The application effectively identifies and suppresses the influence of high-frequency regular operation and maintenance terms by calculating the global regular interference degree; the application enhances the truly fault-related words by constructing the fault semantic specificity factor, and comprehensively considers the word length, regular interference degree and network distribution characteristics, so as to more accurately distinguish between regular words and fault-related words; the application improves the accuracy of keyword extraction by taking the fault semantic specificity factor as the initial weight of the node in the TextRank algorithm, and better avoids the problem that the traditional TextRank algorithm misjudges regular words as keywords in the equipment maintenance log; the application realizes comprehensive evaluation of the dynamic characteristics of words by time window division and distribution entropy analysis, and can more accurately extract keywords indicating the implicit fault chain, thereby providing high-quality input information for the large model and improving the accuracy of fault evolution analysis and the reliability of maintenance suggestions.

[0009] Further, the unstructured text data comprises maintenance logs of coal mine equipment, inspection records of coal mine equipment and shift handover records of coal mine production.

[0010] Further, the word segmentation processing comprises: performing word segmentation on the cleaned unstructured text data by using a word segmentation tool loaded with an industry-specific word library to obtain initial candidate words; removing general stop words in the initial candidate words and retaining a plurality of negative words to obtain candidate words.

[0011] Further, the global regular interference degree satisfies:

[0012] In the formula, is the candidate word a global regular interference degree of the candidate word, is a candidate word a standard deviation of the appearance frequency in all time windows, is a candidate word a standard deviation of the appearance frequency in all time windows, is a candidate word a mean value of the appearance frequency in all time windows, is a first hyperparameter, is a natural constant, is a natural logarithm function.

[0013] The application realizes the evaluation of the global regular interference degree by constructing a composite function containing a total frequency logarithmic term and a standard deviation exponential term, the total frequency term ensures that high-frequency words obtain higher interference degrees, and the standard deviation term suppresses words with uneven distribution through exponential attenuation, thereby effectively identifying and suppressing the influence of high-frequency regular operation terms, and providing a more reliable interference degree evaluation basis for the extraction of fault-related words.

[0014] Further, the calculation method of the fault semantic specificity factor is:

[0015] ; in the formula, is a candidate word a fault semantic specificity factor of the candidate word, is a candidate word a character length of the candidate word, is a candidate word a global regular interference degree of the candidate word, is a candidate word a distribution entropy of adjacent nodes in a word co-occurrence network, is a second hyperparameter, is a natural constant, is a max-min normalization function, is a natural logarithm function, is a natural exponential function.

[0016] The application realizes the comprehensive evaluation of the fault semantic specificity factor by constructing a composite function containing a character length logarithmic term, a regular interference degree normalization term and a network distribution entropy exponential term, the character length term ensures that long words obtain higher weights, the regular interference degree term suppresses high-frequency regular words, and the distribution entropy term evaluates the connection characteristics of words in the network, thereby more accurately distinguishing fault-related words from regular words.

[0017] Further, the initial weight satisfies:

[0018] ; in the formula, is a candidate word an initial weight of the candidate word, a fault semantic specificity factor of the candidate word, a total number of the candidate words.

[0019] The application realizes reasonable distribution of initial weights by normalizing the fault semantic specificity factor, ensures that the sum of weights of all candidate words is 1, avoids deviation in weight distribution, and the normalized weight accurately reflects the fault correlation of each candidate word, thereby providing a more reliable initial weight basis for the TextRank algorithm and improving the accuracy and pertinence of keyword extraction.

[0020] Further, the extracted implicit fault chain summary comprises: constructing a word co-occurrence graph based on the cleaned unstructured text data, taking the candidate words as nodes of the word co-occurrence graph, and taking the co-occurrence relationship between the candidate words within a time window as edges; assigning a value to each node in the word co-occurrence graph using the node initial weight; performing iterative calculation using the weighted jump probability formula until the node weight converges; sorting the converged node weight, and selecting a plurality of candidate words with the highest weight as key fault words; extracting a sentence sequence containing the key fault words from the uncleaned unstructured text data to obtain the implicit fault chain summary.

[0021] The application realizes more accurate identification of fault-related words by constructing a weighted word co-occurrence graph, ensures that the graph sorting algorithm pays attention to fault-related words first by using the fault semantic specificity factor as the initial weight, accurately extracts key fault words through iterative convergence and weight sorting, and extracts sentences containing key fault words from the original text, thereby maintaining semantic integrity and providing a high-quality fault chain summary for large model analysis.

[0022] Further, the damping coefficient of the TextRank algorithm is preset to 0.85, the maximum number of iterations is preset to 100, and the convergence threshold is preset to 0.0001.

[0023] Further, the instructions instruct the large model to perform fault evolution analysis and output maintenance suggestions, comprising: constructing a large model prompt word, inputting the implicit fault chain summary as context into the large model, instructing the large model to analyze the evolution trend of the corresponding device state, identifying potential causal fault chains, and outputting structured maintenance suggestions.

[0024] In a second aspect, the application provides an unstructured text data intelligent analysis system based on a large model, which adopts the following technical scheme:

[0025] ​The application discloses a big model-based intelligent analysis system for unstructured text data, which comprises a processor and a memory, wherein the memory stores computer program instructions.

[0026] By adopting the technical scheme, the big model-based intelligent analysis method for unstructured text data is generated into a computer program and stored in the memory to be loaded and executed by the processor, so that a terminal device is manufactured according to the memory and the processor, and use is facilitated.

[0027] The application has the following technical effects:

[0028] In view of the problem that, due to the dependence on co-occurrence frequency and connection degree, the traditional TextRank algorithm causes high-frequency routine maintenance words such as routine inspection and equipment operation to occupy core weights, and hides implicit fault words such as microseisms and slight abnormal sounds, the application evaluates the interference characteristics of routine words by global routine interference degree, the higher the total occurrence frequency and the more uniform the distribution, the higher the interference degree, and the initial weight of the node is inversely weakened in combination with a fault semantic specificity factor, so as to suppress the excessive influence of routine words and avoid that low-frequency but key fault indicating words are submerged by high-frequency interference words, thereby providing high-quality input information for extraction of an implicit fault chain.

[0029] The application breaks through the limitation of the traditional TextRank algorithm that only measures the importance of nodes in terms of co-occurrence relationship, and comprehensively considers the character length, global routine interference degree and adjacent node distribution entropy of candidate words by using a fault semantic specificity factor, so as to identify the semantic specificity of fault-related words in multiple dimensions, and then input the normalized factor as the initial weight of the node into the TextRank algorithm, so as to improve the weight proportion of low-frequency implicit fault words, and make the extracted abstract more accurately reflect the key information of equipment fault evolution, thereby forming a high-quality implicit fault chain abstract.

[0030] The implicit fault chain abstract extracted by the weighted TextRank algorithm eliminates a large amount of invalid routine maintenance information in the traditional method, and inputs only core semantic content directly related to faults into the big model, thereby avoiding the problems of context window overflow or key information being submerged by noise, making the big model more accurately capture the evolution logic of equipment implicit faults, such as a fault chain from slight abnormal sound to component wear, and then outputting a targeted maintenance suggestion, thereby solving the problem that the big model cannot effectively reason about fault evolution due to invalid information input in the traditional method, and improving the scientificity and timeliness of equipment maintenance decision. BRIEF DESCRIPTION OF DRAWINGS

[0031] Figure 1 The application discloses a big model-based intelligent analysis system for unstructured text data, which comprises a processor and a memory, wherein the memory stores computer program instructions.

[0032] Figure 2 is a comparison diagram of initial weights of candidate words of a traditional TextRank algorithm and an optimized TextRank algorithm in an embodiment of the present application based on large model intelligent analysis method of unstructured text data. DETAILED DESCRIPTION

[0033] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0034] The embodiments of the present application disclose a large model based unstructured text data intelligent analysis method, referring to Figure 1 , comprising steps S001-S006:

[0035] S001: obtaining unstructured text data to be analyzed, and performing cleaning and word segmentation processing to obtain a plurality of candidate words, and dividing the cleaned unstructured text data into a plurality of time windows.

[0036] Specifically, the unstructured text data such as the past year is exported from the coal mine equipment operation and maintenance management system, including the maintenance log of coal mine equipment, the inspection record of coal mine equipment, and the shift handover record of coal mine production. First, the text is de-duplicated and cleaned to remove random codes and meaningless symbols. Then, the text is segmented using a word segmentation tool loaded with a coal industry special vocabulary library, thereby obtaining a plurality of initial candidate words. Finally, general stop words are removed, but negative words such as no, none, and not are retained, and the cleaned unstructured text data is divided into a plurality of time windows in units of days.

[0037] S002: calculating the global regular interference degree of each candidate word.

[0038] It should be noted that the main purpose of this step is to identify background noise in the text data from a statistical point of view. In the coal mine log, normal, inspection, and other regular words not only have high total frequency, but also are extremely uniform in the time axis, almost appearing every day. In comparison, fault words usually have burstiness and sparsity. Therefore, the higher the total frequency of a candidate word, and the smaller the frequency fluctuation in each time window, the more uniform the distribution, the more the candidate word tends to be an invalid regular word, and the greater the interference degree of the candidate word to fault chain mining, and the greater the corresponding global regular interference degree.

[0039] The global regular interference degree is positively correlated with the total frequency of occurrence of each candidate word in all time windows, and is negatively correlated with the standard deviation of the frequency of occurrence of each candidate word in all time windows.

[0040] Specifically, the global regular interference degree satisfies:

[0041] ;

[0042] In the formula, is the global regular interference degree of the candidate word is the total frequency of occurrence of the candidate word in all time windows, is the standard deviation of the frequency of occurrence of the candidate word in all time windows, is the mean of the frequency of occurrence of the candidate word in all time windows, is a first hyperparameter, and an exemplary value is , which is used to prevent the denominator from being 0, resulting in a meaningless relationship, is a natural constant, is a natural logarithm function.

[0043] wherein, reflects the magnitude of the frequency, The larger the value is, the more common the candidate word is, and the greater the potential interference possibility is, and thus the global regular interference degree is greater. is a coefficient of variation, which reflects the degree of dispersion of data, The smaller the value is, the smaller the frequency fluctuation of the candidate word in each time window is, and the more uniform the distribution is, indicating that the candidate word has a greater possibility of being regular background noise, and thus the global regular interference degree is greater.

[0044] S003: Calculate a fault semantic specificity factor of each candidate word based on the global regular interference degree.

[0045] ​It should be noted that a special case exists in coal mine operation and maintenance scenarios: chronic systemic faults such as conveyor belt misalignment. These fault terms may also appear frequently within a certain period of time, resulting in high global general interference. Consequently, they are incorrectly identified as background noise that needs to be suppressed, leading to the loss of key chronic fault information. This step corrects this by introducing semantic topological features. General noise terms, such as status, usually co-occur with almost all equipment and components in the system, and their semantic adjacency nodes are very messy and diverse. However, although chronic fault terms, such as misalignment, are high-frequency, their semantics are very focused, usually only appearing with specific equipment, such as conveyor belts. In addition, in Chinese industrial texts, single-character words are often ambiguous in meaning, while multi-character words usually contain more specific equipment or fault information. Therefore, this step comprehensively considers the information entropy of the distribution of adjacent nodes and the word length to correct the global general interference. If the information entropy of the distribution of adjacent nodes of a candidate word is lower and the character length of the candidate word is longer, it indicates that the semantics of the candidate word is more focused. The candidate word is less likely to be a meaningless general word and more likely to be a chronic fault word that has been misjudged. Its weight should be greater and the corresponding fault semantic specificity factor should be greater.

[0046] The fault semantic specificity factor is positively correlated with the character length of each candidate word and negatively correlated with the global general interference degree and the distribution entropy of each candidate word's neighboring nodes in the word co-occurrence network.

[0047] Specifically, the calculation method for the fault semantic specificity factor is as follows:

[0048] ;

[0049] In the formula, Candidate words The semantic specificity factor of the fault, Candidate words The length of the characters, Candidate words Global normal interference, Candidate words Distribution entropy of neighboring nodes in a word co-occurrence network For example, the second hyperparameter is... This is used to prevent the denominator from being 0. It is a natural constant. For the minimum normalization function, It is the natural logarithm function. It is a natural exponential function.

[0050] in, The larger the value, the more likely it is to be a candidate word. The greater the likelihood that it is high-frequency background noise, the smaller the corresponding fault semantic specificity factor. The larger the value, the more likely it is to be a candidate word. The longer the character length, the greater the likelihood that it contains specific semantic information, the greater the weight it should be assigned, and the greater the corresponding fault semantic specificity factor. , Candidate words With the The co-occurrence probability of adjacent words, The smaller the value, the better the candidate word. The more concentrated the distribution of adjacent nodes, the more focused the semantics, the more it conforms to the characteristics of chronic fault words, and the greater the corresponding fault semantic specificity factor.

[0051] S004: Determine the initial node weights of candidate words in the TextRank algorithm.

[0052] It should be noted that, having obtained the fault semantic specificity factor that can accurately distinguish between noise and fault information, this step will optimize the initialization process of the TextRank algorithm based on the fault semantic specificity factor. The traditional TextRank algorithm usually sets the weight of all nodes to the default value of 1 before the iteration begins. This invention uses the fault semantic specificity factor to assign differentiated weights to nodes, so that the TextRank algorithm focuses on key information at the beginning of the iteration.

[0053] The fault semantic specificity factor is normalized, and the normalization result is used as the initial node weight of the corresponding candidate word in the TextRank algorithm.

[0054] Specifically, the initial weights satisfy:

[0055] ;

[0056] In the formula, Candidate words The initial weights, Candidate words The semantic specificity factor of the fault, The total number of candidate words.

[0057] like Figure 2As shown, in the traditional TextRank algorithm, the initial weight of the high-frequency general word is high, and the microseismicity of the fault-related word is obviously low, which leads to the fault word being easily covered by the general word; the optimized TextRank algorithm of the scheme reduces the running weight and significantly increases the microseismicity weight, thereby verifying the optimization effect of the application, suppressing the weight of the general word through the global general interference degree, enhancing the weight of the fault word with the help of the fault semantic specificity factor, realizing the weight inversion of the key fault word and the general word, and enabling the implicit fault-related word to obtain higher attention at the beginning of the TextRank iteration, thereby providing more reliable data support for subsequent accurate extraction of the implicit fault chain summary.

[0058] S005: running a TextRank algorithm based on the node initial weight to extract an implicit fault chain summary.

[0059] Specifically, the extraction of the implicit fault chain summary comprises:

[0060] Based on the cleaned unstructured text data, a word co-occurrence graph is constructed, the candidate words are taken as nodes of the word co-occurrence graph, and the co-occurrence relationship between the candidate words within a time window is taken as an edge.

[0061] Each node in the word co-occurrence graph is valued using the node initial weight.

[0062] Iterative calculation is performed using the weighted jump probability formula until the node weight converges.

[0063] The converged node weight is sorted, and a plurality of candidate words with the highest weight are selected as key fault words; for example, the number of the plurality of candidate words with the highest weight is 5% of the total number of all candidate words.

[0064] In the uncleaned unstructured text data, a sentence sequence containing the key fault words is extracted to obtain an implicit fault chain summary.

[0065] The damping coefficient of the TextRank algorithm is pre-set to 0.85, for example; the maximum number of iterations is pre-set to 100, for example; and the convergence threshold is pre-set to 0.0001, for example.

[0066] S006: inputting the implicit fault chain summary into a large model and instructing the large model to perform fault evolution analysis and output maintenance suggestions.

[0067] Specifically, the instruction to the large model to perform fault evolution analysis and output maintenance suggestions comprises:

[0068] A large model prompt word is constructed, the implicit fault chain summary is input into the large model as a context, the large model is instructed to analyze the evolution trend of the corresponding device state, identify potential causal fault chains, and output structured maintenance suggestions.

[0069] The embodiment of the present application also discloses a large model-based intelligent analysis system for unstructured text data, comprising a processor and a memory, and the memory stores computer program instructions which realize the large model-based intelligent analysis method for unstructured text data according to the present application when executed by the processor.

[0070] The above system also comprises other components such as a communication bus and a communication interface which are well known to those skilled in the art, and the settings and functions thereof are known in the art, so they will not be described here.

[0071] The above are preferred embodiments of the present application, which do not limit the protection scope of the present application, therefore: any equivalent changes made on the structure, shape and principle of the present application should be covered within the protection scope of the present application.

Claims

1. A method for intelligent analysis of unstructured text data based on a large model, characterized by, The method comprises the following steps: obtaining unstructured text data to be analyzed, and performing cleaning and word segmentation processing to obtain a plurality of candidate words, and dividing the cleaned unstructured text data into a plurality of time windows; calculating a global regular interference degree of each candidate word, which is positively correlated with the total frequency of occurrence of each candidate word in all time windows and negatively correlated with the standard deviation of the frequency of occurrence of each candidate word in all time windows; based on the global regular interference degree, calculating a fault semantic specificity factor of each candidate word, which is positively correlated with the character length of each candidate word and negatively correlated with the global regular interference degree and the distribution entropy of adjacent nodes of each candidate word in a word co-occurrence network; normalizing the fault semantic specificity factor, and taking the normalized result as the initial weight of the node of the corresponding candidate word in the TextRank algorithm; running the TextRank algorithm based on the initial weight of the node, and extracting an implicit fault chain summary; inputting the implicit fault chain summary into a large model, and instructing the large model to perform fault evolution analysis and output maintenance suggestions. 2.The method of claim 1, wherein, The unstructured text data includes maintenance logs of coal mine equipment, inspection records of coal mine equipment, and shift handover records of coal mine production. 3.The method of claim 1, wherein, The word segmentation processing comprises the following steps: segmenting the cleaned unstructured text data using a word segmentation tool loaded with an industry-specific word library to obtain initial candidate words; removing general stop words from the initial candidate words and retaining a plurality of negative words to obtain candidate words. 4.The method of claim 1, wherein, The global regular interference degree satisfies: ； wherein, the global regularity of the candidate word , the global regularity of the candidate word , the global regularity of the candidate word , the global regularity of the candidate word , is a first hyperparameter, is a natural constant, is a natural logarithm function. 5.The method of claim 1, wherein, The calculation method of the fault semantic specificity factor is: ； wherein, is a failure semantics specificity factor for the candidate word , is a character length for the candidate word , is a global regularity of interference for the candidate word , is a distribution entropy of the neighboring nodes in the word co-occurrence network for the candidate word , is a second hyperparameter, is a natural constant, is a max-min normalization function, is a natural logarithm function, is a natural exponential function. 6.The method of claim 1, wherein, The initial weight satisfies: ； wherein, is the initial weight of the candidate word is the failure semantic specificity factor of the candidate word is the initial weight of the candidate word is the failure semantic specificity factor of the candidate word is the total number of candidate words. 7.The method of claim 1, wherein, The extraction of the implicit fault chain summary comprises the following steps: constructing a word co-occurrence graph based on the cleaned unstructured text data, taking the candidate words as nodes of the word co-occurrence graph, and taking the co-occurrence relationship between the candidate words in the time window as edges; assigning a value to each node in the word co-occurrence graph using the initial weight of the node; performing iterative calculation using a weighted jump probability formula until the node weight converges; sorting the converged node weight, and selecting a plurality of candidate words with the highest weight as key fault words; extracting a sentence sequence containing the key fault words from the uncleaned unstructured text data to obtain an implicit fault chain summary. 8.The method of claim 7, wherein, The damping coefficient of the TextRank algorithm is preset to 0.85, the maximum number of iterations is preset to 100, and the convergence threshold is preset to 0.0001. 9.The method of claim 1, wherein, The instruction to the large model to perform fault evolution analysis and output maintenance suggestions comprises the following steps: constructing a large model prompt word, inputting the implicit fault chain summary as a context into the large model, instructing the large model to analyze the evolution trend of the corresponding device state, identifying a potential causal fault chain, and outputting a structured maintenance suggestion.

10. A large model-based intelligent analysis system for unstructured text data, characterized by, The method comprises the following steps: a processor and a memory, the memory storing computer program instructions which, when executed by the processor, implement the large model-based intelligent analysis method of unstructured text data according to any one of claims 1-9.

Images