Unstructured text data intelligent analysis method and system based on large model

Abstract

The invention relates to the technical field of data processing, in particular to an unstructured text data intelligent analysis method and system based on a large model. The method comprises the steps of obtaining to-be-analyzed unstructured text data, and performing cleaning and word segmentation processing to obtain a plurality of candidate words; calculating the global conventional interference degree of each candidate word; calculating a fault semantic specificity factor of each candidate word based on the global conventional interference degree; node initial weights of the candidate words in the TextRank algorithm are determined; running a TextRank algorithm based on the node initial weight, and extracting to obtain a hidden fault chain abstract; and inputting the hidden fault chain abstract into a large model, instructing the large model to carry out fault evolution analysis and outputting a maintenance suggestion. According to the method, keyword extraction is optimized through interference degree suppression, semantic factor enhancement and dynamic entropy analysis, so that fault vocabularies are recognized more accurately, and the large model input quality and the fault analysis reliability are improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data processing, and in particular to a method and system for intelligent analysis of unstructured text data based on a large model. BACKGROUND

[0002] In the operation and maintenance management of modern industry, especially coal mine equipment, the unstructured text data generated by front-line workers, such as equipment maintenance logs, shift handover records, and on-site inspection descriptions, contains key information about the evolution of hidden equipment failures. In order to mine key information about equipment failures, the industry usually uses large model technology for deep semantic analysis.

[0003] However, due to the characteristics of non-standard expression and high noise density of log data, directly inputting the original data into the large model will cause context window overflow or key information to be submerged. Therefore, the TextRank algorithm is usually used to extract keywords and summaries from the logs as pre-input for the large model. The TextRank algorithm is based on graph sorting principles and can extract core words from text unsupervisedly.

[0004] However, the traditional TextRank algorithm mainly relies on the co-occurrence frequency and connectivity between words when calculating node weights. In unstructured text data such as equipment maintenance logs, there are a large number of high-frequency regular maintenance terms, such as normal inspection, equipment operation, and no abnormalities. Due to the extremely high frequency of occurrence and wide co-occurrence with various equipment words, regular words occupy a very high weight in the word co-occurrence graph of the TextRank algorithm, and are therefore misjudged as core keywords by the traditional TextRank algorithm. On the contrary, words that truly indicate hidden fault chains, such as microseisms and slight abnormal sounds, are submerged by high-frequency regular words due to their relatively low frequency of occurrence, and cannot be effectively extracted, resulting in a large amount of invalid information being input into the large model, which prevents the large model from reasoning about the evolution of equipment failures and affects the identification of key fault information. SUMMARY

[0005] To solve the problem that the traditional TextRank algorithm is easily covered by high-frequency regular words, which prevents the subsequent large model from reasoning about the evolution of equipment failures and affects the accuracy of fault chain mining, the present application provides a method and system for intelligent analysis of unstructured text data based on a large model.

[0006] In a first aspect, the present application provides a method for intelligent analysis of unstructured text data based on a large model, which adopts the following technical solution: A large-scale model-based intelligent analysis method for unstructured text data includes: acquiring unstructured text data to be analyzed, cleaning and segmenting it to obtain several candidate words, and dividing the cleaned unstructured text data into multiple time windows; calculating the global general interference degree of each candidate word, which is positively correlated with the total frequency of occurrence of each candidate word in all time windows and negatively correlated with the standard deviation of the frequency of occurrence of each candidate word in all time windows; calculating the fault semantic specificity factor of each candidate word based on the global general interference degree, which is positively correlated with the character length of each candidate word and negatively correlated with the global general interference degree and the distribution entropy of the neighboring nodes of each candidate word in the word co-occurrence network; normalizing the fault semantic specificity factor and using the normalization result as the initial node weight of the corresponding candidate word in the TextRank algorithm; running the TextRank algorithm based on the initial node weight to extract a hidden fault chain summary; inputting the hidden fault chain summary into a large-scale model, instructing the large-scale model to perform fault evolution analysis and output maintenance suggestions.

[0007] This invention effectively identifies and suppresses the influence of high-frequency routine maintenance terms by calculating global routine interference. Through the construction of fault semantic specificity factors, it enhances truly fault-related vocabulary, comprehensively considering vocabulary length, routine interference, and network distribution characteristics, enabling more accurate differentiation between routine and fault-related vocabulary. By using fault semantic specificity factors as the initial node weights in the TextRank algorithm, it improves the accuracy of keyword extraction, better avoiding the problem of traditional TextRank algorithms misclassifying routine words as keywords in equipment maintenance logs. Through time window partitioning and distribution entropy analysis, it achieves a comprehensive evaluation of the dynamic features of vocabulary, enabling more accurate extraction of keywords indicating hidden fault chains, providing high-quality input information for large models, and improving the accuracy of fault evolution analysis and the reliability of maintenance recommendations.

[0008] Furthermore, the unstructured text data includes maintenance logs of coal mine equipment, inspection records of coal mine equipment, and shift handover records of coal mine production.

[0009] Furthermore, the word segmentation process includes: using a word segmentation tool loaded with an industry-specific thesaurus to segment the cleaned unstructured text data to obtain initial candidate words; removing common stop words from the initial candidate words and retaining multiple negative words to obtain candidate words.

[0010] Furthermore, the global normal interference degree satisfies: In the formula, Candidate words Global normal interference, Candidate words Total frequency of occurrence across all time windows Candidate words The standard deviation of the frequency of occurrence across all time windows. Candidate words The mean frequency of occurrence across all time windows. As the first hyperparameter, It is a natural constant. It is the natural logarithm function.

[0011] This invention achieves global general interference assessment by constructing a composite function containing a logarithmic term of total frequency and an exponential term of standard deviation. The total frequency term ensures that high-frequency words have a high interference level, while the standard deviation term suppresses unevenly distributed words through exponential decay. This effectively identifies and suppresses the influence of high-frequency general maintenance terms, providing a more reliable basis for interference assessment for the extraction of fault-related terms.

[0012] Furthermore, the calculation method for the fault semantic specificity factor is as follows: In the formula, Candidate words The semantic specificity factor of the fault, Candidate words The length of the characters, Candidate words Global normal interference, Candidate words Distribution entropy of neighboring nodes in a word co-occurrence network This is the second hyperparameter. It is a natural constant. For the minimum normalization function, It is the natural logarithm function. It is a natural exponential function.

[0013] This invention achieves a comprehensive evaluation of fault semantic specificity factors by constructing a composite function that includes a logarithmic term for character length, a normalized term for conventional interference, and a network distribution entropy exponent. The character length term ensures that long words receive higher weights, the conventional interference term suppresses high-frequency conventional words, and the distribution entropy term evaluates the connectivity characteristics of words in the network, thereby enabling a more accurate distinction between fault-related words and conventional words.

[0014] Furthermore, the initial weights satisfy: In the formula, Candidate words The initial weights, Candidate words a failure semantic specificity factor, total number of candidate words.

[0015] The application realizes reasonable distribution of initial weights by normalizing the failure semantic specificity factor, ensures that the sum of all candidate word weights is 1, avoids deviation in weight distribution, and the normalized weight accurately reflects the failure correlation of each candidate word, thereby providing a more reliable initial weight basis for the TextRank algorithm and improving the accuracy and pertinence of keyword extraction.

[0016] Further, the extraction obtains an implicit failure chain summary, including: constructing a word co-occurrence graph based on the cleaned unstructured text data, taking the candidate words as nodes of the word co-occurrence graph, and taking the co-occurrence relationship between the candidate words within a time window as edges; using the node initial weight to assign a value to each node in the word co-occurrence graph; using a weighted jump probability formula to perform iterative calculation until the node weight converges; sorting the converged node weight, and selecting a plurality of candidate words with the highest weight as key failure words; extracting a sentence sequence containing the key failure words from the uncleaned unstructured text data to obtain an implicit failure chain summary.

[0017] The application realizes more accurate identification of failure-related words by constructing a weighted word co-occurrence graph, ensures that the graph sorting algorithm pays attention to failure-related words first by using the failure semantic specificity factor as the initial weight, accurately extracts key failure words through iterative convergence and weight sorting, and extracts sentences containing key failure words from the original text, thereby maintaining semantic integrity and providing a high-quality failure chain summary for large model analysis.

[0018] Further, the damping coefficient of the TextRank algorithm is preset to 0.85, the maximum number of iterations is preset to 100, and the convergence threshold is preset to 0.0001.

[0019] Further, the instructions instruct the large model to perform failure evolution analysis and output maintenance suggestions, including: constructing a large model prompt word, inputting the implicit failure chain summary as context into the large model, instructing the large model to analyze the evolution trend of the corresponding device state, identifying potential causal failure chains, and outputting structured maintenance suggestions.

[0020] In a second aspect, the application provides a large model-based intelligent analysis system for unstructured text data, which adopts the following technical scheme: The large model-based intelligent analysis system for unstructured text data includes a processor and a memory, and the memory stores computer program instructions, which, when executed by the processor, implement the above-mentioned large model-based intelligent analysis method for unstructured text data.

[0021] By adopting the technical scheme, the above-mentioned big model-based intelligent analysis method for unstructured text data is generated into a computer program and stored in a memory to be loaded and executed by a processor, so that a terminal device is manufactured according to the memory and the processor, and use is facilitated.

[0022] The present application has the following technical effects: In view of the problem that the traditional TextRank algorithm depends on co-occurrence frequency and connection degree, leading to the problem that high-frequency routine maintenance words such as routine inspection and equipment operation occupy the core weight, and cover up implicit fault words such as microseismic and slight abnormal sound, the present application evaluates the interference characteristics of routine words through global routine interference degree, the higher the total occurrence frequency and the more uniform the distribution, the higher the interference degree, and combines a fault semantic specificity factor to weaken the initial weight of the node in reverse, thereby inhibiting the excessive influence of routine words and avoiding that low-frequency but key fault indicating words are submerged by high-frequency interference words, to provide high-quality input information for the extraction of implicit fault chains.

[0023] The present application breaks through the limitation of the traditional TextRank algorithm that only measures the importance of nodes in terms of co-occurrence relationship, and comprehensively considers the character length, global routine interference degree and adjacent node distribution entropy of candidate words through a fault semantic specificity factor, identifies the semantic specificity of fault-related words through multi-dimensional features, and inputs the normalized factor as the initial weight of the node into the TextRank algorithm, so as to improve the weight proportion of low-frequency implicit fault words, and make the extracted abstract more accurately reflect the key information of the evolution of equipment faults, and form a high-quality implicit fault chain abstract.

[0024] The implicit fault chain abstract extracted by the weighted TextRank algorithm eliminates a large amount of invalid routine maintenance information in the traditional method, and inputs only the core semantic content directly related to the fault into the big model, avoiding the problem of context window overflow or key information being submerged by noise, so that the big model can more accurately capture the evolution logic of the implicit fault of the equipment, such as the fault chain from slight abnormal sound to component wear, and then output targeted maintenance suggestions, solving the problem that the big model cannot effectively reason about the evolution of the fault due to invalid information input in the traditional method, and improving the scientificity and timeliness of equipment maintenance decisions. BRIEF DESCRIPTION OF DRAWINGS

[0025] Figure 1 is a method flowchart in the big model-based intelligent analysis method for unstructured text data according to an embodiment of the present application.

[0026] Figure 2 is a comparison diagram of the initial weight of candidate words of the traditional TextRank algorithm and the optimized TextRank algorithm of the present application in the big model-based intelligent analysis method for unstructured text data according to an embodiment of the present application. DETAILED DESCRIPTION

[0027] With reference to the drawings of the embodiments of the present application, the technical solutions in the embodiments of the present application will be clearly and completely described. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without creative work fall within the scope of the present application.

[0028] The embodiments of the present application disclose a method for intelligent analysis of unstructured text data based on a large model, referring to Figure 1 , comprising steps S001-S006: S001: obtaining unstructured text data to be analyzed, and performing cleaning and word segmentation processing to obtain a plurality of candidate words, and dividing the cleaned unstructured text data into a plurality of time windows.

[0029] Specifically, unstructured text data such as data in the past year is exported from a coal mine equipment operation and maintenance management system, including maintenance logs of coal mine equipment, inspection records of coal mine equipment, and shift handover records of coal mine production. First, the text is de-duplicated and cleaned to remove random codes and meaningless symbols. Then, a word segmentation tool loaded with a special vocabulary for the coal industry is used to perform word segmentation processing on the text, thereby obtaining a plurality of initial candidate words. Finally, general stop words are removed, but negative words such as no, none, and not that are crucial to fault judgment are retained, and the cleaned unstructured text data is divided into a plurality of time windows in units of days.

[0030] S002: calculating the global regular interference degree of each candidate word.

[0031] It should be noted that the main purpose of this step is to identify background noise in the text data from a statistical point of view. In coal mine logs, normal, inspection, and other regular words not only have high total frequencies, but also have extremely uniform distribution on the time axis, almost appearing every day. In comparison, fault words usually have burstiness and sparsity. Therefore, the higher the total frequency of a candidate word, and the smaller the frequency fluctuation and the more uniform the distribution of the candidate word in each time window, the more the candidate word tends to be an invalid regular word, and the greater the interference degree of the candidate word to fault chain mining, and the greater the corresponding global regular interference degree.

[0032] The global regular interference degree is positively correlated with the total appearance frequency of each candidate word in all time windows, and negatively correlated with the standard deviation of the appearance frequency of each candidate word in all time windows.

[0033] Specifically, the global regular interference degree satisfies: ; In the formula, is the global regular interference degree of the candidate word , is the candidate word is the total frequency of appearance in all time windows, is the candidate word is the standard deviation of the frequency of appearance in all time windows, is the candidate word is the mean of the frequency of appearance in all time windows, is the first hyper-parameter, exemplary is used to prevent the denominator from being 0, causing the relationship to be meaningless, is a natural constant, is a natural logarithm function.

[0034] wherein, reflects the magnitude of the word frequency, The larger, the more common the candidate word, the greater the potential interference, and the greater the global regular interference degree. is the coefficient of variation, reflecting the degree of dispersion of the data, The smaller, the smaller the frequency fluctuation of the candidate word in each time window, the more uniform the distribution, the greater the possibility of the candidate word as regular background noise, and the greater the global regular interference degree.

[0035] S003: Calculate the fault semantic specificity factor of each candidate word based on the global regular interference degree.

[0036] It should be noted that there is a special case in the coal mine operation and maintenance scene, namely chronic systemic failure such as belt deviation, etc. Such fault words may also appear frequently in a certain period of time, resulting in a higher global regular interference degree, so they are incorrectly determined as background noise that needs to be suppressed, resulting in the loss of key chronic fault information. This step is modified by introducing semantic topology features. Regular noise words, such as states, usually co-occur with almost all devices and components in the system, and their semantic adjacent nodes are very chaotic and diverse. Chronic fault words, such as deviation, although high frequency, their semantics are very focused and usually only co-occur with specific devices such as belts. In addition, in Chinese industrial text, single-character words often have ambiguous meaning, while multi-character words usually contain more specific device or fault information. Therefore, this step considers the information entropy of adjacent node distribution and word length to modify the global regular interference degree. If the information entropy of the adjacent node distribution of a candidate word is lower, and the character length of the candidate word is longer, it indicates that the semantic of the candidate word is more focused, the candidate word is less likely to be meaningless regular word, and more likely to be misjudged chronic fault word, and its weight should be greater, and the corresponding fault semantic specificity factor is greater.

[0037] The failure semantic specificity factor is positively correlated with the character length of each candidate word, and is negatively correlated with the global regular interference degree, and the distribution entropy of the adjacent nodes of each candidate word in the word co-occurrence network.

[0038] Specifically, the calculation method of the failure semantic specificity factor is: ; In the formula, is the failure semantic specificity factor of the candidate word , is the character length of the candidate word , is the global regular interference degree of the candidate word , is the distribution entropy of the adjacent nodes of the candidate word in the word co-occurrence network, is a second hyperparameter, and an exemplary value is , used to prevent the denominator from being 0, is a natural constant, is a max-min normalization function, is a natural logarithm function, is a natural exponential function.

[0039] Wherein, The larger the value is, the greater the possibility of the candidate word as high-frequency background noise is, and the smaller the corresponding failure semantic specificity factor is. The larger the value is, the longer the character length of the candidate word is, the greater the possibility of containing specific semantic information is, the greater the weight should be given, and the larger the corresponding failure semantic specificity factor is. , is the co-occurrence probability of the candidate word and the first adjacent word, The smaller the value is, the more concentrated the adjacent node distribution of the candidate word is, the more focused the semantics is, the more in line with the characteristics of the chronic failure word is, and the larger the corresponding failure semantic specificity factor is.

[0040] S004: Determine the initial node weight of the candidate word in the TextRank algorithm.

[0041] It should be noted that, having obtained the fault semantic specificity factor that can accurately distinguish between noise and fault information, this step will optimize the initialization process of the TextRank algorithm based on the fault semantic specificity factor. The traditional TextRank algorithm usually sets the weight of all nodes to the default value of 1 before the iteration begins. This invention uses the fault semantic specificity factor to assign differentiated weights to nodes, so that the TextRank algorithm focuses on key information at the beginning of the iteration.

[0042] The fault semantic specificity factor is normalized, and the normalization result is used as the initial node weight of the corresponding candidate word in the TextRank algorithm.

[0043] Specifically, the initial weights satisfy: ; In the formula, Candidate words The initial weights, Candidate words The semantic specificity factor of the fault, The total number of candidate words.

[0044] like Figure 2 As shown, in the traditional TextRank algorithm, high-frequency regular words have a higher initial weight, while fault-related words and micro-vibrations have a significantly lower weight, making fault words easily masked by regular words. The optimized TextRank algorithm in this scheme reduces the weight of regular words and significantly increases the weight of micro-vibrations, thus verifying the optimization effect of this invention. By suppressing the weight of regular words through global regular interference and enhancing the weight of fault words with fault semantic specificity factors, the weight reversal between key fault words and regular words is achieved, allowing latent fault-related words to receive higher attention at the beginning of the TextRank iteration, providing more reliable data support for the subsequent accurate extraction of latent fault chain summaries.

[0045] S005: Run the TextRank algorithm based on the initial weights of the nodes to extract a summary of the hidden fault chain.

[0046] Specifically, the extraction of the hidden fault chain summary includes: A word co-occurrence graph is constructed based on the cleaned unstructured text data, with candidate words as nodes in the word co-occurrence graph and the co-occurrence relationships between candidate words within a time window as edges; The initial weights of the nodes are used to assign values ​​to each node in the word co-occurrence graph; The weighted jump probability formula is used for iterative calculation until the node weights converge. The node weights after convergence are sorted, and a plurality of candidate words with the highest weights are selected as key fault words, for example, the number of the plurality of candidate words with the highest weights is 5% of the total number of all candidate words. The sentence sequence containing the key fault words is extracted from the uncleaned unstructured text data to obtain an implicit fault chain summary.

[0047] In the TextRank algorithm, the damping coefficient is preset to 0.85, the maximum number of iterations is preset to 100, and the convergence threshold is preset to 0.0001.

[0048] S006: Input the implicit fault chain summary into a large model, and instruct the large model to perform fault evolution analysis and output a maintenance suggestion.

[0049] Specifically, the instruction to the large model to perform fault evolution analysis and output a maintenance suggestion comprises: Constructing a large model prompt word, inputting the implicit fault chain summary as a context into the large model, instructing the large model to analyze the evolution trend of the corresponding device state, identifying a potential causal fault chain, and outputting a structured maintenance suggestion.

[0050] The embodiment of the application also discloses an unstructured text data intelligent analysis system based on a large model, comprising a processor and a memory, and the memory stores computer program instructions, which realize the unstructured text data intelligent analysis method based on a large model according to the application when the computer program instructions are executed by the processor.

[0051] The above system also comprises a communication bus and a communication interface and other components familiar to those skilled in the art, and the settings and functions thereof are known in the art, so they will not be described here.

[0052] The above are preferred embodiments of the application, and do not limit the protection scope of the application, so: any equivalent changes made on the structure, shape, principle of the application should be covered in the protection scope of the application.

Claims

1. A method for intelligent analysis of unstructured text data based on a large model, characterized in that: include: The unstructured text data to be analyzed is obtained, cleaned and segmented to obtain several candidate words, and the cleaned unstructured text data is divided into multiple time windows. Calculate the global regular interference degree for each candidate word. The global regular interference degree is positively correlated with the total frequency of occurrence of each candidate word in all time windows and negatively correlated with the standard deviation of the frequency of occurrence of each candidate word in all time windows. Based on the global general interference degree, the fault semantic specificity factor of each candidate word is calculated. The fault semantic specificity factor is positively correlated with the character length of each candidate word and negatively correlated with the global general interference degree and the distribution entropy of the neighboring nodes of each candidate word in the word co-occurrence network. The fault semantic specificity factor is normalized, and the normalization result is used as the initial weight of the corresponding candidate word in the TextRank algorithm. The TextRank algorithm is run based on the initial weights of the nodes to extract a summary of the hidden fault chains; The summary of the hidden fault chain is input into the large model, which is then instructed to perform fault evolution analysis and output maintenance suggestions.

2. The intelligent analysis method for unstructured text data based on a large model according to claim 1, characterized in that, The unstructured text data includes maintenance logs of coal mine equipment, inspection records of coal mine equipment, and shift handover records of coal mine production.

3. The intelligent analysis method for unstructured text data based on a large model according to claim 1, characterized in that, The word segmentation process includes: The cleaned unstructured text data is segmented using a word segmentation tool that loads an industry-specific thesaurus to obtain initial candidate words; Remove common stop words from the initial candidate words and retain multiple negative words to obtain the candidate words.

4. The intelligent analysis method for unstructured text data based on a large model according to claim 1, characterized in that, The global normal interference degree satisfies: ； In the formula, Candidate words Global normal interference, Candidate words Total frequency of occurrence across all time windows Candidate words The standard deviation of the frequency of occurrence across all time windows. Candidate words The mean frequency of occurrence across all time windows. As the first hyperparameter, It is a natural constant. It is the natural logarithm function.

5. The intelligent analysis method for unstructured text data based on a large model according to claim 1, characterized in that, The calculation method for the fault semantic specificity factor is as follows: ； In the formula, Candidate words The semantic specificity factor of the fault, Candidate words The length of the characters, Candidate words Global normal interference, Candidate words Distribution entropy of neighboring nodes in a word co-occurrence network This is the second hyperparameter. It is a natural constant. For the minimum normalization function, It is the natural logarithm function. It is a natural exponential function.

6. The intelligent analysis method for unstructured text data based on a large model according to claim 1, characterized in that, The initial weights satisfy: ； In the formula, Candidate words The initial weights, Candidate words The semantic specificity factor of the fault, The total number of candidate words.

7. The intelligent analysis method for unstructured text data based on a large model according to claim 1, characterized in that, The extracted hidden fault chain summary includes: A word co-occurrence graph is constructed based on the cleaned unstructured text data. Candidate words are used as nodes in the word co-occurrence graph, and the co-occurrence relationships between candidate words within a time window are used as edges. The initial weights of the nodes are used to assign values ​​to each node in the word co-occurrence graph; The weighted jump probability formula is used for iterative calculation until the node weights converge. The node weights after convergence are sorted, and the candidate words with the highest weights are selected as key fault words; Extract sentence sequences containing key fault words from uncleaned unstructured text data to obtain a summary of hidden fault chains.

8. The intelligent analysis method for unstructured text data based on a large model according to claim 7, characterized in that, The damping coefficient of the TextRank algorithm is preset to 0.85, the maximum number of iterations is preset to 100, and the convergence threshold is preset to 0.0001.

9. The intelligent analysis method for unstructured text data based on a large model according to claim 1, characterized in that, The instruction describes how the large model performs fault evolution analysis and outputs maintenance suggestions, including: The large model is constructed with prompt words, and the summary of the hidden fault chain is used as the context input to the large model. The large model is instructed to analyze the evolution trend of the corresponding equipment status, identify potential causal fault chains, and output structured maintenance suggestions.

10. An intelligent analysis system for unstructured text data based on a large model, characterized in that: include: A processor and a memory, wherein the memory stores computer program instructions that, when executed by the processor, implement the intelligent analysis method for unstructured text data based on a large model according to any one of claims 1-9.

Images