Generation method and device of power dispatching preprocessing text

Abstract

The invention discloses a power dispatching preprocessing text generation method and device, relates to the technical field of text processing, and mainly aims at solving the problem that an existing large language model is poor in preprocessing accuracy. Comprising the steps that power dispatching texts are acquired, and the power dispatching texts comprise a dispatching standard text, a dispatching procedure text and a dispatching plan text; if the keyword proportion of the text keywords in the power dispatching text is larger than a preset proportion threshold value, a first keyword set is constructed based on the text keywords, a target statement of the power dispatching text is determined, and the first keyword set comprises a plurality of first keywords; and if the statement segmented word in the target statement is not in the first keyword set, calling a second keyword of the target statement, and generating a pre-processing text of the target statement based on the first keyword and the second keyword, so as to pre-process a large language model corresponding to the power dispatching text based on the pre-processing text.

Description

Technical Field

[0001] The present invention relates to the technical field of text processing, and in particular to a method and device for generating preprocessing text for electric power dispatching. Background Art

[0002] In power dispatching, the application of Large Language Model (LLM) is an important part of the modernization and intelligence of power systems. Power dispatching focuses on the real-time operation and management of power systems, including the coordination of various links such as power generation, transmission, and distribution to ensure the stability and security of the power grid. Among them, in order to achieve the purpose of using large language models for power dispatching analysis and search, it is necessary to fine-tune the large language model in a targeted manner, that is, to adapt to different application scenarios through preprocessing and increase the adaptability of the large language model.

[0003] At present, the existing large language models usually adopt the method of screening keywords when performing preprocessing, that is, counting the frequency of a certain word, determining it as a keyword, and using this keyword as the preprocessed text for preprocessing training. However, since keywords may not only be words with high frequency, words with low frequency but semantic influence cannot be accurately identified, which in turn affects the effectiveness of the preprocessed text, thereby reducing the preprocessing accuracy and application accuracy of the large language model. Summary of the invention

[0004] In view of this, the present invention provides a method and device for generating preprocessing text for electric power dispatching, the main purpose of which is to solve the problem of poor preprocessing accuracy of existing large language models.

[0005] According to one aspect of the present invention, a method for generating a preprocessing text for power dispatching is provided, comprising:

[0006] Obtaining a power dispatch text, wherein the power dispatch text includes a dispatch standard text, a dispatch procedure text, and a dispatch plan text;

[0007] If the keyword ratio of the text keywords in the power dispatch text is greater than a preset ratio threshold, constructing a first keyword set based on the text keywords, and determining a target sentence of the power dispatch text, the first keyword set including a plurality of first keywords;

[0008] If the sentence segmentation in the target sentence is not in the first keyword set, the second keyword of the target sentence is retrieved, and a preprocessed text of the target sentence is generated based on the first keyword and the second keyword, so as to preprocess the large language model corresponding to the power dispatching text based on the preprocessed text.

[0009] Furthermore, before constructing the first keyword set based on the text keywords, the method further includes:

[0010] Obtaining text keywords of the electric power dispatching text, and counting the number of the text keywords in the electric power dispatching text;

[0011] The keyword ratio is calculated based on the number, the number of words of the keyword and the total number of words in the power dispatching text, so as to be compared with a preset ratio threshold.

[0012] Further, the retrieving the second keyword of the target sentence includes:

[0013] Generating a second word ranking of the sentence segmentation according to the first word ranking of the target sentence, wherein the first word ranking is the ranking of the sentence segmentation in the target sentence, and the second word ranking is an independent ranking of the sentence segmentation;

[0014] Deleting the sentence participles ranked first from the target sentence one by one according to the ranking of the second words, to obtain a plurality of reference sentences;

[0015] When the sentence similarity between the reference sentence and the target sentence is less than a preset similarity threshold, the sentence segment ranked first is determined as the second keyword, and the sentence similarity is determined based on the vector angle between the reference sentence and the target sentence.

[0016] Furthermore, after deleting the sentence participles ranked first from the target sentence one by one according to the second word ranking to obtain a plurality of reference sentences, the method further includes:

[0017] When the sentence similarity between the reference sentence and the target sentence is greater than or equal to a preset similarity threshold, and the difference between the sentence similarity and the preset similarity matches a preset similarity difference threshold range, the sentence segmentation ranked first is determined as the second keyword.

[0018] Furthermore, after obtaining the power dispatch text, the method further includes:

[0019] Querying target characters in the power dispatch text based on regular expressions, wherein the target characters include at least one of symbols, formulas, picture links, non-Chinese language expressions, and serial numbers;

[0020] The target characters are replaced with preset characters to obtain a reorganized power dispatch text, so as to extract text keywords based on the reorganized power dispatch text.

[0021] Furthermore, the method further comprises:

[0022] Acquire a sample set of electric power dispatching sentences, wherein the sample set of electric power dispatching sentences includes sentence samples with word segmentation tags;

[0023] Constructing a word segmentation model, and calculating reference word frequencies and co-occurrence frequencies from the power dispatch sentence sample set;

[0024] Training the word segmentation model through the power dispatch sentence sample set, the reference word frequency, and the co-occurrence frequency;

[0025] After determining the target sentence of the power dispatch text, the method further includes:

[0026] The target sentence is segmented using the word segmentation model that has completed model training to obtain sentence segmentation.

[0027] Furthermore, after generating the preprocessed text of the target sentence based on the first keyword and the second keyword, the method further includes:

[0028] Acquire a target large language model corresponding to the electric power dispatching text, and train the target large language model based on the preprocessed text;

[0029] In response to a key sentence query request, the target large language model that has completed model training is retrieved, and the key text carried in the key sentence query request is processed based on the target large language model to obtain a query sentence that matches the power dispatch text.

[0030] According to another aspect of the present invention, there is provided a device for generating a preprocessing text for power dispatching, comprising:

[0031] An acquisition module, used for acquiring a power dispatch text, wherein the power dispatch text includes a dispatch standard text, a dispatch procedure text and a dispatch plan text;

[0032] A determination module, configured to construct a first keyword set based on the text keywords and determine a target sentence of the power dispatch text if a keyword ratio of the text keywords in the power dispatch text is greater than a preset ratio threshold, wherein the first keyword set includes a plurality of first keywords;

[0033] A generation module is used to retrieve the second keyword of the target sentence if the sentence segmentation in the target sentence is not in the first keyword set, and generate a preprocessed text of the target sentence based on the first keyword and the second keyword, so as to preprocess the large language model corresponding to the power dispatching text based on the preprocessed text.

[0034] Furthermore, the device also includes:

[0035] A statistical module, used for obtaining text keywords of the electric power dispatch text, and counting the number of the text keywords in the electric power dispatch text;

[0036] A calculation module is used to calculate the keyword ratio based on the number, the number of words of the keyword and the total number of words in the power dispatching text, so as to compare it with a preset ratio threshold.

[0037] Furthermore, the generation module is specifically used to generate a second word ranking of the sentence participles according to the first word ranking of the target sentence, the first word ranking being the ranking of the sentence participles in the target sentence, and the second word ranking being an independent ranking of the sentence participles; deleting the sentence participles ranked first from the target sentence one by one according to the second word ranking to obtain multiple reference sentences; when the sentence similarity between the reference sentence and the target sentence is less than a preset similarity threshold, determining the sentence participle ranked first as the second keyword, and the sentence similarity being determined based on the vector angle between the reference sentence and the target sentence.

[0038] Furthermore, the determination module is also used to determine the sentence segment ranked first as the second keyword when the sentence similarity between the reference sentence and the target sentence is greater than or equal to a preset similarity threshold, and the difference between the sentence similarity and the preset similarity matches a preset similarity difference threshold range.

[0039] Furthermore, the method further comprises:

[0040] A query module, used for querying target characters in the power dispatch text based on regular expressions, wherein the target characters include at least one of symbols, formulas, picture links, non-Chinese language expressions, and serial numbers;

[0041] The replacement module is used to replace the target characters with preset characters to obtain a reorganized power dispatch text, so as to extract text keywords based on the reorganized power dispatch text.

[0042] Further, building modules, training modules, processing modules,

[0043] The acquisition module is further used to acquire a sample set of electric power dispatching statements, wherein the sample set of electric power dispatching statements includes statement samples with word segmentation marks;

[0044] The construction module is used to construct a word segmentation model and to calculate reference word frequencies and co-occurrence frequencies from the power dispatch sentence sample set;

[0045] The training module is used to train the word segmentation model through the power dispatch sentence sample set, the reference word frequency, and the co-occurrence frequency;

[0046] The processing module is used to perform word segmentation processing on the target sentence using the word segmentation model that has completed model training to obtain sentence segmentation.

[0047] Further,

[0048] The acquisition module is further used to acquire a target large language model corresponding to the power dispatch text, and train the target large language model based on the preprocessed text;

[0049] The processing module is also used to respond to a key statement query request, retrieve the target large language model that has completed model training, and process the key text carried in the key statement query request based on the target large language model to obtain a query statement that matches the power dispatch text.

[0050] According to another aspect of the present invention, there is provided a storage medium storing at least one executable instruction, wherein the executable instruction enables a processor to execute operations corresponding to the above-mentioned method for generating a preprocessing text for power dispatching.

[0051] According to another aspect of the present invention, there is provided a terminal, comprising: a processor, a memory, a communication interface and a communication bus, wherein the processor, the memory and the communication interface communicate with each other via the communication bus;

[0052] The memory is used to store at least one executable instruction, and the executable instruction enables the processor to execute operations corresponding to the above-mentioned method for generating the power dispatch preprocessing text.

[0053] By means of the above technical solution, the technical solution provided by the embodiment of the present invention has at least the following advantages:

[0054] The present invention provides a method and device for generating a preprocessed text for electric power dispatching. Compared with the prior art, the embodiment of the present invention obtains an electric power dispatching text, wherein the electric power dispatching text includes a dispatching standard text, a dispatching procedure text and a dispatching plan text; if the keyword ratio of text keywords in the electric power dispatching text is greater than a preset ratio threshold, a first keyword set is constructed based on the text keywords, and a target sentence of the electric power dispatching text is determined, wherein the first keyword set includes multiple first keywords; if the sentence segmentation in the target sentence is not in the first keyword set, a second keyword of the target sentence is retrieved, and a preprocessed text of the target sentence is generated based on the first keyword and the second keyword, so as to preprocess a large language model corresponding to the electric power dispatching text based on the preprocessed text, so as to achieve the purpose of accurately generating the preprocessed text, so as to use more accurate preprocessed text to process the large language model, improve the fine-tuning accuracy of the large language model, meet the flexibility scenario applicability requirements of using preprocessed text to fine-tune the large language model, and make the application of the large language model more accurate.

[0055] The above description is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the present invention, it can be implemented according to the contents of the specification. In order to make the above and other purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] Various other advantages and benefits will become apparent to those of ordinary skill in the art by reading the detailed description of the preferred embodiments below. The accompanying drawings are only for the purpose of illustrating the preferred embodiments and are not to be considered as limiting the present invention. Moreover, the same reference symbols are used throughout the accompanying drawings to represent the same components. In the accompanying drawings:

[0057] Figure 1 A flow chart of a method for generating a preprocessing text for power dispatching provided by an embodiment of the present invention is shown;

[0058] Figure 2 A block diagram showing the composition of a device for generating a preprocessing text for power dispatching provided by an embodiment of the present invention is shown;

[0059] Figure 3 A schematic diagram of the structure of a terminal provided by an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0060] The exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided to enable a more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art.

[0061] The embodiment of the present invention provides a method for generating a preprocessing text of power dispatching, such as Figure 1 As shown, the method includes:

[0062] 101. Obtain power dispatch text.

[0063] In an embodiment of the present invention, the current execution end, as the execution subject for processing the power dispatch text, can be a cloud server or a terminal device. The current execution end can receive the power dispatch query request triggered by the user to retrieve the large language model for intelligent text processing to obtain the dispatch text for reply. At this time, in order to make the large language model better applicable to the intelligent dialogue scenario of the power dispatch scenario, it is necessary to fine-tune the large language model, that is, it is necessary to use a more accurate and effective learning text as the basis for fine-tuning. Therefore, a pre-processed text is generated to process the large language model so that the large language model is more applicable to the power dispatch scenario. Among them, the power dispatch text includes a dispatch standard text, a dispatch procedure text, and a dispatch plan text. At this time, the power dispatch text can be obtained based on real-time input by the user, or it can be pre-stored in the current execution end to be obtained when the pre-processed text instruction is triggered. The embodiment of the present invention does not make specific limitations.

[0064] 102. If the keyword ratio of text keywords in the power dispatching text is greater than a preset ratio threshold, construct a first keyword set based on the text keywords, and determine a target sentence of the power dispatching text.

[0065] In the embodiment of the present invention, after the power dispatch text is obtained, the text keywords are extracted. At this time, the text keywords are keywords for all parts of the text of the same type, not keywords for each sentence. Different types of texts are dispatch standard type texts, dispatch procedure type texts, and dispatch plan type texts. Matching and extraction can be performed based on different types of keyword libraries, which are not specifically limited in the embodiment of the present invention. Furthermore, the current execution end calculates the keyword ratio of the text keywords and compares it with the preset ratio threshold. At this time, if the keyword ratio is greater than the preset ratio threshold, a first keyword set is constructed based on this text keyword. The first keyword set includes multiple first keywords, which are not specifically limited in the embodiment of the present invention.

[0066] It should be noted that the power dispatch text is composed of multiple sentences. In order to avoid ambiguity in the understanding of the semantics between the sentences and the complete text content, the current execution end obtains at least one target sentence of the power dispatch text, that is, the power dispatch text is split sentence by sentence, such as according to symbols representing complete sentences such as periods and exclamation marks, to obtain the target sentence. The embodiment of the present invention does not make specific limitations.

[0067] 103. If the sentence segmentation in the target sentence is not in the first keyword set, retrieve the second keyword of the target sentence, and generate a preprocessed text of the target sentence based on the first keyword and the second keyword.

[0068] In the embodiment of the present invention, the sentence segmentation is obtained by segmenting the target sentence, and it is queried whether each sentence segmentation exists in the first keyword set. If not, it means that this sentence segmentation does not belong to the text keyword. At this time, the second keyword as the sentence is extracted from the target sentence, and the first keyword as the keyword of the complete text and the second keyword as a single sentence are combined to generate and process the text, so as to pre-process the large language model corresponding to the power dispatching text based on the pre-processed text. Among them, the large language model (Large Language Models, LLMs for short) is a type of deep learning model with a large number of parameters. In the field of natural language processing, it learns language patterns, grammar and semantics by processing a large amount of text data to generate and understand human language. The large language template includes but is not limited to GPT-3, GPT-4, LLaMA, etc., and the embodiment of the present invention does not make specific limitations.

[0069] In another embodiment of the present invention, for further definition and explanation, before the step of constructing the first keyword set based on the text keywords, the method further includes:

[0070] Obtaining text keywords of the electric power dispatching text, and counting the number of the text keywords in the electric power dispatching text;

[0071] The keyword ratio is calculated based on the number, the number of words of the keyword and the total number of words in the power dispatching text, so as to be compared with a preset ratio threshold.

[0072] In order to accurately filter out the keywords in the power dispatch text, so as to use this keyword as the basis for generating the preprocessed text and improve the effectiveness of generating the preprocessed text, the current execution end first obtains the text keywords of the power dispatch text. Specifically, the power dispatch text can be queried based on the words written in the pre-created keyword library. If there are words in the keyword thesaurus in the power dispatch text, they can be used as text keywords, and the number of text keywords in the power dispatch text is counted by full-text search. At the same time, the total number of words in the power dispatch text is counted, and the word count ratio is calculated, B = (D*n) / Dove, Dove represents the total number of words in the target text, D is the number of keywords, and n is the number of text keywords counted in the power dispatch text, so as to compare with the preset ratio threshold, which is not specifically limited in the embodiment of the present invention. Among them, the preset ratio threshold can be configured based on the total word data of the power dispatch text, which is not specifically limited in the embodiment of the present invention.

[0073] It should be noted that, since the power dispatch text may include dispatch standard text, dispatch procedure text and dispatch plan text, when using the keyword library to obtain text keywords, specifically, in order to avoid the extraction of words with different meanings of similar texts, when using the keyword library to obtain text keywords, specifically, first in the first round of query, the keywords existing in the keyword library are queried from the power dispatch text as text keywords. At this time, you can directly compare them one by one according to Chinese words, or use the word vector conversion to compare vector values, and use the same words as the query basis. If no query is found, you can enter the second round of query. Specifically, first convert the keywords in the keyword library into word vectors, which can be implemented based on methods such as Word2Vec and convolutional neural network (CNN). After obtaining each word vector, calculate the Euclidean distance with each word vector in the power dispatch text, obtain the semantic similarity between each word in the power dispatch text and each word in the keyword library, and calculate the similarity matching range of each word vector in the power dispatch text one by one. Specifically, S = (N) k, where N is the vector value of the word vector, k is the number of characters in the word, if the word consists of 2 characters, then k is 2, and the calculation of N can be implemented based on the calculation method of the vector modulus, which is not specifically limited in the embodiment of the present invention. After the similarity matching range is obtained, the similarity of this word in each keyword library is adjusted. When the similarity matching range is greater than 0, the similarity calculated in each keyword library is added and adjusted with the similarity matching range. When the similarity matching range is less than or equal to 0, the similarity calculated in each keyword library is subtracted from the similarity matching range, so as to reuse the preset similarity threshold based on the adjusted similarity to determine whether the word in the power dispatch text can be extracted as a word with similar semantics to the word in the keyword library as a text keyword. Among them, the preset similarity threshold can be configured based on the query requirements. If the similarity between the word vector in the power dispatch text and the word vector in the keyword library does not match the preset similarity threshold, an empty string is directly returned to indicate that there is no text keyword in the power dispatch text, which is not specifically limited in the embodiment of the present invention.

[0074] In another embodiment of the present invention, for further definition and explanation, the step of retrieving the second keyword of the target sentence includes:

[0075] Generating a second word ranking of the sentence segmentation according to the first word ranking of the target sentence;

[0076] Deleting the sentence participles ranked first from the target sentence one by one according to the ranking of the second words, to obtain a plurality of reference sentences;

[0077] When the sentence similarity between the reference sentence and the target sentence is less than a preset similarity threshold, the sentence segmentation ranked first is determined as the second keyword.

[0078] In order to accurately extract keywords from the power dispatching text, when the current execution end retrieves the second keyword of the target sentence, the current execution end first generates the second word ranking of the sentence segmentation in the target sentence according to the first word ranking of the target sentence. Among them, the first word ranking is the ranking of the sentence segmentation in the target sentence, and the second word ranking is the independent ranking of the sentence segmentation. For a sentence segmentation, the first word ranking in a complete text and the second word ranking in the reference sentence are usually different. If the target sentence is the first sentence in the complete text, the first word ranking and the second word ranking can be the same, and the embodiment of the present invention does not make specific restrictions. Further, according to the second word ranking, the first-ranked sentence segmentation is deleted one by one from the target sentence to obtain multiple reference sentences. At this time, for each sentence segmentation ranked first, it is deleted separately to obtain a corresponding reference sentence. Correspondingly, the sentence similarity between the reference sentence corresponding to the sentence segmentation of the adjacent ranking and the target sentence is calculated, and compared with the preset similarity threshold. If the sentence similarity between the reference sentence and the target sentence is less than the preset similarity threshold, the sentence segmentation ranked first is determined as the second keyword. For example, the target sentence is "allocate users A and B according to the maximum monthly power X", and the sentence segmentations after splitting are "according to", "monthly", "maximum", "power", "X", "to", "user A and B", "conduct", and "allocate". "According to", "X", "to", "user A and B", and "conduct" do not exist in the first keyword set. The sentence words of "according to", "X", "to", and "user A and B" are sorted to obtain the first word sorting, such as according to is ranked first, "X" is ranked second, and so on. At this time, the sentence word "according to" is deleted, the remaining sentence segmentations are sorted, and the ranking of "X" is updated to the first, and reference sentence 1 is obtained. Correspondingly, after deleting "X", the remaining sentence segmentations are sorted, and "to" is ranked first to obtain reference sentence 2, and all reference sentences are obtained one by one. Then, the sentence similarity between reference sentence 1 and the target sentence, the sentence similarity between reference sentence 2 and the target sentence, and so on are calculated respectively to obtain all sentence similarities, and then, each sentence similarity is compared with the preset similarity threshold. When the sentence similarity is less than a preset similarity threshold, the sentence segmentation word ranked first is the deleted sentence word, and the sentence word is determined as the second keyword.

[0079] It should be noted that the sentence similarity is determined based on the vector angle between the reference sentence and the target sentence. Specifically, the reference sentence and the target sentence can be respectively input into the trained BERT (Bidirectional Encoder Representations from Transformers) model to generate a reference sentence embedding vector F and a target sentence embedding vector F', respectively. The cosine value of the angle between the reference sentence embedding vector F and the target sentence embedding vector F' is calculated by the angle cosine formula, and is used as the similarity between the reference sentence and the target sentence. The embodiment of the present invention does not make any specific limitation.

[0080] In the embodiment of the present invention, some words that are not included in the first keyword set, when used as common words, although they appear less frequently in the entire power dispatching text, may have a significant impact on the meaning of the sentence in a specific sentence. Therefore, the embodiment of the present invention gradually removes common words (i.e., words in each sentence) and calculates sentence similarity. For example, the negative word "not" may not appear frequently in the entire text, but in a certain sentence it may determine the meaning of the entire sentence. By gradually removing common words and calculating similarity, those common words that have a significant impact on the semantics of the sentence can be identified to ensure that the original meaning of the sentence will not be destroyed, so as to improve the recognition accuracy of the sentence.

[0081] In a specific scenario, for example, the target sentence is "In an emergency, the backup power supply should not be turned off", and "not" which is not in the first keyword set may be an ordinary word in the target text, but in "In an emergency, the backup power supply should not be turned off", if "not" is directly removed, the sentence becomes "In an emergency, the backup power supply should be turned off", resulting in the completely opposite meaning of the sentence, which in turn leads to errors when using the keywords in "In an emergency, the backup power supply should be turned off" to fine-tune the large language model. Therefore, in an embodiment of the present invention, the first keyword "turn off" and the second keyword "not" in the target sentence are used as fine-tuning keywords for the target sentence, and the standard sentence and the target sentence are composed to jointly fine-tune the large language model.

[0082] In another embodiment of the present invention, for further limitation and explanation, after the step of deleting the first-ranked sentence segmentation from the target sentence one by one according to the second word ranking and obtaining a plurality of reference sentences, the method further includes:

[0083] When the sentence similarity between the reference sentence and the target sentence is greater than or equal to a preset similarity threshold, and the difference between the sentence similarity and the preset similarity matches a preset similarity difference threshold range, the sentence segmentation ranked first is determined as the second keyword.

[0084] In order to ensure the validity as the second keyword, after the current execution end obtains the sentence similarity, if it is greater than or equal to the preset similarity threshold, it is necessary to simultaneously calculate the difference between the preset similarity threshold and the preset similarity threshold, and further compare it with the preset similarity difference threshold range. At this time, the preset similarity threshold is preferably 0.8, and the difference between the sentence similarity and the preset similarity matches the preset similarity difference threshold range, which can be expressed as |P-0.8|<P', where P' is the preset similarity difference threshold, and P is the sentence similarity. The embodiment of the present invention does not make specific restrictions. At this time, when the sentence similarity between the reference sentence and the target sentence is greater than or equal to the preset similarity threshold, and the difference between the sentence similarity and the preset similarity matches the preset similarity difference threshold range, it indicates that the sentence participle ranked first is of high importance as an influence on the semantics of the target sentence, and therefore, the sentence participle ranked first is determined as the second keyword.

[0085] In another embodiment of the present invention, for further definition and explanation, after the step of obtaining the power dispatch text, the method further includes:

[0086] Querying target characters in the power dispatch text based on regular expressions;

[0087] The target characters are replaced with preset characters to obtain a reorganized power dispatch text, so as to extract text keywords based on the reorganized power dispatch text.

[0088] In order to ensure the validity of text keywords and avoid extracting invalid words, the current execution end organizes the power dispatch text after obtaining the power dispatch text. First, the target characters in the power dispatch text can be queried based on regular expressions. Among them, the target characters include at least one of symbols, formulas, picture links, non-Chinese language expressions, and serial numbers. At this time, regular expressions are used to describe and match a series of strings that conform to a certain syntactic rule. In the embodiment of the present invention, the logical relationship between ordinary characters (such as letters between a and z) and special characters (called "meta characters") is pre-configured to retrieve and replace text that conforms to the query rules. At this time, the query rules include target characters, such as RegularExpression, referred to as regex or regexp. The embodiment of the present invention does not specifically limit the specific content of the regular expression. When the target character is queried, the target character is replaced with the preset character to obtain the reorganized power dispatch text, so as to extract the text keywords based on the reorganized power dispatch text. At this time, the preset character can be a specific number or specific text that does not affect the semantics of the text, which is not specifically limited in the embodiment of the present invention.

[0089] In another embodiment of the present invention, for further definition and description, the steps further include:

[0090] Obtain a sample set of power dispatch statements;

[0091] Constructing a word segmentation model, and calculating reference word frequencies and co-occurrence frequencies from the power dispatch sentence sample set;

[0092] The word segmentation model is trained using the power dispatch sentence sample set, the reference word frequency, and the co-occurrence frequency.

[0093] In order to meet the demand for word segmentation of power dispatching statements, the current execution end first obtains a sample set of power dispatching statements. Among them, the sample set of power dispatching statements includes sentence samples with word segmentation tags. At this time, the word segmentation tags may include but are not limited to noun tags, verb tags, adjective tags, etc., so as to mark and classify according to the word class of the words, which is not specifically limited in the embodiment of the present invention. At the same time, a word segmentation model is constructed. At this time, the word segmentation model can be a hidden Markov model, and then the reference word frequency and co-occurrence frequency are counted from the sample set of power dispatching statements. Among them, the hidden Markov model needs to identify the boundaries and structures of words. Therefore, the reference word frequency and co-occurrence frequency are counted as training parameters. The hidden Markov model can better capture the relationship between words and improve the accuracy of word segmentation. At the same time, through the co-occurrence frequency within the preset distance window, more complex word combinations can be identified to avoid mis-segmentation. The specific size of the distance window can be configured according to the model training requirements, which is not specifically limited in the embodiment of the present invention.

[0094] It should be noted that the Hidden Markov Model (HMM) is a statistical model used to describe a random process with Markov properties. In the Hidden Markov Model HMM, the system is described as a sequence of hidden states. The transitions between these states have Markov properties, that is, each state depends only on the previous state, and each state generates an observation. The power dispatch text is a typical sequence data. The order of words and phrases in a sentence is very important, and the Hidden Markov Model HMM can model this sequence relationship well. Therefore, in the training process of the Hidden Markov Model, random values ​​or prior knowledge are used for initialization, and the word segmentation model is trained through the sample set of power dispatch sentences and reference word frequency and co-occurrence frequency.

[0095] Specifically, after determining the target sentence of the electric power dispatching text, the method further includes: performing word segmentation processing on the target sentence using the word segmentation model that has completed model training to obtain sentence segmentation, and then using sentence segmentation to determine keywords.

[0096] In the embodiment of the present invention, for the sentence samples with word segmentation marks included in the power dispatch sentence sample set, a number of sentence samples can be randomly selected and manually word segmented. When there is a sentence sample whose manually obtained word segmentation result is different from the word segmentation result of the optimized hidden Markov model, the sentence sample and the manually obtained word segmentation result are input into the optimized hidden Markov model together, and it is trained and optimized again.

[0097] It should be noted that the results of manual word segmentation can be used as a reference standard for model optimization. By comparing the results of manual word segmentation with those of the model, the deficiencies of the model's word segmentation can be discovered, thereby further improving the model's word segmentation accuracy. The word segmentation results are the words after word segmentation. By continuously introducing manual word segmentation results with scheduling professional terms for optimization training, the model can better recognize and understand scheduling professional terms, thereby improving its adaptability and generalization capabilities.

[0098] In another embodiment of the present invention, for further definition and explanation, after the step of generating the preprocessed text of the target sentence based on the first keyword and the second keyword, the method further includes:

[0099] Acquire a target large language model corresponding to the electric power dispatching text, and train the target large language model based on the preprocessed text;

[0100] In response to a key sentence query request, the target large language model that has completed model training is retrieved, and the key text carried in the key sentence query request is processed based on the target large language model to obtain a query sentence that matches the power dispatch text.

[0101] In order to meet the flexible adaptation requirements of the pre-processed large language model, the current execution end first obtains the target large language model corresponding to the power dispatch text. At this time, since the power dispatch text includes the dispatch standard text, the dispatch procedure text and the dispatch plan text, the corresponding large language models suitable for the dispatch standard text, the dispatch procedure text and the dispatch plan text can be pre-configured, and then the target large language model is trained based on the pre-processed text. After the training is completed, Yonghui can request the current execution end to query the power dispatch strategy based on the query instruction, that is, after receiving the key statement query request, the current execution end calls the target large language model that has completed the model training, so as to process the key text carried in the key statement query request based on the target large language model, and obtain the query statement that matches the power dispatch text. At this time, the key words carried in the key statement query request are input into the target large language model. The target large language model after pre-processing training can generate query statements more accurately so that the data can be given to the user.

[0102] In another embodiment of the present invention, for further limitation and explanation, after retrieving the second keyword of the target sentence and before generating the preprocessed text of the target sentence based on the first keyword and the second keyword, the method also includes: constructing a second keyword set including the second keyword; extracting from the second keyword set according to the text processing conditions of the power dispatching text to obtain the extracted set reference words and set deposition words; calculating the word association between the set reference words and the set deposition words, and filtering out the second keyword to generate the preprocessed text based on the sorting result of the word association.

[0103] In order to improve the accuracy of second keyword screening, and thus improve the effectiveness of preprocessing text generation, the current execution end verifies the second keyword after retrieving the second keyword and before generating the preprocessing text. Specifically, a second keyword set containing all second keywords is first constructed. Then, according to the text processing conditions of the power dispatching text, a preset number of set reference words (the preset number is configured based on the verification requirements and is less than or equal to the number of second keywords) are extracted from the second keyword set. At this time, the text processing conditions include at least one of nouns, verbs, adverbs, and place adverbials as conditions for extraction to obtain combined reference words, and the remaining words in the set that have not been extracted are used as set sedimentation words. Among them, when calculating the word association between the set reference words and the set sedimentation words, specifically, the set reference words and the set sedimentation words are respectively vectorized, and the element values ​​in the obtained vector matrix are respectively mapped to a two-dimensional space, so as to calculate the first data mean by taking all the element data points of a specific set reference word as the object, and calculate the second data mean by taking all the element data points of a corresponding set sedimentation word as the object. At this time, compare whether the difference between the first data mean and the second data mean is greater than the preset data difference (which can be configured based on the relevance requirement). If it is greater than, it means that there is no word association between the set sedimentation word and the combined reference word. If it is less than, the difference is used as the word association to sort them, and the second keyword of the preprocessed text to be generated is screened out based on the sorting result of the word association (such as the first 5 words). The embodiment of the present invention does not make specific limitations.

[0104] An embodiment of the present invention provides a method for generating a preprocessed text for electric power dispatching. Compared with the prior art, the embodiment of the present invention obtains an electric power dispatching text, wherein the electric power dispatching text includes a dispatching standard text, a dispatching procedure text, and a dispatching plan text; if the keyword ratio of text keywords in the electric power dispatching text is greater than a preset ratio threshold, a first keyword set is constructed based on the text keywords, and a target sentence of the electric power dispatching text is determined, wherein the first keyword set includes multiple first keywords; if the sentence segmentation in the target sentence is not in the first keyword set, a second keyword of the target sentence is retrieved, and a preprocessed text of the target sentence is generated based on the first keyword and the second keyword, so as to preprocess a large language model corresponding to the electric power dispatching text based on the preprocessed text, so as to achieve the purpose of accurately generating the preprocessed text, so as to use more accurate preprocessed text to process the large language model, improve the fine-tuning accuracy of the large language model, meet the flexibility scenario applicability requirements of fine-tuning the large language model using preprocessed text, and make the application of the large language model more accurate.

[0105] Furthermore, as a response to the above Figure 1 The embodiment of the present invention provides a device for generating a preprocessing text for power dispatching, such as Figure 2 As shown, the device comprises:

[0106] An acquisition module 21 is used to acquire a power dispatch text, wherein the power dispatch text includes a dispatch standard text, a dispatch procedure text, and a dispatch plan text;

[0107] A determination module 22, configured to construct a first keyword set based on the text keywords and determine a target sentence of the power dispatch text if the keyword ratio of the text keywords in the power dispatch text is greater than a preset ratio threshold, wherein the first keyword set includes a plurality of first keywords;

[0108] A generation module 23 is used to retrieve the second keyword of the target sentence if the sentence segmentation in the target sentence is not in the first keyword set, and generate a preprocessed text of the target sentence based on the first keyword and the second keyword, so as to preprocess the large language model corresponding to the power dispatching text based on the preprocessed text.

[0109] Furthermore, the device also includes:

[0110] A statistical module, used for obtaining text keywords of the electric power dispatch text, and counting the number of the text keywords in the electric power dispatch text;

[0111] A calculation module is used to calculate the keyword ratio based on the number, the number of words of the keyword and the total number of words in the power dispatching text, so as to compare it with a preset ratio threshold.

[0112] Furthermore, the generation module is specifically used to generate a second word ranking of the sentence participles according to the first word ranking of the target sentence, the first word ranking being the ranking of the sentence participles in the target sentence, and the second word ranking being an independent ranking of the sentence participles; deleting the sentence participles ranked first from the target sentence one by one according to the second word ranking to obtain multiple reference sentences; when the sentence similarity between the reference sentence and the target sentence is less than a preset similarity threshold, determining the sentence participle ranked first as the second keyword, and the sentence similarity being determined based on the vector angle between the reference sentence and the target sentence.

[0113] Furthermore, the determination module is also used to determine the sentence segment ranked first as the second keyword when the sentence similarity between the reference sentence and the target sentence is greater than or equal to a preset similarity threshold, and the difference between the sentence similarity and the preset similarity matches a preset similarity difference threshold range.

[0114] Furthermore, the method further comprises:

[0115] A query module, used for querying target characters in the power dispatch text based on regular expressions, wherein the target characters include at least one of symbols, formulas, picture links, non-Chinese language expressions, and serial numbers;

[0116] The replacement module is used to replace the target characters with preset characters to obtain a reorganized power dispatch text, so as to extract text keywords based on the reorganized power dispatch text.

[0117] Further, building modules, training modules, processing modules,

[0118] The acquisition module is further used to acquire a sample set of electric power dispatching statements, wherein the sample set of electric power dispatching statements includes statement samples with word segmentation marks;

[0119] The construction module is used to construct a word segmentation model and to calculate reference word frequencies and co-occurrence frequencies from the power dispatch sentence sample set;

[0120] The training module is used to train the word segmentation model through the power dispatch sentence sample set, the reference word frequency, and the co-occurrence frequency;

[0121] The processing module is used to perform word segmentation processing on the target sentence using the word segmentation model that has completed model training to obtain sentence segmentation.

[0122] Further,

[0123] The acquisition module is further used to acquire a target large language model corresponding to the power dispatch text, and train the target large language model based on the preprocessed text;

[0124] The processing module is also used to respond to a key statement query request, retrieve the target large language model that has completed model training, and process the key text carried in the key statement query request based on the target large language model to obtain a query statement that matches the power dispatch text.

[0125] An embodiment of the present invention provides a device for generating a preprocessed text for electric power dispatching. Compared with the prior art, the embodiment of the present invention obtains an electric power dispatching text, wherein the electric power dispatching text includes a dispatching standard text, a dispatching procedure text, and a dispatching plan text; if the keyword ratio of text keywords in the electric power dispatching text is greater than a preset ratio threshold, a first keyword set is constructed based on the text keywords, and a target sentence of the electric power dispatching text is determined, wherein the first keyword set includes multiple first keywords; if the sentence segmentation in the target sentence is not in the first keyword set, a second keyword of the target sentence is retrieved, and a preprocessed text of the target sentence is generated based on the first keyword and the second keyword, so as to preprocess a large language model corresponding to the electric power dispatching text based on the preprocessed text, so as to achieve the purpose of accurately generating the preprocessed text, so as to use more accurate preprocessed text to process the large language model, improve the fine-tuning accuracy of the large language model, meet the flexibility scenario applicability requirements of fine-tuning the large language model using preprocessed text, and make the application of the large language model more accurate.

[0126] According to one embodiment of the present invention, a storage medium is provided, wherein the storage medium stores at least one executable instruction, and the computer executable instruction can execute the method for generating a preprocessing text for power dispatching in any of the above method embodiments.

[0127] Figure 3 A schematic diagram of the structure of a terminal provided according to an embodiment of the present invention is shown. The specific embodiment of the present invention does not limit the specific implementation of the terminal.

[0128] like Figure 3 As shown, the terminal may include: a processor (processor) 302 , a communication interface (Communications Interface) 304 , a memory (memory) 306 , and a communication bus 308 .

[0129] The processor 302 , the communication interface 304 , and the memory 306 communicate with each other via the communication bus 308 .

[0130] The communication interface 304 is used to communicate with other devices such as clients or other servers.

[0131] The processor 302 is used to execute the program 310, and specifically can execute the relevant steps in the above-mentioned embodiment of the method for generating the preprocessing text of the power dispatching.

[0132] Specifically, the program 310 may include program codes, which include computer operation instructions.

[0133] The processor 302 may be a central processing unit (CPU), or an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiment of the present invention. The one or more processors included in the terminal may be processors of the same type, such as one or more CPUs; or processors of different types, such as one or more CPUs and one or more ASICs.

[0134] The memory 306 is used to store the program 310. The memory 306 may include a high-speed RAM memory, and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory.

[0135] The program 310 may be specifically configured to enable the processor 302 to perform the following operations:

[0136] Obtaining a power dispatch text, wherein the power dispatch text includes a dispatch standard text, a dispatch procedure text, and a dispatch plan text;

[0137] If the keyword ratio of the text keywords in the power dispatch text is greater than a preset ratio threshold, constructing a first keyword set based on the text keywords, and determining a target sentence of the power dispatch text, the first keyword set including a plurality of first keywords;

[0138] If the sentence segmentation in the target sentence is not in the first keyword set, the second keyword of the target sentence is retrieved, and a preprocessed text of the target sentence is generated based on the first keyword and the second keyword, so as to preprocess the large language model corresponding to the power dispatching text based on the preprocessed text.

[0139] Obviously, those skilled in the art should understand that the above modules or steps of the present invention can be implemented by a general computing device, they can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices, and optionally, they can be implemented by a program code executable by a computing device, so that they can be stored in a storage device and executed by the computing device, and in some cases, the steps shown or described can be executed in a different order than here, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. Thus, the present invention is not limited to any specific combination of hardware and software.

[0140] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for generating a preprocessing text for power dispatching, characterized in that: include: Obtaining a power dispatch text, wherein the power dispatch text includes a dispatch standard text, a dispatch procedure text, and a dispatch plan text; If the keyword ratio of the text keywords in the power dispatch text is greater than a preset ratio threshold, constructing a first keyword set based on the text keywords, and determining a target sentence of the power dispatch text, the first keyword set including a plurality of first keywords; If the sentence segmentation in the target sentence is not in the first keyword set, the second keyword of the target sentence is retrieved, and a preprocessed text of the target sentence is generated based on the first keyword and the second keyword, so as to preprocess the large language model corresponding to the power dispatching text based on the preprocessed text.

2. The method according to claim 1, characterized in that: Before constructing the first keyword set based on the text keywords, the method further includes: Obtaining text keywords of the electric power dispatching text, and counting the number of the text keywords in the electric power dispatching text; The keyword ratio is calculated based on the number, the number of words of the keyword and the total number of words in the power dispatching text, so as to be compared with a preset ratio threshold.

3. The method according to claim 2, characterized in that: The second keyword of the target sentence includes: Generating a second word ranking of the sentence segmentation according to the first word ranking of the target sentence, wherein the first word ranking is the ranking of the sentence segmentation in the target sentence, and the second word ranking is an independent ranking of the sentence segmentation; Deleting the sentence participles ranked first from the target sentence one by one according to the ranking of the second words, to obtain a plurality of reference sentences; When the sentence similarity between the reference sentence and the target sentence is less than a preset similarity threshold, the sentence segment ranked first is determined as the second keyword, and the sentence similarity is determined based on the vector angle between the reference sentence and the target sentence.

4. The method according to claim 3, characterized in that: After deleting the sentence segmentations ranked first from the target sentence one by one according to the second word ranking to obtain a plurality of reference sentences, the method further includes: When the sentence similarity between the reference sentence and the target sentence is greater than or equal to a preset similarity threshold, and the difference between the sentence similarity and the preset similarity matches a preset similarity difference threshold range, the sentence segmentation ranked first is determined as the second keyword.

5. The method according to claim 1, characterized in that: After obtaining the power dispatch text, the method further includes: Querying target characters in the power dispatch text based on regular expressions, wherein the target characters include at least one of symbols, formulas, picture links, non-Chinese language expressions, and serial numbers; The target characters are replaced with preset characters to obtain a reorganized power dispatch text, so as to extract text keywords based on the reorganized power dispatch text.

6. The method according to claim 1, characterized in that: The method further comprises: Acquire a sample set of electric power dispatching sentences, wherein the sample set of electric power dispatching sentences includes sentence samples with word segmentation tags; Constructing a word segmentation model, and calculating reference word frequencies and co-occurrence frequencies from the power dispatch sentence sample set; Training the word segmentation model through the power dispatch sentence sample set, the reference word frequency, and the co-occurrence frequency; After determining the target sentence of the power dispatch text, the method further includes: The target sentence is segmented using the word segmentation model that has completed model training to obtain sentence segmentation.

7. The method according to claim 1, characterized in that: After generating the preprocessed text of the target sentence based on the first keyword and the second keyword, the method further includes: Acquire a target large language model corresponding to the electric power dispatching text, and train the target large language model based on the preprocessed text; In response to a key sentence query request, the target large language model that has completed model training is retrieved, and the key text carried in the key sentence query request is processed based on the target large language model to obtain a query sentence that matches the power dispatch text.

8. A device for generating a preprocessing text for power dispatching, characterized in that: include: An acquisition module, used to acquire a power dispatch text, wherein the power dispatch text includes a dispatch standard text, a dispatch procedure text, and a dispatch plan text; A determination module, configured to construct a first keyword set based on the text keywords and determine a target sentence of the power dispatch text if a keyword ratio of the text keywords in the power dispatch text is greater than a preset ratio threshold, wherein the first keyword set includes a plurality of first keywords; A generation module is used to retrieve the second keyword of the target sentence if the sentence segmentation in the target sentence is not in the first keyword set, and generate a preprocessed text of the target sentence based on the first keyword and the second keyword, so as to preprocess the large language model corresponding to the power dispatching text based on the preprocessed text.

9. A storage medium storing at least one executable instruction, wherein the executable instruction enables a processor to execute operations corresponding to the method for generating a preprocessing text for power dispatching as described in any one of claims 1 to 7.

10. A terminal, comprising: A processor, a memory, a communication interface and a communication bus, wherein the processor, the memory and the communication interface communicate with each other via the communication bus; The memory is used to store at least one executable instruction, and the executable instruction enables the processor to perform operations corresponding to the method for generating a preprocessing text for power dispatching as described in any one of claims 1-7.

Images