Keyword set determination method and device, storage medium and electronic equipment

Abstract

The embodiment of the invention provides a keyword set determination method and device, a storage medium and electronic equipment, and the method comprises the steps: obtaining a seed keyword set corresponding to target data, and dividing the target data into a title and content; keyword extraction is carried out on the title and the content to generate a candidate keyword set corresponding to the target data, and the candidate keyword set comprises a first keyword set corresponding to the title and a second keyword set corresponding to the content; and determining a target keyword set corresponding to the target data according to the seed keyword set and the candidate keyword set. According to the method, the problem that target data is difficult to find in a large amount of file data due to the fact that the large amount of file data is not finely divided in the related technology can be solved.

Description

Technical Field

[0001] The present application relates to the field of data processing, and in particular, to a method and device for determining a keyword set, a storage medium, and an electronic device. Background Art

[0002] As an important energy facility, the operation and maintenance of hydropower stations involve a large amount of technical data, including design drawings, equipment manuals, operation records, maintenance reports, etc. These technical data are crucial to the safe and stable operation of hydropower stations. However, technical data accumulates over time during the production and operation of power plants, forming a large number of files. It is difficult to find the desired technical documents in a large number of files, and there is no fine division of document content, which also makes the use of them inefficient.

[0003] In the related art, a large amount of file data is not finely divided, which makes it difficult to find the target data in a large amount of file data. No effective solution has been proposed yet.

[0004] Therefore, it is necessary to improve the related technology to overcome the above-mentioned defects in the related technology. Summary of the invention

[0005] The embodiments of the present application provide a method and device for determining a keyword set, a storage medium, and an electronic device to at least solve the problem in the related art that a large amount of file data is not finely divided, making it difficult to find target data in a large amount of file data.

[0006] According to one embodiment of the present application, a method for determining a keyword set is provided, comprising: obtaining a seed keyword set corresponding to a target material, and dividing the target material into a title and content; performing keyword extraction on the title and the content respectively to generate a candidate keyword set corresponding to the target material, wherein the candidate keyword set comprises: a first keyword set corresponding to the title and a second keyword set corresponding to the content; and determining a target keyword set corresponding to the target material based on the seed keyword set and the candidate keyword set.

[0007] In an exemplary embodiment, dividing the target data into a title and content includes: determining a data format corresponding to the target data, wherein the data format includes: a text format, a picture format, and a video format; determining a target parsing method for the target data according to the data format; and dividing the target data into the title and the content according to the target parsing method.

[0008] In an exemplary embodiment, determining a target parsing method for the target data according to the data format includes: when the data format is determined to be the text format, determining the target parsing method to be a first parsing method, wherein the first parsing method is a text parsing method; when the data format is determined to be the image format, determining the target parsing method to be a second parsing method, wherein the second parsing method is a method of parsing the target data through a first recognition model; when the data format is determined to be the video format, determining the target parsing method to be a third parsing method, wherein the third parsing method is a method of extracting video frames for the target data, inputting each frame of video into a second recognition model, and parsing the target data according to an output result corresponding to the second recognition model and a text accumulation algorithm.

[0009] In an exemplary embodiment, keyword extraction is performed on the title and the content respectively to generate a candidate keyword set corresponding to the target material, including: performing word frequency statistics on the content, and extracting a third keyword set from the content according to the statistical results; inputting the third keyword set into a keyword extractor to generate the second keyword set corresponding to the content; and inputting the title into the keyword extractor to generate the first keyword set corresponding to the title.

[0010] In an exemplary embodiment, determining a target keyword set corresponding to the target data based on the seed keyword set and the candidate keyword set includes: respectively determining a target similarity between each candidate keyword in the candidate keyword set and each seed keyword in the seed keyword set; determining an average value corresponding to a plurality of target similarities, and determining the average value as an average similarity between the candidate keywords and the seed keywords; and determining a maximum similarity among the plurality of target similarities; and determining the target keyword set based on the average similarity and the maximum similarity.

[0011] In an exemplary embodiment, determining the target keyword set according to the average similarity and the maximum similarity includes: assigning scores to candidate keywords corresponding to each target similarity according to the average similarity and the maximum similarity; and determining the target keyword set in the candidate keyword set according to the score of each candidate keyword.

[0012] In an exemplary embodiment, the target similarity of each candidate keyword in the candidate keyword set and each seed keyword in the seed keyword set is determined respectively, including: converting each candidate keyword into each first vector according to a word embedding model, and converting each seed keyword into each second vector according to the word embedding model; determining the cosine similarity of each first vector and each second vector, and determining the cosine similarity as the target similarity.

[0013] According to another embodiment of the present application, a device for determining a keyword set is provided, comprising: an acquisition module, used to acquire a seed keyword set corresponding to a target material, and divide the target material into a title and content; an extraction module, used to perform keyword extraction on the title and the content, respectively, to generate a candidate keyword set corresponding to the target material, wherein the candidate keyword set includes: a first keyword set corresponding to the title and a second keyword set corresponding to the content; a determination module, used to determine a target keyword set corresponding to the target material based on the seed keyword set and the candidate keyword set.

[0014] According to another embodiment of the present application, a computer-readable storage medium is provided, in which a computer program is stored, wherein the computer program is configured to execute the steps of any of the above method embodiments when run.

[0015] According to another embodiment of the present application, an electronic device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to execute the steps in any one of the above method embodiments.

[0016] According to another embodiment of the present application, a computer program product is provided, including a computer program, and when the computer program is executed by a processor, the steps in any of the above method embodiments are implemented.

[0017] Through the embodiment of the present application, a seed keyword set corresponding to the target material is obtained, and the target material is divided into a title and content; keyword extraction is performed on the title and content respectively to generate a first keyword set corresponding to the title and a second keyword set corresponding to the content, and the first keyword set and the second keyword set are collectively referred to as a candidate keyword set. The target keyword set corresponding to the target material is determined according to the seed keyword set and the candidate keyword set. In other words, the embodiment of the present application divides the target material into a title and content, and then generates a first keyword set corresponding to the title and a second keyword set corresponding to the content. The target keyword set corresponding to the target material can be determined according to the obtained seed keyword set, the first keyword set and the second keyword set. Through the embodiment of the present application, the problem in the related art that a large amount of file materials are not finely divided, resulting in difficulty in finding the target material in a large amount of file materials can be solved, and then the target material can be quickly found in a large amount of file materials according to the target keyword set, thereby improving the search efficiency of the target material. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0019] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0020] Figure 1 It is a hardware structure block diagram of a computer terminal device of a method for determining a keyword set in an embodiment of the present application;

[0021] Figure 2 is a flowchart of a method for determining a keyword set according to an embodiment of the present application;

[0022] Figure 3 It is a schematic diagram of data title and text extraction according to an optional embodiment of the present application.

[0023] Figure 4 is a flowchart of extracting a keyword set according to an optional embodiment of the present application;

[0024] Figure 5 It is a structural block diagram of a device for determining a keyword set according to an embodiment of the present application. DETAILED DESCRIPTION

[0025] The embodiments of the present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0026] It should be noted that the terms "first", "second", etc. in the specification and claims of this application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0027] The method embodiments provided in the embodiments of the present application can be executed in a computer terminal device or a similar computing device. Taking running on a computer terminal device as an example, Figure 1 1 is a hardware structure block diagram of a computer terminal device for determining a keyword set in an embodiment of the present application. Figure 1 As shown, the computer terminal device may include one or more ( Figure 1 Only one is shown in the figure) a processor 102 (the processor 102 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA) and a memory 104 for storing data, wherein the above-mentioned computer terminal device may also include a transmission device 106 and an input and output device 108 for communication functions. It can be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the above-mentioned computer terminal device. Figure 1 More or fewer components as shown, or with Figure 1 Different configurations shown.

[0028] The memory 104 can be used to store computer programs, for example, software programs and modules of application software, such as the computer program corresponding to the method for determining the keyword set in the embodiment of the present application. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, that is, to implement the above method. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 104 may further include a memory remotely arranged relative to the processor 102, and these remote memories may be connected to the computer terminal device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0029] The transmission device 106 is used to receive or send data via a network. The specific example of the above network may include a wireless network provided by a communication provider of a computer terminal device. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, referred to as NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 can be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0030] In this embodiment, a method for determining a keyword set is provided. Figure 2 is a flowchart of a method for determining a keyword set according to an embodiment of the present application, such as Figure 2 As shown, the process includes the following steps:

[0031] Step S202, obtaining a seed keyword set corresponding to the target material, and dividing the target material into title and content;

[0032] Among them, the above-mentioned target data can be the technical data of the hydropower station, which can include: equipment manuals, drawings, technical solutions, operation records, inspection reports, maintenance records, procedure drawings, typical cases, laws and regulations, etc.

[0033] Step S204, extracting keywords from the title and the content respectively to generate a candidate keyword set corresponding to the target data, wherein the candidate keyword set includes: a first keyword set corresponding to the title and a second keyword set corresponding to the content;

[0034] Step S206: determining a target keyword set corresponding to the target data according to the seed keyword set and the candidate keyword set.

[0035] The above seed keyword set includes: basic keywords in a specific field of the hydropower technology field.

[0036] Through the above steps, the seed keyword set corresponding to the target material is obtained, and the target material is divided into title and content; keyword extraction is performed on the title and content respectively to generate a first keyword set corresponding to the title and a second keyword set corresponding to the content, and the first keyword set and the second keyword set are collectively referred to as a candidate keyword set. The target keyword set corresponding to the target material is determined according to the seed keyword set and the candidate keyword set. In other words, the embodiment of the present application divides the target material into title and content, and then generates a first keyword set corresponding to the title and a second keyword set corresponding to the content. The target keyword set corresponding to the target material can be determined according to the obtained seed keyword set, the first keyword set and the second keyword set. Through the embodiment of the present application, the problem in the related art that a large amount of file materials are not finely divided, resulting in difficulty in finding the target material in a large amount of file materials can be solved, and then the target material can be quickly found in a large amount of file materials according to the target keyword set, thereby improving the search efficiency of the target material.

[0037] Optionally, the above step S202 of dividing the target data into title and content includes: determining a data format corresponding to the target data, wherein the data format includes: text format, picture format and video format; determining a target parsing method for the target data according to the data format; and dividing the target data into the title and the content according to the target parsing method.

[0038] Understandably, because technical information may come in a variety of formats, from PDF documents, image scans to video recordings, each format is handled differently. Specifically:

[0039] First, you should identify the format of the target data, i.e., text format (e.g., txt, doc, PDF), image format (e.g., jpg, png), or video format (e.g., mp4, avi). You can determine the data type of the target data by checking the file extension or file type.

[0040] Next, select the appropriate parsing method to extract the title and content:

[0041] Different data types correspond to different target parsing schemes. Wherein, determining the target parsing method of the target data according to the data format includes: when the data format is determined to be the text format, determining the target parsing method to be the first parsing method, wherein the first parsing method is: a text parsing method; when the data format is determined to be the image format, determining the target parsing method to be the second parsing method, wherein the second parsing method is: a method of parsing the target data through the first recognition model; when the data format is determined to be the video format, determining the target parsing method to be the third parsing method, wherein the third parsing method is: extracting video frames for the target data, inputting each frame of video into the second recognition model, and parsing the target data according to the output result corresponding to the second recognition model and the text accumulation algorithm.

[0042] That is, for text formats: the target parsing approach may involve reading the file directly and doing text processing. For example, a doc file can be read using Python's `docx` library, and then the title and content can be identified based on the document's structure, paragraph headings, or specific text tags. The title is usually at the beginning of the document and may use a larger font or heading tags (such as the "Heading 1" style).

[0043] For image formats: The parsing method will involve optical character recognition (OCR) technology. The OCR model can recognize text in an image and convert it into an editable text format. For example, for a scanned image containing a hydropower equipment drawing, OCR can recognize the text in the image, including the drawing title and equipment name, and may even include detailed instructions at the bottom of the drawing.

[0044] For video formats: parsing methods will include video frame extraction, image recognition, and speech-to-text technology. For example, for a video about the maintenance of hydropower station equipment, the system will first extract the key frames of the video, and then apply OCR technology to these image frames to identify the text information in them. At the same time, if the video contains sound descriptions, speech recognition technology can be used to convert them into text. The title of the video can usually be found in the metadata of the video file, such as in the file's properties or the description of the video platform.

[0045] Then, you can separate the title from the content according to the target parsing method above:

[0046] For example, PDF files often contain metadata fields such as titles, which can be read directly.

[0047] For another example, when processing a doc file, the title can be identified by the title style or position information (such as the beginning of the document).

[0048] For another example: For materials in certain formats, such as standard equipment manuals, predefined title and content format templates can be used for matching.

[0049] Through the above steps, the title and content can be accurately separated.

[0050] Optionally, the above step S204 performs keyword extraction on the title and the content respectively to generate a candidate keyword set corresponding to the target material, including: performing word frequency statistics on the content, and extracting a third keyword set from the content according to the statistical results; inputting the third keyword set into a keyword extractor to generate the second keyword set corresponding to the content; and, inputting the title into the keyword extractor to generate the first keyword set corresponding to the title.

[0051] It is understandable that the method for determining the candidate keyword set is:

[0052] Word frequency statistics: Word frequency statistics is a basic text analysis technique used to identify words that appear frequently in a text. For the content part, word frequency statistics are performed, which can be done using methods in NLP (natural language processing) libraries, such as `nltk` or `spaCy` in Python. The result of the word frequency statistics will produce a third keyword set, which contains words that appear frequently in the content. For example, in a report on the maintenance of hydropower station equipment, the words "generator", "stator", "inspection" and "maintenance" may be included in the third keyword set due to their high frequency of use in the document.

[0053] Application of keyword extractors: A keyword extractor is a more advanced text analysis tool that can evaluate the importance of words in a document based on statistics, graph models, or deep learning algorithms and select keywords from them. Keyword extractors are used to process the third set of keywords and the title part to generate a more accurate second set of keywords (for content) and first set of keywords (for title). Keyword extractors can be algorithm-based, such as TextRank or TF-IDF, or deep learning-based, such as Word2Vec or BERT model (a deep learning model).

[0054] For example, after processing the third set of keywords, the TextRank algorithm may be used to evaluate the relative importance of these words. TextRank is based on a graph model, which regards the words in the document as nodes in the graph, and the co-occurrence relationship between words as edges between nodes. The importance of the words is determined by calculating the PageRank value of the node. For the title part, the TextRank or TF-IDF algorithm may be used directly, because the titles are usually shorter and their vocabulary is more important. For example, for the title "Generator Maintenance Technology", the keyword extractor may directly divide it into "Generator", "Maintenance" and "Technology" without further word frequency statistics.

[0055] Finally, two sets of keywords will be generated: the first set of keywords (for the title) may contain all the important words in the title "Generator Maintenance Technology"; the second set of keywords (for the content) may contain "generator", "stator", "maintenance", "maintenance" and other words that are identified as important in the content. These two sets of keywords will be further combined and screened to generate the final set of candidate keywords for subsequent document analysis and retrieval tasks.

[0056] Optionally, the above-mentioned step S206 of determining the target keyword set corresponding to the target data based on the seed keyword set and the candidate keyword set includes: separately determining the target similarity between each candidate keyword in the candidate keyword set and each seed keyword in the seed keyword set; determining an average value corresponding to multiple target similarities, and determining the average value as the average similarity between the candidate keywords and the seed keywords; and determining the maximum similarity among the multiple target similarities; and determining the target keyword set based on the average similarity and the maximum similarity.

[0057] It can be understood that the method for determining the target keyword set is: for each word in the candidate keyword set, the similarity between each word in the seed keyword set is calculated. Here, cosine similarity is mainly used to measure the semantic similarity between words. Cosine similarity is obtained by calculating the cosine value of the angle between two vectors (here, vocabulary vectors), and the value range is between -1 and 1. The closer the value is to 1, the higher the similarity.

[0058] For example, suppose there is the following seed keyword set: {"generator", "repair", "maintenance", "equipment"}, and the candidate keyword set: {"generator", "stator", "bearing", "oscillation", "renovation"}. The cosine similarity between "stator", "bearing", "oscillation", "renovation" and each word in the seed keyword set can be calculated respectively.

[0059] Determination of average similarity and maximum similarity: For each candidate keyword, its similarity with all seed keywords is calculated, and then the average of these similarities is calculated as the average similarity of the candidate keyword. At the same time, the maximum similarity is determined from the calculated similarities as the best match between the candidate keyword and the seed keyword.

[0060] For example, for the candidate keyword "stator", the cosine similarity between it and "generator", "repair", "maintenance", and "equipment" is calculated respectively, and it is assumed that the obtained similarities are 0.8, 0.3, 0.4, and 0.7 respectively. Then, the average similarity will be (0.8+0.3+0.4+0.7) / 4=0.525, and the maximum similarity is 0.8 (the similarity between "generator").

[0061] Generation of a target keyword set: determining the target keyword set according to the average similarity and the maximum similarity, including: assigning scores to candidate keywords corresponding to each target similarity according to the average similarity and the maximum similarity; determining the target keyword set in the candidate keyword set according to the score of each candidate keyword.

[0062] That is to say, the target keyword set is determined according to the average similarity and the maximum similarity. Generally, a threshold can be set, and if the average similarity or the maximum similarity of the candidate keyword is higher than the threshold, the keyword will be added to the target keyword set.

[0063] For example, if the threshold is set to 0.5, candidate keywords "stator" and "bearing" may be selected into the target keyword set because their average similarity or maximum similarity is higher than the threshold, while "oscillation" and "renovation" may be excluded because of their low similarity.

[0064] In addition, different strategies can be set according to actual needs, such as selecting keywords based only on the maximum similarity or combining the average similarity and the weighted average of the maximum similarity. The choice of this strategy will directly affect the generation of the target keyword set and needs to be adjusted according to specific application scenarios and needs.

[0065] Through the above process, a target keyword set closely related to the seed keyword set can be generated. These keywords not only reflect the theme and content of the document, but also have a high relevance to specific fields (such as the hydropower field). This step is the core part of keyword extraction, which ensures the domain relevance and semantic accuracy of keywords, and provides a high-quality vocabulary foundation for subsequent document analysis, retrieval and understanding.

[0066] Wherein, respectively determining the target similarity of each candidate keyword in the candidate keyword set and each seed keyword in the seed keyword set includes: converting each candidate keyword into each first vector according to a word embedding model, and converting each seed keyword into each second vector according to the word embedding model; determining the cosine similarity of each first vector and each second vector, and determining the cosine similarity as the target similarity.

[0067] It is understandable that word embedding models (such as Word2Vec, GloVe or BERT) convert words into multi-dimensional vectors that can capture the semantic information of the words. By calculating the cosine similarity between these vectors, the correlation between words can be evaluated and the target similarity can be determined. Specifically:

[0068] Generation of word vectors: A word embedding model can be used to convert words into vector representations. The word embedding model learns the representation of words in a multidimensional space by training on a large-scale text corpus, so that semantically similar words are close in distance in the vector space. For example, using the Word2Vec model, the system can convert the seed keywords "generator", "repair", "maintenance", and "equipment" into corresponding vector representations, assuming that they are represented as vectors A, B, C, and D respectively.

[0069] For words in the candidate keyword set, such as "stator", "bearing", "vibration", and "renovation and transformation", the system also uses the same word embedding model to convert them into vectors, assuming that they are represented as vectors X, Y, Z, and W respectively.

[0070] Calculation of cosine similarity: After obtaining the vector representation of the vocabulary, the cosine similarity between each candidate keyword vector and the seed keyword vector is calculated. Cosine similarity is obtained by calculating the dot product of two vectors divided by the product of their moduli. The result is a value between -1 and 1. The closer the value is to 1, the higher the similarity.

[0071] For example, for the candidate keyword "stator" (vector X), the system will calculate the cosine similarity between it and the seed keywords "generator" (vector A), "overhaul" (vector B), "maintenance" (vector C), and "equipment" (vector D). Assume that the calculated similarities are: cosθ(X, A) = 0.8, cosθ(X, B) = 0.3, cosθ(X, C = 0.4, cosθ(X, D) = 0.7. Then, these similarities will be used as the basis for evaluating the relevance of "stator" to the hydropower field.

[0072] For other candidate keywords, such as "bearing" (vector Y), "oscillation" (vector Z) and "renovation and transformation" (vector W), the system will also calculate their cosine similarity with each seed keyword in the seed keyword set in the same way to evaluate their relevance to the hydropower field.

[0073] Determination of target similarity: These cosine similarities are used as target similarities for subsequent keyword screening and sorting. For example, for "stator", its target similarity set will be {0.8, 0.3, 0.4, 0.7}, which reflects its semantic relevance to the seed keywords "generator", "repair", "maintenance" and "equipment".

[0074] Through the above process, the similarity between candidate keywords and seed keywords in specific fields (such as hydropower field) can be quantitatively evaluated based on the word embedding model and cosine similarity calculation, thereby providing an objective basis for screening and generating target keyword sets.

[0075] In order to better understand the process of the method for determining the above keyword set, the implementation method flow of determining the above keyword set is described below in combination with an optional embodiment, but it is not used to limit the technical solution of the embodiment of the present application.

[0076] Keyword extraction refers to the process of automatically extracting words or phrases from text that can summarize or express the theme, content or intention of the text. These keywords usually have high information value and can help readers or systems quickly understand the core content of the text.

[0077] In the related art, there are the following technical solutions for keyword extraction:

[0078] 1) Statistical generation method:

[0079] Through TF-IDF (Term Frequency-Inverse Document Frequency): TF (term frequency) measures the frequency of a word in a document; IDF (inverse document frequency) measures the prevalence of a word in all documents and penalizes common words. That is, combining TF and IDF can evaluate the importance of a word in a document. And further word frequency statistics can be performed: simply count the number of times each word appears in the document, and then select keywords based on the set threshold or ranking.

[0080] 2) Rule-based and heuristic methods:

[0081] (1) Rules based on part of speech: Keywords are selected based on part of speech (e.g., noun, verb, etc.). Nouns and verbs are generally considered more representative.

[0082] (2) Rules based on position information: Select keywords based on the position of the word in the document (such as title, beginning of paragraph, etc.).

[0083] 3) Graph-based methods:

[0084] The document is modeled using a graph model, where each word in the document is considered a node in the graph, and the co-occurrence relationship between words is considered an edge between nodes. The importance of each word is evaluated by calculating the PageRank value of the node (or other similar graph sorting algorithms).

[0085] 4) Deep learning-based methods:

[0086] (1) Supervised Learning:

[0087] Use a labeled keyword dataset for training and learn a classifier to distinguish keywords from non-keywords. This method usually requires a large amount of labeled data, and the performance of the model depends on the quality and quantity of the training data.

[0088] (2) Unsupervised Learning:

[0089] Use unsupervised learning methods such as clustering and dimensionality reduction to preprocess documents, and then select keywords based on the processing results. This method does not require labeled data, but may be affected by data noise and sparsity.

[0090] The above keyword extraction task is usually an important initial step in unsupervised information extraction, laying the foundation for tasks such as topic modeling or document classification. Although the above methods have been proven to be very effective in extracting keywords, identifying keywords of a specific category, or only keywords related to the hydropower field, remains challenging.

[0091] The optional embodiment of the present application defines a keyword extraction method guided by seed keywords, which aims to extract high-frequency words, keywords, and document features of hydropower technical documents by constructing a set of extraction schemes that conform to the actual situation on site, and provide an attribute basis for subsequent graph construction, specifically:

[0092] (1) Extraction of data titles and text content: The technical data of hydropower stations (i.e., target data) include equipment manuals, drawings, technical plans, operation records, inspection reports, maintenance records, procedure drawings, typical cases, laws and regulations, etc. There are many types of data formats, so different types of data need to be processed.

[0093] Figure 3 is a schematic diagram of extracting a document title and text according to an optional embodiment of the present application, such as Figure 3As shown: data is identified according to different types of input data, titles and text contents of technical information in the hydropower field are extracted, titles and text contents are input as data, and data types are determined. Among them, PDF and picture types (i.e. picture formats) are extracted using an optical character recognition model (OCR model for short, i.e. the first recognition model), word, excel, txt, etc. (i.e. text formats) are extracted using text parsing, and videos (i.e. video formats) are separated from titles and texts through frame extraction, OCR recognition (i.e. the second recognition model), and text accumulation algorithm.

[0094] (2) Extract keyword set:

[0095] Figure 4 is a flowchart of extracting a keyword set according to an optional embodiment of the present application, such as Figure 4 As shown:

[0096] The title enters the keyword filter to directly split the candidate keywords (generate the first keyword set). The content input extracts the common text keywords (i.e. the third keyword set) through word frequency statistics and graph analysis, and enters the keyword filter as candidate keywords (i.e. the second keyword set). By calculating the scores of the seed keywords (i.e. the seed keyword set) and the candidate keywords (i.e.), the expected keywords are selected as a member of the extracted word set.

[0097] Candidate keywords are keywords extracted from the content of unstructured input documents through keyword extraction methods.

[0098] The score is calculated by taking a set of seed keywords as input parameters and extracting candidate keywords using the weighted average of seed keyword embedding and document embedding. When focusing on seed keywords of a specific class, it is configured to focus 100% on the results of seed keyword embedding. That is, the seed keywords are added to the keyword filter to extract the word set.

[0099] The steps for calculating the score are: average score, compare each candidate word with each seed keyword using similarity, and average these results into an average score.

[0100] Highest score: Similar to the average score, but only with the maximum cosine similarity score, the maximum score can be obtained.

[0101] For example, there are n candidate keywords and m seed keywords. Each candidate keyword will be similar to all seed keywords one by one (the similarity algorithm used is cosine), and a set of similarity results {cos_theta_1, ..., cos_theta_m} will be obtained. Then, the average and maximum of this set will be calculated to obtain the average similarity and maximum similarity of this candidate keyword.

[0102] in, Cosine Similarity : is a method to measure the angle between two vectors in vector space. The formula is: cosθ=(A·B) / (||A||×||B||).

[0103] Among them, A and B are two vectors, cosθ - Represents the cosine similarity between vectors A and B, with a value between -1 and 1 The specific parameters are as follows:

[0104] A·B: represents the dot product of vectors A and B. The calculation formula is A·B=Σ(ai×bi), where ai and bi are Elements in vectors A and B. The dot product can be calculated by multiplying corresponding elements and then summing them.

[0105] ||A|| and ||B||: represent the modulus (length) of vectors A and B respectively, and the calculation formula is ||A||=sqrt(Σ (ai2)). The magnitude can be found by summing the squares of each element in the vector and then taking the square root.

[0106] (3) Seed keyword selection:

[0107] Seed keywords are basic keywords that describe a specific field or topic, usually short-tail keywords of one or two words.

[0108] Seed keywords are mainly summarized based on vocabulary accumulated from implementation experience, field equipment feature data, material classification types, professional-related vocabulary, etc.

[0109] For example, generator, stator, bearing, vibration, etc. are derived from equipment characteristic data. Repair, maintenance, renovation and transformation are derived from data classification, and 110kV, 400V, 24V, 48V are derived from professional related vocabulary, etc.

[0110] The optional embodiment of the present application uses a hybrid keyword extraction scheme that separates the title and content for discrimination. At the same time, it is necessary to establish a seed keyword set for hydropower technical information instead of randomly extracting keywords, which enhances the effect of keyword extraction in specific fields.

[0111] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, computer, server, or network device, etc.) to execute the methods described in each embodiment of the present application.

[0112] In the present embodiment, a device for determining a keyword set is also provided, and the device is used to implement the above-mentioned embodiments and preferred implementation modes, and the descriptions that have been made will not be repeated. As used below, the term "module" can implement a combination of software and / or hardware of a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceived.

[0113] Figure 5 is a structural block diagram of a device for determining a keyword set according to an embodiment of the present application, such as Figure 5 As shown, the device comprises:

[0114] An acquisition module 52 is used to acquire a seed keyword set corresponding to the target data, and divide the target data into a title and a content;

[0115] An extraction module 54 is used to extract keywords from the title and the content respectively to generate a candidate keyword set corresponding to the target data, wherein the candidate keyword set includes: a first keyword set corresponding to the title and a second keyword set corresponding to the content;

[0116] The determination module 56 is used to determine the target keyword set corresponding to the target data according to the seed keyword set and the candidate keyword set.

[0117] Through the above-mentioned device, a seed keyword set corresponding to the target data is obtained, and the target data is divided into a title and a content; keywords are extracted for the title and the content respectively to generate a first keyword set corresponding to the title and a second keyword set corresponding to the content, and the first keyword set and the second keyword set are collectively referred to as a candidate keyword set. The target keyword set corresponding to the target data is determined according to the seed keyword set and the candidate keyword set. In other words, the embodiment of the present application divides the target data into a title and a content, and then generates a first keyword set corresponding to the title and a second keyword set corresponding to the content. The target keyword set corresponding to the target data can be determined according to the obtained seed keyword set, the first keyword set and the second keyword set. Through the embodiment of the present application, the problem in the related art that a large amount of file data is not finely divided, resulting in difficulty in finding the target data in a large amount of file data can be solved, and then the target data can be quickly found in a large amount of file data according to the target keyword set, thereby improving the search efficiency of the target data.

[0118] In an exemplary embodiment, the acquisition module 52 is also used for the data formats including: text format, picture format and video format; determining the target parsing method of the target data according to the data format; and dividing the target data into the title and the content according to the target parsing method.

[0119] In an exemplary embodiment, the extraction module 54 is further used to, when it is determined that the data format is the text format, determine that the target parsing method is a first parsing method, wherein the first parsing method is: a text parsing method; when it is determined that the data format is the image format, determine that the target parsing method is a second parsing method, wherein the second parsing method is: a method of parsing the target data through a first recognition model; when it is determined that the data format is the video format, determine that the target parsing method is a third parsing method, wherein the third parsing method is: a method of extracting video frames for the target data, inputting each frame of video into the second recognition model, and parsing the target data according to the output result corresponding to the second recognition model and a text accumulation algorithm.

[0120] In an exemplary embodiment, the extraction module 54 is also used to perform word frequency statistics on the content, and extract a third keyword set from the content based on the statistical results; input the third keyword set into the keyword extractor to generate the second keyword set corresponding to the content; and input the title into the keyword extractor to generate the first keyword set corresponding to the title.

[0121] In an exemplary embodiment, the determination module 56 is further used to respectively determine the target similarity between each candidate keyword in the candidate keyword set and each seed keyword in the seed keyword set; determine an average value corresponding to multiple target similarities, and determine the average value as the average similarity between the candidate keyword and the seed keyword; and determine the maximum similarity among the multiple target similarities; and determine the target keyword set based on the average similarity and the maximum similarity.

[0122] In an exemplary embodiment, the determination module 56 is further configured to assign a score to each candidate keyword corresponding to the target similarity according to the average similarity and the maximum similarity; and determine the target keyword set in the candidate keyword set according to the score of each candidate keyword.

[0123] In an exemplary embodiment, the determination module 56 is further used to convert each candidate keyword into each first vector according to the word embedding model, and convert each seed keyword into each second vector according to the word embedding model; determine the cosine similarity between each first vector and each second vector, and determine the cosine similarity as the target similarity.

[0124] It should be noted that the above modules can be implemented by software or hardware. For the latter, it can be implemented in the following ways, but not limited to: the above modules are all located in the same processor; or the above modules are located in different processors in any combination.

[0125] An embodiment of the present application further provides a computer-readable storage medium, in which a computer program is stored, wherein the computer program is configured to execute the steps of any of the above method embodiments when running.

[0126] Optionally, in this embodiment, the storage medium may be configured to store program codes for executing the following steps:

[0127] S1, obtaining a seed keyword set corresponding to a target material, and dividing the target material into a title and content;

[0128] S2, extracting keywords from the title and the content respectively to generate a candidate keyword set corresponding to the target data, wherein the candidate keyword set includes: a first keyword set corresponding to the title and a second keyword set corresponding to the content;

[0129] S3, determining a target keyword set corresponding to the target data according to the seed keyword set and the candidate keyword set.

[0130] In an exemplary embodiment, the computer-readable storage medium may include, but is not limited to, various media that can store computer programs, such as a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk.

[0131] An embodiment of the present application further provides an electronic device, including a memory and a processor, wherein a computer program is stored in the memory, and the processor is configured to run the computer program to execute the steps in any one of the above method embodiments.

[0132] In an exemplary embodiment, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor, and the input / output device is connected to the processor.

[0133] Optionally, in this embodiment, the processor may be configured to perform the following steps through a computer program:

[0134] S1, obtaining a seed keyword set corresponding to a target material, and dividing the target material into a title and content;

[0135] S2, extracting keywords from the title and the content respectively to generate a candidate keyword set corresponding to the target data, wherein the candidate keyword set includes: a first keyword set corresponding to the title and a second keyword set corresponding to the content;

[0136] S3, determining a target keyword set corresponding to the target data according to the seed keyword set and the candidate keyword set.

[0137] An embodiment of the present application further provides a computer program product, which includes a computer program. When the computer program is executed by a processor, the steps in any one of the above method embodiments are implemented.

[0138] An embodiment of the present application further provides another computer program product, including a non-volatile computer-readable storage medium, wherein the non-volatile computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in any of the above method embodiments are implemented.

[0139] An embodiment of the present application also provides a computer program, which includes computer instructions, which are stored in a computer-readable storage medium; a processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device performs the steps in any one of the above method embodiments.

[0140] Optionally, in this embodiment, the processor may be configured to perform the following steps through a computer program:

[0141] S1, obtaining a seed keyword set corresponding to a target material, and dividing the target material into a title and content;

[0142] S2, extracting keywords from the title and the content respectively to generate a candidate keyword set corresponding to the target data, wherein the candidate keyword set includes: a first keyword set corresponding to the title and a second keyword set corresponding to the content;

[0143] S3, determining a target keyword set corresponding to the target data according to the seed keyword set and the candidate keyword set.

[0144] For specific examples in this embodiment, reference may be made to the examples described in the above embodiments and exemplary implementation modes, and this embodiment will not be described in detail herein.

[0145] Obviously, those skilled in the art should understand that the above modules or steps of the present application can be implemented by a general computing device, they can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices, they can be implemented by a program code executable by a computing device, so that they can be stored in a storage device and executed by the computing device, and in some cases, the steps shown or described can be executed in a different order from that herein, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. Thus, the present application is not limited to any specific combination of hardware and software.

[0146] The above description is only the preferred embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the principles of the present application shall be included in the protection scope of the present application.

Claims

1. A method for determining a keyword set, characterized in that: include: Obtaining a seed keyword set corresponding to the target material, and dividing the target material into a title and content; Performing keyword extraction on the title and the content respectively to generate a candidate keyword set corresponding to the target data, wherein the candidate keyword set includes: a first keyword set corresponding to the title and a second keyword set corresponding to the content; A target keyword set corresponding to the target data is determined according to the seed keyword set and the candidate keyword set.

2. The method according to claim 1, characterized in that Divide the target material into titles and contents, including: Determine a data format corresponding to the target data, wherein the data format includes: text format, image format and video format; Determining a target parsing method for the target data according to the data format; The target data is divided into the title and the content according to the target parsing method.

3. The method according to claim 2, characterized in that Determining a target parsing method of the target data according to the data format includes: When the data format is determined to be the text format, the target parsing method is determined to be a first parsing method, wherein the first parsing method is: a text parsing method; When it is determined that the data format is the picture format, the target parsing method is determined to be a second parsing method, wherein the second parsing method is: a method of parsing the target data through a first recognition model; When it is determined that the data format is the video format, the target parsing method is determined to be the third parsing method, wherein the third parsing method is: extracting video frames for the target data, inputting each frame of video into a second recognition model, and parsing the target data according to the output result corresponding to the second recognition model and a text accumulation algorithm.

4. The method according to claim 1, characterized in that: Keyword extraction is performed on the title and the content respectively to generate a candidate keyword set corresponding to the target data, including: Performing word frequency statistics on the content, and extracting a third keyword set from the content according to the statistical results; inputting the third keyword set into a keyword extractor to generate the second keyword set corresponding to the content; and, The title is input into the keyword extractor to generate the first keyword set corresponding to the title.

5. The method according to claim 1, characterized in that Determining a target keyword set corresponding to the target data according to the seed keyword set and the candidate keyword set includes: respectively determining a target similarity between each candidate keyword in the candidate keyword set and each seed keyword in the seed keyword set; Determine an average value corresponding to a plurality of target similarities, and determine the average value as an average similarity between the candidate keyword and the seed keyword; and, Determining a maximum similarity among the plurality of target similarities; The target keyword set is determined according to the average similarity and the maximum similarity.

6. The method according to claim 5, characterized in that Determining the target keyword set according to the average similarity and the maximum similarity includes: Assigning points to the candidate keywords corresponding to each target similarity according to the average similarity and the maximum similarity; The target keyword set is determined from the candidate keyword set according to the score of each candidate keyword.

7. The method according to claim 5, characterized in that Respectively determining a target similarity between each candidate keyword in the candidate keyword set and each seed keyword in the seed keyword set includes: Convert each candidate keyword into each first vector according to the word embedding model, and convert each seed keyword into each second vector according to the word embedding model; The cosine similarity between each of the first vectors and each of the second vectors is determined, and the cosine similarity is determined as the target similarity.

8. A device for determining a keyword set, characterized in that: include: An acquisition module, used to acquire a seed keyword set corresponding to the target data, and divide the target data into a title and content; An extraction module is used to extract keywords from the title and the content respectively to generate a candidate keyword set corresponding to the target material, wherein the candidate keyword set includes: a first keyword set corresponding to the title and a second keyword set corresponding to the content; a determination module is used to determine a target keyword set corresponding to the target material based on the seed keyword set and the candidate keyword set.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored program, wherein the program executes the method of any one of claims 1 to 7 when executed.

10. An electronic device comprising a memory and a processor, characterized in that: A computer program is stored in the memory, and the processor is configured to execute the method according to any one of claims 1 to 7 through the computer program.

Images