Keyword extraction method and device for water conservancy data, equipment and storage medium
By extracting and screening water conservancy data features, the problem of inaccurate keyword extraction in the prior art is solved, more accurate keyword extraction is achieved, and the efficiency of information retrieval and analysis is enhanced.
Patent Information
- Application Number
- CN202411306909.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-19
- Publication Date
- 2024-12-27
AI Technical Summary
The prior art has insufficient understanding of semantic information and contextual relationships when extracting keywords in water conservancy data, resulting in inaccurate keywords.
By extracting the water conservancy data features, statistical feature vectors, vocabulary feature vectors and context feature vectors are obtained, and input them into the trained keyword extraction model, and filtering out the target keywords based on the scoring threshold to improve the accuracy of the extraction.
It has achieved more accurate and effective keyword extraction of water conservancy data, comprehensively captured important information in the data, and improved the efficiency of information retrieval and analysis.
Smart Images
Figure CN119204002A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the technical field of natural language processing, and in particular to a method, device, equipment and storage medium for extracting keywords from water conservancy data. Background Art
[0002] With the rapid development of information technology and the popularization of the Internet, massive amounts of text data continue to emerge. How to quickly and accurately extract valuable key information from these massive amounts of data has become a key focus.
[0003] As one of the key technologies in the field of text mining and natural language processing, keyword extraction technology can effectively extract important words that summarize the text from the text, providing important support for text content analysis, information retrieval and knowledge discovery, and therefore has important significance in different application scenarios. For example, in the intelligent search scenario, the use of keyword extraction technology can improve the search engine's understanding and matching capabilities of text content, thereby improving the efficiency and accuracy of information retrieval; for another example, in the scenario of topic detection and sentiment analysis, the use of keyword extraction technology to extract keywords from the text can effectively perform topic detection and sentiment analysis, helping companies and researchers understand the public's concerns and emotional tendencies; for another example, in the process of water conservancy monitoring, the use of keyword extraction technology to extract important water conservancy data from water conservancy data can help detection personnel to effectively locate and analyze.
[0004] At present, statistical algorithms are mainly used for keyword extraction. For example, statistical features such as word frequency, word co-occurrence, and term frequency-inverse document frequency (TF-IDF) are used to extract keywords. However, this method has limitations when it comes to semantic information and contextual relationships, resulting in inaccurate extracted keywords. Summary of the invention
[0005] The present application provides a method, device, equipment and storage medium for extracting keywords from water conservancy data, so as to improve the accuracy of keyword extraction.
[0006] In a first aspect, a method for extracting keywords from water conservancy data is provided, comprising:
[0007] Performing feature extraction on the water conservancy data to obtain N feature vectors of the water conservancy data; wherein the N feature vectors include a statistical feature vector, a vocabulary feature vector, and a context feature vector of the water conservancy data, and N is an integer greater than 2;
[0008] Inputting the statistical feature vector, the vocabulary feature vector, and the context feature vector into a trained keyword extraction model to obtain M candidate keywords of the water conservancy data and keyword scores of each of the M candidate keywords; wherein M is an integer greater than 0;
[0009] According to the keyword scores of the M candidate keywords, P target keywords with scores greater than a threshold are screened out from the M candidate keywords.
[0010] In some embodiments, before extracting features from the water conservancy data to obtain N feature vectors of the water conservancy data, the method further includes:
[0011] Removing noise data from the original water conservancy data to obtain the water conservancy data: wherein the noise data at least includes HTML tags, special characters, and stop words; or
[0012] Using a word segmenter to segment the original water conservancy data to obtain the water conservancy data; or
[0013] Adding keyword tags to key data in the original water conservancy data to obtain the water conservancy data; or
[0014] Part-of-speech tagging is performed on all data in the original water conservancy data to obtain the water conservancy data.
[0015] In some embodiments, before extracting features from the water conservancy data to obtain N feature vectors of the water conservancy data, the method further includes:
[0016] Performing feature extraction on the collected sample data set to obtain a statistical feature sample set, a vocabulary feature sample set, and a context feature sample set of the sample data set;
[0017] The statistical feature sample set, the vocabulary feature sample set, and the context feature sample set are trained until the loss function of the keyword extraction model meets the loss function requirement.
[0018] In some embodiments, the keyword extraction model is a lightweight neural network model.
[0019] In some embodiments, after selecting P target keywords with scores greater than a score threshold from the M candidate keywords, the method further includes:
[0020] The P target keywords are arranged in descending order, G target keywords are screened out from the P target keywords according to the ranking order requirement, and the G target keywords are output.
[0021] In a second aspect, a keyword extraction device for water conservancy data is provided, comprising:
[0022] A feature extraction module, used for performing feature extraction on the water conservancy data to obtain N feature vectors of the water conservancy data; wherein the N feature vectors include a statistical feature vector, a vocabulary feature vector, and a context feature vector of the water conservancy data, and N is an integer greater than 2;
[0023] A keyword extraction module is used to input the statistical feature vector, the vocabulary feature vector, and the context feature vector into a trained keyword extraction model to obtain M candidate keywords for the water conservancy data and keyword scores for each of the M candidate keywords; wherein M is an integer greater than 0; and is used to screen out P target keywords with a score greater than a threshold from the M candidate keywords based on the keyword scores of each of the M candidate keywords.
[0024] In some embodiments, the apparatus further comprises:
[0025] A preprocessing module; used to remove noise data from the original water conservancy data to obtain the water conservancy data: wherein the noise data at least includes HTML tags, special characters, and stop words; or
[0026] Using a word segmenter to segment the original water conservancy data to obtain the water conservancy data; or
[0027] Adding keyword tags to key data in the original water conservancy data to obtain the water conservancy data; or
[0028] Part-of-speech tagging is performed on all data in the original water conservancy data to obtain the water conservancy data.
[0029] In some embodiments, the apparatus further comprises:
[0030] A keyword extraction model building module is used to extract features from the collected sample data set to obtain a statistical feature sample set, a vocabulary feature sample set, and a context feature sample set of the sample data set;
[0031] The statistical feature sample set, the vocabulary feature sample set, and the context feature sample set are trained until the loss function of the keyword extraction model meets the loss function requirement.
[0032] According to a third aspect, an electronic device is provided, including:
[0033] A memory for storing a computer program; a processor for implementing any one of the method steps in the first aspect when executing the computer program stored in the memory.
[0034] According to a fourth aspect, a computer-readable storage medium is provided, wherein a computer program is stored in the computer-readable storage medium, and when the computer program is executed by a processor, the method steps described in any one of the first aspects are implemented.
[0035] In an embodiment of the present application, first, feature extraction is performed on the water conservancy data to obtain N feature vectors of the water conservancy data, which include statistical feature vectors, lexical feature vectors, and contextual feature vectors of the water conservancy data. Feature extraction is performed from different dimensions to comprehensively capture key data in the water conservancy data; then, the statistical feature vectors, lexical feature vectors, and contextual feature vectors are input into a trained keyword extraction model to obtain M candidate keywords for the water conservancy data and keyword scores for each of the M candidate keywords; based on the keyword scores of each of the M candidate keywords, P target keywords that are greater than a score threshold are screened out from the M candidate keywords to improve the accuracy of keyword extraction.
[0036] For each aspect from the second to the fourth aspect and the technical effects that may be achieved by each aspect, please refer to the above description of the technical effects that can be achieved by the first aspect or various possible schemes in the first aspect, and no further details will be given here. BRIEF DESCRIPTION OF THE DRAWINGS
[0037] Figure 1 A schematic diagram of an application scenario applicable to the embodiments of the present application;
[0038] Figure 2 A flowchart of a keyword extraction method for water conservancy data provided in an embodiment of the present application;
[0039] Figure 3 A logical schematic diagram of keyword extraction of water conservancy data provided in an embodiment of the present application;
[0040] Figure 4 A schematic diagram of the structure of a keyword extraction device for water conservancy data provided in an embodiment of the present application;
[0041] Figure 5 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0042] In order to make the purpose, technical scheme and advantages of the present application clearer, the technical scheme in the embodiment of the present application will be clearly and completely described below in conjunction with the drawings in the embodiment of the present application. Obviously, the described embodiment is only a part of the embodiment of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present application. In the absence of conflict, the embodiments in the present application and the features in the embodiments can be arbitrarily combined with each other. In addition, although the logical order is shown in the flow chart, in some cases, the steps shown or described can be performed in an order different from that here.
[0043] The terms "first" and "second" in the specification and claims of the present application and the above-mentioned drawings are used to distinguish different objects, rather than to describe a specific order. In addition, the term "comprising" and any of their variations are intended to cover non-exclusive protection. For example, a process, method, system, product or device comprising a series of steps or units is not limited to the listed steps or units, but optionally also includes steps or units that are not listed, or optionally also includes other steps or units inherent to these processes, methods, products or devices. "Multiple" in the present application can mean at least two, for example, two, three or more, and the embodiments of the present application are not limited.
[0044] The following is a description of exemplary embodiments of the present application in conjunction with the accompanying drawings, including various details of the embodiments of the present application to facilitate understanding, which should be considered as merely exemplary. Therefore, those of ordinary skill in the art should recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope of the present application. Similarly, for the sake of clarity and conciseness, the description of well-known functions and structures is omitted in the following description. It should be noted that in the embodiments of the present application, certain software, components, models and other existing solutions in the industry may be mentioned, which should be considered as exemplary, and their purpose is only to illustrate the feasibility of the implementation of the technical solution of the present application, but it does not mean that the applicant has or will necessarily use the solution.
[0045] In order to better understand the embodiments of the present application, the technical terms involved in the embodiments of the present application are first explained below.
[0046] (1) TF-IDF (Term Frequency-Inverse Document Frequency) is a method to measure the importance of a word in a text. It takes into account the frequency of a word in a text and its inverse document frequency in the entire document collection.
[0047] (2) The bag-of-words model is a simple and commonly used text representation method. It treats text as an unordered collection of words and ignores the order of words in the text. The bag-of-words model does not consider the grammar and word order in the text, but only focuses on the frequency of word occurrence.
[0048] The following is a brief introduction to the application scenarios to which the technical solutions of the embodiments of the present application can be applied. It should be noted that the application scenarios introduced below are only used to illustrate the embodiments of the present application and are not limited. In specific implementation, the technical solutions provided by the embodiments of the present application can be flexibly applied according to actual needs.
[0049] Figure 1 Schematic diagram of application scenarios applicable to the embodiments of this application. Figure 1 As shown, the scenario includes a terminal 10 and a server 11. The terminal 10 and the server 11 may exchange information via a communication network, and the communication mode adopted by the communication network may include: a wireless communication mode and a wired communication mode.
[0050] In some scenarios, the terminal 10 can access the network and communicate with the server 11 through cellular mobile communication technology, and the cellular mobile communication technology may include the fifth generation mobile communication (5th Generation Mobile Networks, 5G) technology.
[0051] In some scenarios, the terminal 10 may access the network and communicate with the server 11 via short-range wireless communication. The short-range wireless communication may include Wireless Fidelity (Wi-Fi) technology.
[0052] The present application embodiment does not impose any restriction on the number of the above-mentioned devices. Figure 1 As shown, only the terminal 10 and the server 11 are described as examples, and the above-mentioned devices and their respective functions are briefly introduced below.
[0053] The terminal 10 is a device that can provide voice and / or data connectivity to the user, including: a handheld terminal device with a wireless connection function, a vehicle-mounted terminal device, etc.
[0054] In some scenarios, the terminal 10 includes but is not limited to: mobile phones, tablet computers, laptop computers, PDAs, mobile Internet devices (MID), wearable devices, virtual reality (VR) devices, augmented reality (AR) devices, wireless terminal devices in industrial control, wireless terminal devices in unmanned driving, wireless terminal devices in smart grids, wireless terminal devices in transportation safety, wireless terminal devices in smart cities, or wireless terminal devices in smart homes, etc.
[0055] In addition, a client related to keyword extraction may be installed on the terminal 10, and the client may be software (e.g., APP, browser, etc.), or a web page, applet, etc. In the embodiment of the present application, the terminal 10 may use the client related to keyword extraction, and may interact with the server 11 to perform keyword extraction of different water conservancy data.
[0056] In some scenarios, server 11 can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDN), as well as big data and artificial intelligence platforms.
[0057] Furthermore, in an embodiment of the present application, the server 11 may be equipped with a keyword extraction device for the water conservancy data corresponding to the client. The keyword extraction device is used to extract keywords from various data input by the terminal 10 (for example, water conservancy data collected in real time) to locate important water conservancy data for the convenience of users.
[0058] To further illustrate the technical solution provided by the embodiment of the present application, this is described in detail below in conjunction with the accompanying drawings and specific implementation methods. Although the embodiment of the present application provides the method operation steps as shown in the following embodiments or drawings, more or fewer operation steps may be included in the method based on routine or no creative labor. In the steps where there is no necessary causal relationship logically, the execution order of these steps is not limited to the execution order provided by the embodiment of the present application. The method may be executed or executed in parallel in the actual processing process or when the device is executed in accordance with the method sequence shown in the embodiment or drawings.
[0059] Figure 2The flowchart of a keyword extraction method for water conservancy data provided in an embodiment of the present application. The process can be executed by a keyword extraction device for water conservancy data, which can be implemented by software and / or hardware to improve the accuracy of keyword extraction. Figure 2 As shown, the process includes the following steps:
[0060] 201: Extract features from the water conservancy data to obtain N feature vectors of the water conservancy data.
[0061] The N feature vectors include statistical feature vectors, vocabulary feature vectors, and context feature vectors of water conservancy data, and N is an integer greater than 2.
[0062] In some embodiments, the statistical feature vector of the water conservancy data can be extracted using a statistical algorithm (for example, a bag-of-words model, TF-DF, etc.) to measure the importance of different data in the water conservancy data. The vocabulary feature vector can be extracted using a trained classifier (for example, a support vector machine, a random forest, a gradient boosting model, etc.) to filter out irrelevant data in the water conservancy data. The context feature vector can be extracted using a deep learning model (for example, a convolutional neural network model, a recurrent neural network model, an attention mechanism model, a BERT model, etc.) to capture the semantic information of the data and the relationship between the data. Features of different dimensions can be extracted from the water conservancy data to provide richer feature information for subsequent keyword extraction.
[0063] In some embodiments, before extracting features from water conservancy data to obtain N feature vectors of water conservancy data, the acquired raw water conservancy data may be preprocessed to obtain the water conservancy data, so that subsequent keyword extraction based on the water conservancy data is more accurate. Specifically, preprocessing the raw water conservancy data may include the following methods:
[0064] Method 1: remove noise data from the original water conservancy data; wherein the noise data at least includes HTML tags, special characters, and stop words.
[0065] Method 2: Use a word segmenter to segment the original water conservancy data; for example, the original water conservancy data is segmented into words or phrases. Further, the word segmenter can be flexibly selected according to the text type of the original water conservancy data, and the embodiment of the present application does not limit this.
[0066] Method 3, adding keyword labels to the key data in the original water conservancy data; for example, the key data can be labeled as True, and other data can be labeled as False for subsequent supervised learning.
[0067] Method 4, performs part-of-speech tagging for all data in the original water conservancy data; for example, the corresponding part of speech (including conjunctions, auxiliary words, participles, etc.) is marked for each data to facilitate subsequent feature extraction.
[0068] Method 5 can also perform stem extraction on the English data contained in the original water conservancy data to reduce the impact of word form changes.
[0069] It should be noted that when preprocessing the original water conservancy data, one or more methods can be selected for processing, and the above-listed methods are only examples. Other preprocessing methods can also be used according to actual needs, and the embodiments of the present application are not limited here.
[0070] In some embodiments, before extracting features from water conservancy data and obtaining N feature vectors of water conservancy data, feature extraction may be performed on the collected sample data set to obtain a statistical feature sample set, a vocabulary feature sample set, and a context feature sample set of the sample data set; the statistical feature sample set, the vocabulary feature sample set, and the context feature sample set are trained until the loss function of the keyword extraction model meets the loss function requirements. The sample data set may be formed by collecting water conservancy data from different historical time periods.
[0071] Furthermore, the keyword extraction model can be a lightweight neural network model (for example, ShuffleNet, MobileNet, etc.), thereby reducing the demand for computing resources, making the keyword extraction process more efficient, and solving the problem of low efficiency when using deep learning for keyword extraction in the prior art.
[0072] 202: Input the above statistical feature vector, vocabulary feature vector, and context feature vector into the trained keyword extraction model to obtain M candidate keywords for water conservancy data and keyword scores of the M candidate keywords.
[0073] In this step, the above-mentioned statistical feature vector, vocabulary feature vector, and context feature vector are input into the trained keyword extraction model for evaluation, thereby obtaining M candidate keywords for water conservancy data and keyword scores of the M candidate keywords to reflect the importance of different keywords.
[0074] 203: According to the keyword scores of the M candidate keywords, P target keywords with scores greater than a threshold are selected from the M candidate keywords.
[0075] In this step, the scoring threshold can be adaptively adjusted according to different water conservancy data to filter out keywords with low keyword scores and improve the accuracy of keyword extraction.
[0076] In some embodiments, after P target keywords with a score greater than a scoring threshold are screened out from the M candidate keywords, the P target keywords can be arranged in descending order, and G target keywords can be screened out from the P target keywords according to the ranking order requirements, and the G target keywords can be output, thereby further improving the accuracy of keyword extraction.
[0077] In an embodiment of the present application, first, feature extraction is performed on the water conservancy data to obtain N feature vectors of the water conservancy data, which include statistical feature vectors, lexical feature vectors, and contextual feature vectors of the water conservancy data. Feature extraction is performed from different dimensions to comprehensively capture key data in the water conservancy data; then, the statistical feature vectors, lexical feature vectors, and contextual feature vectors are input into a trained keyword extraction model to obtain M candidate keywords for the water conservancy data and keyword scores for each of the M candidate keywords; based on the keyword scores of each of the M candidate keywords, P target keywords that are greater than a score threshold are screened out from the M candidate keywords to improve the accuracy of keyword extraction.
[0078] based on Figure 2 , Figure 3 A logical diagram of keyword extraction provided in an embodiment of the present application. Figure 3 As shown, first, the acquired original water conservancy data is preprocessed by data cleaning, word segmentation, part-of-speech tagging, keyword tagging, stem extraction, etc. to obtain the preprocessed water conservancy data; secondly, feature extraction is performed on the water conservancy data to obtain statistical feature vectors, lexical feature vectors, and context feature vectors; then, the statistical feature vectors, lexical feature vectors, and context feature vectors are input into the trained keyword extraction model to obtain M candidate keywords for the water conservancy data and keyword scores for each of the M candidate keywords; finally, according to the score threshold, P target keywords with a score greater than the score threshold are screened out from the M candidate keywords.
[0079] Based on the same technical concept, an embodiment of the present application also provides a keyword extraction device for water conservancy data, which can implement the above-mentioned keyword extraction method process in the embodiment of the present application.
[0080] Figure 4 Schematic diagram of a structure of a keyword extraction device for water conservancy data provided in an embodiment of the present application. Figure 4 As shown, the device includes: a feature extraction module 401, a keyword extraction module 402; and may further include a preprocessing module 403 and a keyword extraction model construction module 404.
[0081] The feature extraction module 401 is used to extract features from water conservancy data to obtain N feature vectors of the water conservancy data; wherein the N feature vectors include statistical feature vectors, vocabulary feature vectors, and context feature vectors of the water conservancy data, and N is an integer greater than 2.
[0082] The keyword extraction module 402 is used to input the statistical feature vector, the vocabulary feature vector, and the context feature vector into the trained keyword extraction model to obtain M candidate keywords for the water conservancy data and keyword scores of each of the M candidate keywords; wherein M is an integer greater than 0; and is used to screen out P target keywords with a score greater than a threshold from the M candidate keywords based on the keyword scores of each of the M candidate keywords.
[0083] Preprocessing module 403; used to remove noise data in the original water conservancy data to obtain the water conservancy data: wherein the noise data at least includes HTML tags, special characters, and stop words; or use a word segmenter to divide the original water conservancy data to obtain the water conservancy data; or add keyword tags to the key data in the original water conservancy data to obtain the water conservancy data; or perform part-of-speech tagging on all data in the original water conservancy data to obtain the water conservancy data.
[0084] The keyword extraction model construction module 404 is used to extract features from the collected sample data set to obtain a statistical feature sample set, a lexical feature sample set, and a context feature sample set of the sample data set; and train the statistical feature sample set, the lexical feature sample set, and the context feature sample set until the loss function of the keyword extraction model meets the loss function requirements.
[0085] In some embodiments, the keyword extraction module 402 is further used to arrange the P target keywords in descending order, filter out G target keywords from the P target keywords according to the ranking order requirement, and output the G target keywords.
[0086] It should be noted here that the above-mentioned device provided in the embodiment of the present application can implement all the method steps in the above-mentioned method embodiment and can achieve the same technical effect. The parts and beneficial effects of this embodiment that are the same as the method embodiment will not be described in detail here.
[0087] Based on the same technical concept, an electronic device is also provided in an embodiment of the present application, and the electronic device can realize the function of the aforementioned keyword extraction device for water conservancy data.
[0088] Figure 5 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application.
[0089] At least one processor 501, and a memory 502 connected to the at least one processor 501. The specific connection medium between the processor 501 and the memory 502 is not limited in the embodiment of the present application. Figure 5 In the example, the processor 501 and the memory 502 are connected via a bus 500. The bus 500 is Figure 5 The connections between other components are shown in bold lines, and are not intended to be limiting. The bus 500 can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 5 Only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus. Alternatively, the processor 501 can also be called a controller, and there is no limitation on the name.
[0090] In the embodiment of the present application, the memory 502 stores instructions that can be executed by at least one processor 501. The at least one processor 501 can execute the keyword extraction method of water conservancy data discussed above by executing the instructions stored in the memory 502. The processor 501 can implement Figure 4 The functions of each module in the device shown.
[0091] Among them, the processor 501 is the control center of the device, and can use various interfaces and lines to connect the various parts of the entire control device. By running or executing instructions stored in the memory 502 and calling data stored in the memory 502, the various functions of the device and processing data, the device can be monitored as a whole.
[0092] In the embodiment of the present application, the processor 501 may include one or more processing units, and the processor 501 may integrate an application processor and a modem processor, wherein the application processor mainly processes an operating system, a user interface, and application programs, and the modem processor mainly processes wireless communications. It is understandable that the modem processor may not be integrated into the processor 501. In some embodiments, the processor 501 and the memory 502 may be implemented on the same chip, and in some embodiments, they may also be implemented separately on separate chips.
[0093] The processor 501 may be a general-purpose processor, such as a central processing unit (CPU), a digital signal processor, an application-specific integrated circuit, a field programmable gate array or other programmable logic device, a discrete gate or transistor logic device, or a discrete hardware component, and may implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of the present application. A general-purpose processor may be a microprocessor or any conventional processor, etc. The steps of the keyword extraction method for water conservancy data disclosed in the embodiments of the present application may be directly embodied as being executed by a hardware processor, or may be executed by a combination of hardware and software modules in the processor.
[0094] The memory 502 is a non-volatile computer-readable storage medium that can be used to store non-volatile software programs, non-volatile computer executable programs and modules. The memory 502 may include at least one type of storage medium, such as a flash memory, a hard disk, a multimedia card, a card-type memory, a random access memory (Random Access Memory, RAM), a static random access memory (Static Random Access Memory, SRAM), a programmable read-only memory (Programmable Read Only Memory, PROM), a read-only memory (Read Only Memory, ROM), an electrically erasable programmable read-only memory (Electrically Erasable Programmable Read-Only Memory, EEPROM), a magnetic memory, a disk, an optical disk, etc. The memory 502 is any other medium that can be used to carry or store a desired program code in the form of an instruction or data structure and can be accessed by a computer, but is not limited thereto. The memory 502 in the embodiment of the present application can also be a circuit or any other device that can realize a storage function, for storing program instructions and / or data.
[0095] By programming the processor 501, the code corresponding to the keyword extraction method of water conservancy data introduced in the above embodiment can be fixed into the chip, so that the chip can execute the code when running. Figure 2 The embodiment shown is a method for extracting keywords from water conservancy data. How to design and program the processor 501 is a technique known to those skilled in the art and will not be described in detail here.
[0096] It should be noted here that the above-mentioned electronic device provided in the embodiment of the present application can implement all the method steps implemented in the above-mentioned method embodiment, and can achieve the same technical effect. The parts and beneficial effects of this embodiment that are the same as the method embodiment will not be described in detail here.
[0097] Based on the same technical concept, an embodiment of the present application provides a computer storage medium, the computer storage medium includes: computer program code, when the computer program code is executed on a computer, the computer executes any one of the methods for extracting keywords from water conservancy data discussed above. Since the principle of solving the problem by the above computer storage medium is similar to that of a method for extracting keywords from water conservancy data, the implementation of the above computer storage medium can refer to the implementation of the method, and the repeated parts will not be repeated.
[0098] In the specific implementation process, the computer storage medium may include: Universal Serial Bus Flash Drive (USB), mobile hard disk, read-only memory (ROM), random access memory (RAM), magnetic disk or optical disk, and other storage media that can store program codes.
[0099] Based on the same technical concept, the embodiment of the present application also provides a computer program product, which includes: computer program code, when the computer program code is run on a computer, the computer executes any one of the methods for extracting keywords from water conservancy data discussed above. Since the principle of solving the problem by the above computer program product is similar to that of a method for extracting keywords from water conservancy data, the implementation of the above computer program product can refer to the implementation of the method, and the repeated parts will not be repeated.
[0100] The computer program product may employ any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination thereof. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more conductors, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof.
[0101] The method in the present application can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented by software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer programs or instructions. When the computer program or instruction is loaded and executed on a computer, the process or function described in the present application is executed in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, a network device, a user device, a core network device, an OAM or other programmable device.
[0102] The computer program or instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer program or instructions may be transmitted from one website, computer, server or data center to another website, computer, server or data center by wired or wireless means. The computer-readable storage medium may be any available medium that can be accessed by a computer or a data storage device such as a server or data center that integrates one or more available media. The available medium may be a magnetic medium, such as a floppy disk, a hard disk, or a magnetic tape; it may also be an optical medium, such as a digital video disk; it may also be a semiconductor medium, such as a solid-state drive. The computer-readable storage medium may be a volatile or non-volatile storage medium, or may include both volatile and non-volatile types of storage media.
[0103] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage and optical storage, etc.) that contain computer-usable program code.
[0104] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1These computer program instructions can also be stored in a computer-readable memory that can guide a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device, which implements the instructions in the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.
[0105] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.
[0106] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.
Claims
1. A keyword extraction method for water conservancy data, characterized in that: include: Performing feature extraction on the water conservancy data to obtain N feature vectors of the water conservancy data; wherein the N feature vectors include a statistical feature vector, a vocabulary feature vector, and a context feature vector of the water conservancy data, and N is an integer greater than 2; Inputting the statistical feature vector, the vocabulary feature vector, and the context feature vector into a trained keyword extraction model to obtain M candidate keywords of the water conservancy data and keyword scores of each of the M candidate keywords; wherein M is an integer greater than 0; According to the keyword scores of the M candidate keywords, P target keywords with scores greater than a threshold are screened out from the M candidate keywords.
2. The method according to claim 1, characterized in that Before extracting features from the water conservancy data to obtain N feature vectors of the water conservancy data, the method further includes: Removing noise data from the original water conservancy data to obtain the water conservancy data: wherein the noise data at least includes HTML tags, special characters, and stop words; or Using a word segmenter to segment the original water conservancy data to obtain the water conservancy data; or Adding keyword tags to key data in the original water conservancy data to obtain the water conservancy data; or Part-of-speech tagging is performed on all data in the original water conservancy data to obtain the water conservancy data.
3. The method according to claim 1, characterized in that Before extracting features from the water conservancy data to obtain N feature vectors of the water conservancy data, the method further includes: Performing feature extraction on the collected sample data set to obtain a statistical feature sample set, a lexical feature sample set, and a context feature sample set of the sample data set; The statistical feature sample set, the vocabulary feature sample set, and the context feature sample set are trained until the loss function of the keyword extraction model meets the loss function requirement.
4. The method according to any one of claims 1 to 3, characterized in that: The keyword extraction model is a lightweight neural network model.
5. The method according to claim 1, characterized in that After selecting P target keywords with scores greater than the score threshold from the M candidate keywords, the method further includes: The P target keywords are arranged in descending order, G target keywords are screened out from the P target keywords according to the ranking order requirement, and the G target keywords are output.
6. A keyword extraction device for water conservancy data, characterized in that: include: A feature extraction module, used for performing feature extraction on the water conservancy data to obtain N feature vectors of the water conservancy data; wherein the N feature vectors include a statistical feature vector, a vocabulary feature vector, and a context feature vector of the water conservancy data, and N is an integer greater than 2; A keyword extraction module is used to input the statistical feature vector, the vocabulary feature vector, and the context feature vector into a trained keyword extraction model to obtain M candidate keywords for the water conservancy data and keyword scores for each of the M candidate keywords; wherein M is an integer greater than 0; and is used to screen out P target keywords with a score greater than a threshold from the M candidate keywords based on the keyword scores of each of the M candidate keywords.
7. The device according to claim 6, characterized in that The device also includes: A preprocessing module; used to remove noise data from the original water conservancy data to obtain the water conservancy data: wherein the noise data at least includes HTML tags, special characters, and stop words; or Using a word segmenter to segment the original water conservancy data to obtain the water conservancy data; or Adding keyword tags to key data in the original water conservancy data to obtain the water conservancy data; or Part-of-speech tagging is performed on all data in the original water conservancy data to obtain the water conservancy data.
8. The device according to claim 6, characterized in that The device also includes: A keyword extraction model building module is used to extract features from the collected sample data set to obtain a statistical feature sample set, a vocabulary feature sample set, and a context feature sample set of the sample data set; The statistical feature sample set, the vocabulary feature sample set, and the context feature sample set are trained until the loss function of the keyword extraction model meets the loss function requirement.
9. An electronic device, characterized in that: include: Memory, used to store computer programs; A processor, configured to implement the method according to any one of claims 1 to 5 when executing a computer program stored in the memory.
10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 5 is implemented.