Method and system for intelligently extracting key knowledge of intelligence data
By preprocessing, vectorizing, generating summaries, and constructing knowledge graphs for internet information, the problem of users struggling to quickly filter relevant information is solved, achieving efficient and accurate data utilization.
Patent Information
- Application Number
- CN202410964594.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-18
- Publication Date
- 2024-11-12
AI Technical Summary
Amidst the vast amount of information on the internet, users struggle to quickly and accurately filter out technical intelligence that is closely related to their needs, resulting in low data utilization efficiency and efficiency.
By acquiring information and data from the Internet, we perform preprocessing, vectorization, summary generation, sentiment analysis, and knowledge graph construction, extract keywords and structured data, and then visualize the data.
It enables automatic, intelligent, and real-time filtering of irrelevant noise or low-quality content, and quickly and accurately extracts information closely related to user needs, thereby improving the efficiency and utilization rate of data.
Smart Images

Figure CN118939793A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a method and system for intelligent extraction of key knowledge from intelligence data. Background Art
[0002] Technical intelligence is highly timely and professional, covering the latest scientific research results, technological developments, industry trends, etc. With the rapid development of the Internet and the explosion of information, technical intelligence exists in a large amount of Internet information. Due to the huge amount of information and its dispersion, obtaining and utilizing these valuable data faces huge challenges. The timeliness requirements of technical intelligence are very high. Information such as scientific research results, technological developments and industry trends need to be obtained and processed in a timely manner so that professionals in related fields can respond quickly and track the latest developments. Although Internet information is rich and diverse, it contains a lot of noise and irrelevant information. Technical intelligence with strong professionalism is often hidden in the vast ocean of information. How to efficiently and accurately screen, summarize and summarize key technical intelligence from massive Internet information requires not only powerful natural language processing and machine learning technology support, but also an efficient information filtering and classification system to improve the utilization rate of intelligence, which is of great significance to improving the competitiveness of scientific research and industry.
[0003] CN112667821A discloses a military knowledge graph construction and question-answering method based on encyclopedia data, including step 1: crawling the classified data of the encyclopedia, treating the above-mentioned classified data as concept data, adding the crawled classified data to the graph database, and establishing the subordinate relationship between concepts; step 2: iterative batch reading and processing of encyclopedia data, the encyclopedia data is a collection of encyclopedia entries, extracting multiple expressions of the same entity name in the encyclopedia entries, the multiple expressions of the same entity name are called synonyms, adding these synonyms to the graph database, and establishing the relationship between the entity name and its synonyms; step 3: using ElasticSearch to create a search library for extended entities, in ElasticSearch, creating an index, configuring the Chinese word segmentation engine, and importing all encyclopedia data into In ElasticSearch, create a search library for extended entities; Step 4: parse and process encyclopedia data, extract entity information, add and update synonym data, use rules and text classification methods to filter military entities, add and update military entity information to the graph database, establish the relationship between entities and concepts, process the extended entity information associated with military entity information, search and obtain extended entity information, determine the category, add extended information to the graph database, and establish the relationship between entities and extended entities; Step 5: Get the input entity query or question, parse the query statement, query the synonyms of the question, obtain the entity mention of the question through entity recognition, expand the entity mention, parse out the relationship-related keywords, match the query graph database, calculate the relevance of the question and the entity attribute or relationship word, and return the entity or attribute information with the highest ranking and above the threshold.
[0004] CN115878811A discloses a method for intelligent analysis and deduction of military intelligence based on a causal map. Step 1: Automatic construction of a causal map. The construction process of the causal map includes four parts: data collection and cleaning, event logic extraction, multimodal data fusion, and causal rule learning; 1) Data collection and cleaning: During data collection, it is connected from the data source and high-concurrency real-time collection is performed, and then the collected data is cleaned, including removing irrelevant content through a customized data cleaning model, and converting the text to be processed into the input format data required for further analysis; 2) Event logic extraction: Constructing a Wasserstein generative adversarial network model to analyze the subject matter for the cleaned data. 1) Automatically extract events, attributes and logical relations to form a logical relation embedding vector to mine the knowledge information of events in the real world; 2) Multimodal data fusion: By using a multimodal joint network synthetic representation, the extracted data from different sources, different means and different structures are uniformly represented and managed to obtain a multimodal unified representation network; 3) Event rule learning: Event rule learning is oriented to the multimodal unified representation network, and through certain cognitive reasoning methods, the logical relationship between event network models is mined, so that the network can be reasoned in a human cognitive understanding way to realize the application of event cognitive maps; in the process of event rule learning, for the multimodal unified representation network, Transform is first used to represent the event network. The encoder structure of ormer extracts the embedding vector of the text, uses the CNN method to extract the embedding vector of the image, forms the embedding vector of the event map, and then fuses the logical relationship embedding vector extracted by the event logic to form a multimodal embedding representation, thereby completing the semantic and temporal and spatial associations between heterogeneous network models, realizing the learning and reasoning of event rules, and obtaining a multimodal event map. The multimodal event map contains pictures, events contained in the text, entity information, and the logical relationship between events; Step 2: Event-centered evolutionary inference Based on the constructed multimodal event map, through the review and analysis of historical events, the actual combat data is simulated and deduced in multiple directions to form a simulation deduction model. The simulation deduction process includes battlefield environment The method comprises the following steps: deducing the key elements of the battlefield environment, troop deployment, combat capability, and dynamic targets by combining the event information and the key elements of the battlefield environment, troop deployment, combat capability, and dynamic targets obtained by the deduction process, and introducing the time dimension information to construct a time series event map. On the basis of the time series event map, an event representation model is established according to objective entities and events. The event representation model includes the basic elements of the event and the key elements that affect the development of the event. Then, the historical case fitting results are compared with the actual results to optimize the simulation deduction model, and finally predict different development trends of future events. Among them, event reasoning and prediction are performed by combining the multimodal event map obtained in step one and the time series event map.First, the multimodal event graph is used for reasoning: for a new event, its similarity with each node in the multimodal event graph is calculated, and the node in the multimodal event graph that is most similar to the new event is found; then, the subsequent evolution direction of the new event based on the multimodal event graph is deduced according to the generalized node in the multimodal event graph of the node that is most similar to the new event; after completing the reasoning based on the multimodal event graph, the task of predicting future events is abstracted into the reasoning task of the future state of the time series event graph according to the time series event graph, wherein, by cyclically encoding the historical event information and aggregating the information of adjacent events in the time series event graph, the joint probability distribution of all events is inferred through the aggregated historical events, time series adjacent events and simultaneous events, thereby predicting the probability of future events; finally, for the event reasoning results based on the multimodal event graph and the time series event graph, the final event reasoning results are obtained by probability fusion. ;
[0005] CN115408532A discloses a method for constructing a knowledge graph of weapons and equipment for open source intelligence, which comprises the following steps: obtaining military text data, preprocessing the military text data, and obtaining standardized military text data; determining military text data to be trained based on the standardized military text data, and annotating the military text data to be trained to obtain a weapon and equipment entity recognition data set to be trained, a weapon and equipment attribute extraction data set to be trained, and a weapon and equipment entity link data set to be trained; inputting the weapon and equipment entity recognition data set to be trained into a corresponding model for training processing, and obtaining a trained entity recognition model; inputting the weapon and equipment attribute extraction data set to be trained into a corresponding model for training processing, and obtaining a trained entity recognition model; The method comprises the following steps: first, inputting the weapon and equipment entity link data set to be trained into the corresponding model for training processing to obtain the trained entity link model; performing weapon and equipment recognition and weapon and equipment attribute extraction operations on the standardized military text data based on the entity recognition model and the relationship extraction model to obtain a weapon and equipment entity data set and a corresponding weapon and equipment attribute data set; based on the entity link model, performing entity linking on the weapon and equipment entity data set and the existing entities in the weapon and equipment knowledge graph to obtain a target candidate entity data set, determining a target attribute data set of the weapon and equipment according to the target candidate entity data set and the weapon and equipment attribute data set, storing the target attribute data set in a database to obtain a weapon and equipment knowledge graph.
[0006] CN112307768A discloses an intelligence monitoring method for artificial intelligence technology enterprises, including an intelligence autonomous subscription module: used to describe the enterprise's demand for intelligence monitoring; a technical intelligence automatic collection module: used to improve the accuracy and comprehensiveness of intelligence collection; a technical intelligence monitoring and analysis module: used to automatically collect, screen, clarify and summarize intelligence content according to the enterprise's subscription needs; a semantic web construction module: used to conduct in-depth analysis and mining of the collected intelligence content; the intelligence autonomous subscription module is connected to the semantic web construction module, the semantic web construction module is also connected to the technical intelligence automatic collection module, and the technical intelligence automatic collection module is also connected to the technical intelligence monitoring and analysis module.
[0007] At present, faced with massive amounts of Internet information data, users need to read, analyze and process large amounts of Internet information intelligence. It is difficult to automatically filter out irrelevant or low-quality information, and it is difficult to quickly and accurately obtain information that is highly relevant to their needs, which leads to low data utilization. Summary of the invention
[0008] Long-term practice has found that the speed of Internet information generation and updating is extremely fast. When faced with massive amounts of data, not only does it require high timeliness, but it is also necessary to screen and obtain relevant technical intelligence as soon as possible, and it is also necessary to eliminate noise information and extract effective content. Users often need to spend a lot of time and energy to screen, interpret and process this information. Due to the lack of effective methods to filter irrelevant or low-quality content, it is difficult for users to quickly and accurately find information closely related to their needs, which ultimately leads to low data utilization efficiency and utilization rate.
[0009] In view of this, the present invention aims to propose a method for intelligent extraction of key knowledge from intelligence data, including:
[0010] Step S1, obtaining first information data from the Internet, wherein the first information data at least includes news information, document information, and multimedia information;
[0011] Step S2, performing a data preprocessing process on the first information to obtain second information data and vectorizing the second information data using a bag-of-words model and / or TF-IDF to obtain third information data, and extracting vocabulary from the third information data according to preset tags;
[0012] Step S3, counting the words according to their frequencies of occurrence, sorting them from high to low according to the frequency statistics, and selecting a preset number of words as keywords;
[0013] Step S4, inputting the third information data into the generative language model to simplify and generate summary data;
[0014] Step S5, inputting the third information data into a machine learning model to perform sentiment analysis to obtain text sentiment data;
[0015] Step S6, constructing a knowledge graph based on the third information data, and structuring and visualizing the vocabulary, keywords, summary data, and text sentiment tendency data.
[0016] In one embodiment, the preprocessing process includes converting image data and / or video data into text data.
[0017] In one embodiment, the preprocessing process further includes data cleaning and standardization of words in the text data.
[0018] In one embodiment, the preset tags include a structured tag library in XML format.
[0019] In one embodiment, the third information data is traversed and calculated according to the preset tags, and after the words are extracted, they are classified and stored according to the preset tags.
[0020] In one embodiment, the machine learning model includes a BERT or LSTM model.
[0021] The present invention also provides a system according to the above-mentioned method for intelligent extraction of key knowledge from intelligence data, the system comprising:
[0022] A data acquisition unit, used to obtain first information data from the Internet, wherein the first information data at least includes news information, document information, and multimedia information;
[0023] A preprocessing unit, configured to perform a data preprocessing process on the first information to obtain second information data, perform vectorization using a bag-of-words model and / or TF-IDF to obtain third information data, and extract vocabulary from the third information data according to preset tags;
[0024] A word frequency unit is used to count the frequency of occurrence of the words, sort them from high to low according to the frequency statistics, and select a preset number of words as keywords;
[0025] A summary unit, used for inputting the third information data into the generative language model to simplify and generate summary data;
[0026] A text sentiment analysis unit, used for inputting the third information data into a machine learning model to perform sentiment tendency analysis to obtain text sentiment tendency data;
[0027] The knowledge graph unit is used to construct a knowledge graph based on the third information data, and to structure and visualize vocabulary, keywords, summary data, and text sentiment tendency data.
[0028] In one embodiment, the preprocessing unit includes a standardization module for performing data cleaning and standardization processing on words in the text data.
[0029] The present invention provides an electronic device, at least one processor; and
[0030] a memory communicatively connected to the at least one processor; wherein,
[0031] The memory stores instructions that can be executed by the at least one processor. The instructions are executed by the at least one processor so that the at least one processor can perform the above method.
[0032] The present invention provides a machine-readable storage medium, on which instructions are stored, and the instructions are used to enable a machine to execute the method of the present application as described above.
[0033] The method for intelligent extraction of key knowledge from intelligence data disclosed in the present invention obtains raw data, pre-processes the raw data and then vectorizes it, extracts vocabulary from the vector space according to preset labels, performs frequency statistics on the extracted vocabulary, and obtains keywords; then generates summary data by simplifying it through a generative language model, and performs sentiment analysis through a machine learning model, and finally extracts structured knowledge data of multiple dimensional categories such as vocabulary, keywords, summary data, and text sentiment data by constructing a knowledge graph, and visualizes them. The present invention also discloses a system, which can not only automatically, intelligently, and in real time convert various types of data generated by the Internet into text data, but also processes it through preset labels, generative language models, machine learning models, and knowledge graphs, filters irrelevant noise or low-quality content, quickly simplifies and accurately extracts information closely related to user needs, and thus improves the efficiency and utilization rate of data.
[0034] Other features and advantages of the present invention will be described in detail in the following detailed description. BRIEF DESCRIPTION OF THE DRAWINGS
[0035] The accompanying drawings constituting a part of the present invention are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the accompanying drawings:
[0036] Figure 1 The present invention is a flowchart of a method for intelligent extraction of key knowledge from intelligence data according to an embodiment of the present invention. DETAILED DESCRIPTION
[0037] The specific implementation of the present invention is described in detail below in conjunction with the accompanying drawings. It should be understood that the specific implementation described here is only used to illustrate and explain the present invention, and is not used to limit the present invention.
[0038] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.
[0039] It should be noted that the terms "first", "second", "third", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so as to describe the embodiments of the present invention described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.
[0040] In order to solve the problem that Internet information is generated and updated very quickly as mentioned in the background technology section, when faced with massive amounts of data, not only high timeliness is required, but also the need to screen and obtain relevant technical intelligence as soon as possible, and to eliminate noise information and extract effective content. Users often need to spend a lot of time and energy to screen, interpret and process this information. Due to the lack of effective methods to filter irrelevant or low-quality content, it is difficult for users to quickly and accurately find information closely related to their needs, which ultimately leads to problems such as low data utilization efficiency and utilization rate. The present invention provides a method for intelligent extraction of key knowledge from intelligence data, such as Figure 1 As shown, a flow chart of a method for intelligent extraction of key knowledge from intelligence data according to an embodiment of the present invention is provided. The method for intelligent extraction of key knowledge from intelligence data comprises:
[0041] Step S1, obtaining first information data from the Internet, wherein the first information data at least includes news information, document information, and multimedia information;
[0042] Step S2, performing a data preprocessing process on the first information to obtain second information data and vectorizing the second information data using a bag-of-words model and / or TF-IDF to obtain third information data, and extracting vocabulary from the third information data according to preset tags;
[0043] Step S3, counting the words according to their frequencies of occurrence, sorting them from high to low according to the frequency statistics, and selecting a preset number of words as keywords;
[0044] Step S4, inputting the third information data into the generative language model to simplify and generate summary data;
[0045] Step S5, inputting the third information data into a machine learning model to perform sentiment analysis to obtain text sentiment data;
[0046] Step S6, constructing a knowledge graph based on the third information data, and structuring and visualizing vocabulary, keywords, summary data, and text sentiment tendency data.
[0047] The method disclosed in the present invention for intelligent extraction of key knowledge from intelligence data obtains raw data, pre-processes the raw data and then vectorizes it, extracts vocabulary from the vector space according to preset labels, performs frequency statistics on the extracted vocabulary, and obtains keywords; then generates summary data by simplifying the generative language model, and performs sentiment analysis through the machine learning model, and finally extracts structured knowledge data of multiple dimensional categories such as vocabulary, keywords, summary data, and text sentiment data by constructing a knowledge graph, and visualizes them. This method can not only automatically, intelligently, and in real time convert various types of data generated by the Internet into text data, but also processes it through preset labels, generative language models, machine learning models, and knowledge graphs, filters irrelevant noise or low-quality content, quickly simplifies and accurately extracts information closely related to user needs, and thus improves the efficiency and utilization rate of data.
[0048] In order to reduce the amount of calculation and improve the efficiency of analysis, in a more preferred embodiment of the present invention, the sentiment tendency analysis of text data adopts a method based on sentiment dictionary, and uses a predefined sentiment dictionary to mark the sentiment words in the text. Calculate the number of positive and negative sentiment words in the intelligence document. Judge the sentiment tendency of the document based on the number of positive and negative words. The sentiment tendency display uses a pie chart control for visualization. For example, the sentiment analysis category visualization uses a sentiment analysis chart to give the words and percentages of positive and negative emotions.
[0049] In step S6, the visual display can provide users with a simple, intuitive and fast means of reading intelligence information. The overall structure of the visual display method adopts an intelligent reading companion layout mode. The entire reading page is divided into two parts, the left part is the original text reading area, and the right part is the knowledge display area. The knowledge display area displays the key knowledge of the intelligence document from top to bottom: tags, keywords, abstracts, word frequencies, sentiment tendencies, and knowledge graphs.
[0050] In step S2, the preset tags are clarified. For example, the preset tags can be divided into people, geographic space, organizations, equipment products, and technologies. When the text data is "OpenAI has developed the latest GPT-4 model for natural language processing tasks. The GPT-4 model performs well in multiple benchmarks and surpasses other leading models." Technology: ["model", "natural language processing", "benchmark"], organization: ["openai"]. In the process of text data preprocessing, it is necessary to segment words and remove stop words, convert the text to lowercase, and remove stop words and punctuation. In step S3, a preset number of words is selected, and the preset number is selected from 5 to 20.
[0051] For example, structured preset tags extract at least five categories of knowledge information from text data: (1) People, the names of people involved in the text data. (2) Geographical space, the geographical names of countries, cities, bases, oceans, islands, etc. involved in the text data. (3) Organizations, the names of government organizations, companies, troops, colleges, consulting agencies, etc. involved in the text data. (4) Equipment products, the names or codes of equipment products such as aircraft, aircraft carriers, and logistic equipment involved in the text data. (5) Technology, the key words in the technical field involved in the text data.
[0052] Vocabulary vectorization is to convert text data into numerical vectors so that it can be input into the machine learning model for training and prediction. For example, the Bag of Words (BoW) model, but the Bag of Words model does not consider the order of vocabulary, but only considers the frequency of each word in the text. TF-IDF (Term Frequency-Inverse Document Frequency) not only considers the word frequency, but also considers the inverse document frequency of the vocabulary in the entire corpus to reduce the weight of high-frequency common words. At the same time, in the process of vectorization, word embeddings are also used to map vocabulary to a high-dimensional vector space to retain the semantic information of the vocabulary. Commonly used word embedding methods include Word2Vec, GloVe, and fastText. In the more preferred case of the present invention, the Transformers library is used for BERT vectorization, and pre-trained language models such as BERT capture more complex contextual information and are suitable for more advanced text representation requirements.
[0053] Text data is more structured compared to image and video data, facilitating natural language processing (NLP) and analysis, thereby improving the processing efficiency, readability, and analyzability of data. At the same time, it is also to reduce storage and transmission costs. Through conversion, mature text analysis tools and algorithms can be utilized to extract information. In a preferred embodiment of the present invention, the preprocessing process includes converting image data and / or video data into text data. To identify and extract text from an image, the conversion of image data to text data (OCR technology) includes grayscale conversion, binarization, denoising, skew correction, etc. of the image. A deep learning model, such as a model based on a convolutional neural network (CNN), is used to locate the text regions in the image. The detected text regions are segmented into individual characters, and then each character is recognized using an OCR algorithm. A deep learning model, such as a model based on SSD (Single Shot MultiBox Detector) or YOLO (You Only Look Once), is used to detect the text regions in the image. For the processing of video, the video is decomposed into a series of frames, and then the same OCR processing steps as those for image data are applied to each frame image.
[0054] To effectively clean, denoise, and format text data, providing a high-quality data foundation for subsequent data analysis and model training, in a preferred embodiment of the present invention, the preprocessing process further includes data cleaning and standardization processing of the vocabulary in the text data. Text data cleaning involves removing useless symbols. Using regular expressions or other text processing tools, punctuation marks, special characters, whitespace characters, etc. in the text are removed, as these characters may affect the accuracy of subsequent processing. Similarly, emojis in the text can also be deleted through regular expressions. Data cleaning also includes text data denoising. For example, stop words are removed, common stop words such as "de", "he", "shi", etc. are removed from the text, as these words usually do not carry important information. Stemming or Lemmatization is used to reduce the diversity of vocabulary and improve the consistency of the text. Standardization processing includes the standardization of separators and data format standardization. The comma separator in the text is replaced with a semicolon, or converted to other separators as needed. Data format standardization, for example, unifying date formats, currency units, etc., and format conversions such as JSON, XML, etc., using specialized libraries.
[0055] In XML (Extensible Markup Language), users are allowed to customize tags as needed. In order to better obtain data consistent with user needs, ensure that documents follow certain standards and conventions, better reflect the meaning and context of the data, and make XML documents more intuitive and easy to understand. The preset tag library is an example of XML extensibility, which is defined and adopted according to the needs of specific fields or applications. In a preferred embodiment of the present invention, the preset tags include a structured tag library in XML format. Users define a set of specific tags to represent different data elements based on their business needs and data structure. For example, an e-commerce platform will define <product> 、 <price> 、 <description>In a more preferred embodiment of the present invention, the tag library can be automatically generated. For example, the corresponding XML tags are automatically generated according to the database table structure to map the fields in the database. The automatically generated tags need to follow certain naming rules to ensure that they can accurately reflect the structure and content of the data.
[0056] In order to label the vocabulary in the text data, a label is assigned to each word to clarify the grammatical role of the word in the sentence, such as noun, verb, adjective, etc., so as to help the machine better understand the semantics of the text. By identifying the part of speech of the word, the emotional tendency of the text can be judged more accurately. For example, adjectives and adverbs are usually closely related to emotional expression. In a preferred embodiment of the present invention, the third information data is traversed and calculated according to the preset label, and the vocabulary is extracted and classified and stored according to the preset label. A simple word segmentation function is used to divide the text into words. For example, for Chinese word segmentation, HanLP is used for Chinese word segmentation, a word list is returned, and each word is classified and stored by checking whether it is a preset label.
[0057] The BERT model can capture the deep semantic information of the input text through the bidirectional Transformer architecture and more accurately understand the meaning of words in context. BERT is pre-trained on a large-scale corpus and can learn rich language features, which is very useful for tasks such as sentiment analysis. The LSTM model can capture long-term dependencies in the text through its unique gating mechanism, which is very important for understanding the overall sentiment tendency of the text. In a preferred embodiment of the present invention, the machine learning model includes a BERT or LSTM model. The two models have been trained with a large amount of data and have good generalization capabilities.
[0058] The present invention also provides a system for the above-mentioned method for intelligent extraction of key knowledge from intelligence data, the system comprising:
[0059] A data acquisition unit, used to obtain first information data from the Internet, wherein the first information data at least includes news information, document information, and multimedia information;
[0060] A preprocessing unit, configured to perform a data preprocessing process on the first information to obtain second information data, perform vectorization using a bag-of-words model and / or TF-IDF to obtain third information data, and extract vocabulary from the third information data according to preset tags;
[0061] A word frequency unit is used to count the frequency of occurrence of the words, sort them from high to low according to the frequency statistics, and select a preset number of words as keywords;
[0062] A summary unit, used for inputting the third information data into the generative language model to simplify and generate summary data;
[0063] A text sentiment analysis unit, used for inputting the third information data into a machine learning model to perform sentiment tendency analysis to obtain text sentiment tendency data;
[0064] The knowledge graph unit is used to construct a knowledge graph based on the third information data, and to structure and visualize vocabulary, keywords, summary data, and text sentiment tendency data.
[0065] The system disclosed in the present invention is used for the intelligent extraction of key knowledge from intelligence data. The data acquisition unit obtains raw data, the preprocessing unit preprocesses the raw data and then vectorizes it. The word frequency unit extracts words from the vector space according to preset labels, and extracts the words for frequency statistics to obtain keywords. The summary unit generates summary data by simplifying the generative language model, and the text sentiment analysis unit performs sentiment tendency analysis by the machine learning model. The knowledge graph is constructed by the knowledge graph unit, and finally structured knowledge data of multiple dimensional categories such as vocabulary, keywords, summary data, and text sentiment tendency data can be extracted and visualized. The system can not only automatically, intelligently, and in real time convert various types of data generated by the Internet into text data, but also process it by means of preset labels, generative language models, machine learning models, and knowledge graphs, filter irrelevant noise or low-quality content, quickly simplify and accurately extract information closely related to user needs, and thus improve the efficiency and utilization rate of data.
[0066] In order to better improve data quality and unify data formats, in a preferred embodiment of the present invention, the preprocessing unit includes a standardization module for performing data cleaning and standardization processing on the vocabulary in the text data. The standardization module can remove noise in the text, such as special characters, punctuation marks and irrelevant information, to improve the quality of the data. The cleaned data is purer, which helps the model to extract effective information from it.
[0067] The present invention provides an electronic device, at least one processor; and
[0068] a memory communicatively connected to the at least one processor; wherein,
[0069] The memory stores instructions that can be executed by the at least one processor. The instructions are executed by the at least one processor so that the at least one processor can perform the above method.
[0070] The present invention provides a machine-readable storage medium, on which instructions are stored, and the instructions are used to enable a machine to execute the method of the present application as described above.
[0071] It should be noted that, for the above-mentioned method embodiments, for the sake of simplicity, they are all described as a series of action combinations, but those skilled in the art should know that the present invention is not limited by the described action sequence, because according to the present invention, certain steps can be performed in other sequences or simultaneously. Secondly, those skilled in the art should also know that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present invention.
[0072] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.
[0073] In the several embodiments provided by the present invention, it should be understood that the disclosed device can be implemented in other ways. For example, the device embodiments described above are only schematic, such as the division of the units, which is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, and the indirect coupling or communication connection of devices or units can be electrical or other forms.
[0074] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
[0075] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.
[0076] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a mobile terminal, a server or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, disk or optical disk and other media that can store program codes.
[0077] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.< / description> < / price> < / product>
Claims
1. A method for intelligent extraction of key knowledge from intelligence data, characterized in that: The method for intelligent extraction of key knowledge from intelligence data includes: Step S1, obtaining first information data from the Internet, wherein the first information data at least includes news information, document information, and multimedia information; Step S2, performing a data preprocessing process on the first information to obtain second information data and vectorizing the second information data using a bag-of-words model and / or TF-IDF to obtain third information data, and extracting vocabulary from the third information data according to preset tags; Step S3, counting the words according to their frequencies of occurrence, sorting them from high to low according to the frequency statistics, and selecting a preset number of words as keywords; Step S4, inputting the third information data into the generative language model to simplify and generate summary data; Step S5, inputting the third information data into a machine learning model to perform sentiment analysis to obtain text sentiment data; Step S6, constructing a knowledge graph based on the third information data, and structuring and visualizing the vocabulary, keywords, summary data, and text sentiment tendency data.
2. The method for intelligent extraction of key knowledge from intelligence data according to claim 1, characterized in that: The preprocessing process includes converting image data and / or video data into text data.
3. The method for intelligent extraction of key knowledge from intelligence data according to claim 2, characterized in that: The preprocessing process also includes data cleaning and standardization of words in the text data.
4. The method for intelligent extraction of key knowledge from intelligence data according to claim 1, characterized in that: The preset tags include a structured tag library in XML format.
5. The method for intelligent extraction of key knowledge from intelligence data according to claim 4 is characterized in that: The third information data is traversed and calculated according to the preset tags, and after the words are extracted, they are classified and stored according to the preset tags.
6. The method for intelligent extraction of key knowledge from intelligence data according to any one of claims 1 to 5, characterized in that: The machine learning model includes a BERT or LSTM model.
7. A system according to any one of claims 1 to 6, characterized in that: The system comprises, A data acquisition unit, used to obtain first information data from the Internet, wherein the first information data at least includes news information, document information, and multimedia information; A preprocessing unit, configured to perform a data preprocessing process on the first information to obtain second information data, perform vectorization using a bag-of-words model and / or TF-IDF to obtain third information data, and extract vocabulary from the third information data according to preset tags; A word frequency unit is used to count the frequency of occurrence of the words, sort them from high to low according to the frequency statistics, and select a preset number of words as keywords; A summary unit, used for inputting the third information data into the generative language model to simplify and generate summary data; A text sentiment analysis unit, used for inputting the third information data into a machine learning model to perform sentiment tendency analysis to obtain text sentiment tendency data; The knowledge graph unit is used to construct a knowledge graph based on the third information data, and to structure and visualize vocabulary, keywords, summary data, and text sentiment tendency data.
8. The system according to claim 7, characterized in that The preprocessing unit includes a standardization module for performing data cleaning and standardization processing on words in text data.
9. An electronic device, characterized in that: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can perform the method according to any one of claims 1 to 6.
10. A machine-readable storage medium, characterized in that: The machine-readable storage medium stores instructions for enabling a machine to execute the method described in any one of claims 1 to 6 of the present application.
Citation Information
Cited By
Intelligent printing and typesetting method and device based on machine learning
CN119885902A
