A method and system for pushing scientific and technological intelligence information

Abstract

The present invention relates to the field of artificial intelligence technology, and in particular to a method, module and system for pushing scientific and technological intelligence information. The method and system for pushing scientific and technological intelligence information of the present invention utilize a ranking recommendation algorithm to intelligently screen, evaluate and sort the collected information, and put the information that users are more concerned about in the front, thereby helping users to screen out the most popular, most relevant and most timely information from a large amount of news information, and providing users with more efficient, more accurate and more professional energy science and technology intelligence services.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a method, module and system for pushing scientific and technological intelligence information. Background Art

[0002] In the era of 3060 dual carbon transformation, new energy technologies continue to emerge, the technological innovation cycle continues to shorten, the transformation of achievements continues to accelerate, and the amount of relevant technical information increases rapidly, showing an increasingly complex, multi-change, and fragmented trend. Traditional manual information collection and management methods can no longer meet the needs of innovation and development. It is urgent to create a unique and leading scientific and technological intelligence information intelligent push platform in the energy field. Summary of the invention

[0003] Therefore, the technical problem to be solved by the present invention is to overcome the problem that traditional manual information collection and management in the prior art cannot meet the development needs.

[0004] In order to solve the above technical problems, the present invention provides a method for pushing scientific and technological intelligence information, comprising:

[0005] Counting business keywords based on previous daily reports, and assigning weights to the business keywords based on the occurrence frequency and information relevance of the business keywords;

[0006] Get the information to be pushed on that day;

[0007] According to the appearance frequency, publishing frequency and information volume of the websites corresponding to the information to be pushed on that day, a relevance level is set for the information to be pushed on that day;

[0008] Scoring the information to be pushed on that day according to the business keyword weight and the relevance level;

[0009] The information to be pushed on that day is sorted according to the scoring results and pushed.

[0010] Preferably, the scoring of the information to be pushed on the day according to the business keyword weight and the relevance level includes:

[0011] Using the BM25 algorithm to normalize the information to be pushed on that day;

[0012] Use the TF-IDF algorithm to calculate the TD-IDF value of each word in the information to be pushed on that day after normalization;

[0013] The information to be pushed on that day is scored according to the TD-IDF value, the business keyword weight and the relevance level.

[0014] Preferably, the calculation formula for scoring the information to be pushed on the day according to the TD-IDF value, the business keyword weight and the relevance level is as follows:

[0015] S=(a1*b1+...+a n *b n )*c

[0016] Among them, S represents the final score, a n Indicates the TF-IDF value of the word segmentation, b n represents the business keyword weight corresponding to the segmentation, c represents the relevance level, n=1...n, and n represents the total number of segmentations.

[0017] The present invention also provides a scientific and technological intelligence information push module, including:

[0018] A weight allocation unit, used to count business keywords based on previous daily reports, and to allocate weights to the business keywords based on the occurrence frequency and information relevance of the business keywords;

[0019] A data acquisition unit is used to acquire information to be pushed on that day;

[0020] A relevance setting unit, used to set a relevance level for the information to be pushed on the day according to the appearance frequency, publishing frequency and amount of information published on the website corresponding to the information to be pushed on the day;

[0021] A scoring unit, used for scoring the information to be pushed on the day according to the business keyword weight and the relevance level;

[0022] The information push unit is used to sort the information to be pushed on the day according to the scoring results and push it.

[0023] The present invention also provides a scientific and technological intelligence information push system, which is characterized by comprising:

[0024] The data collection module is used to crawl and collect data sources regularly or in real time, and crawl relevant information containing target keywords from the website corresponding to the target journal;

[0025] The data optimization module is used to remove duplicate data, screen data, filter data and clean data to obtain the information to be pushed;

[0026] For example, the science and technology intelligence information push module mentioned above is used to push information to users.

[0027] Preferably, crawling relevant information containing target keywords from the website corresponding to the target journal includes:

[0028] For websites that are strictly prohibited from crawling, find data source websites with a similarity greater than a preset threshold and crawl them after replacement;

[0029] For websites that have robots protocol requirements but are not prohibited from crawling, reduce the crawling frequency;

[0030] For websites without robots protocol requirements, crawling is performed normally.

[0031] Preferably, the scientific and technological intelligence information push system also includes:

[0032] The data resource library module is used to store and manage energy technology intelligence data processed by the data optimization module.

[0033] Preferably, the scientific and technological intelligence information push system also includes:

[0034] The keyword library module is used to store keywords about various sub-fields, levels and aspects of energy science and technology intelligence;

[0035] The filter word library module is used to store various filter words that are irrelevant to energy technology intelligence and may interfere with it.

[0036] Preferably, the data deduplication, data screening, data filtering and data cleaning of the relevant information includes:

[0037] Calculate the cosine similarity between any two related information, and perform data deduplication based on the cosine similarity;

[0038] The keyword library module is used to screen and annotate the relevant information after deduplication, and the filtering vocabulary module is used to filter and clean the data.

[0039] Preferably, the scientific and technological intelligence information push system also includes:

[0040] The word library management module is used to maintain, store and manage the information of the keyword library module and the filtering word library module.

[0041] The above technical solution of the present invention has the following advantages compared with the prior art:

[0042] The technology intelligence information push method and system described in the present invention utilizes a ranking recommendation algorithm to intelligently screen, evaluate and sort the collected information, and puts the information that users are more concerned about in the front, thereby helping users to filter out the most popular, most relevant and most timely information from massive news information, and providing users with more efficient, more accurate and more professional energy technology intelligence services. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to make the content of the present invention more clearly understood, the present invention is further described in detail below according to specific embodiments of the present invention in conjunction with the accompanying drawings, wherein:

[0044] Figure 1 It is a flow chart of the implementation of a method for pushing scientific and technological intelligence information provided by the present invention;

[0045] Figure 2 It is a flow chart of a method for pushing scientific and technological intelligence information provided by an embodiment of the present invention;

[0046] Figure 3 It is a flow chart of a method for pushing scientific and technological intelligence information in a specific application scenario provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0047] The core of the present invention is to provide a method, module and system for pushing scientific and technological intelligence information, which effectively provides intelligent services and helps users quickly obtain effective information.

[0048] In order to enable those skilled in the art to better understand the scheme of the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings and specific implementation methods. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0049] Please refer to Figure 1 and Figure 2 , Figure 1 This is a flow chart of a method for pushing scientific and technological intelligence information provided by the present invention. Figure 2 A schematic diagram of a flow chart of a method for pushing scientific and technological intelligence information provided by an embodiment of the present invention; the specific operation steps are as follows:

[0050] S101: Counting business keywords based on previous daily reports, and assigning weights to the business keywords based on the occurrence frequency and information relevance of the business keywords;

[0051] S102: Obtain the information to be pushed on that day;

[0052] S103: setting a relevance level for the information to be pushed on the day according to the appearance frequency, publishing frequency and information volume of the website corresponding to the information to be pushed on the day;

[0053] S104: scoring the information to be pushed on the day according to the business keyword weight and the relevance level;

[0054] S105: Sort the information to be pushed on that day according to the scoring results and push it.

[0055] Based on the above embodiments, this embodiment describes step S104 in detail:

[0056] The scoring of the information to be pushed on the day according to the business keyword weight and the relevance level includes:

[0057] The information to be pushed on that day is normalized using the BM25 algorithm;

[0058] Use the TF-IDF algorithm to calculate the TD-IDF value of each word in the information to be pushed on that day after normalization;

[0059] The information to be pushed on that day is scored according to the TD-IDF value, the business keyword weight and the relevance level:

[0060] S=(a1*b1+...+a n *b n )*c

[0061] Among them, S represents the final score, a n Indicates the TF-IDF value of the word segmentation, b n represents the business keyword weight corresponding to the segmentation, c represents the relevance level, n=1...n, and n represents the total number of segmentations.

[0062] The present invention innovatively fully couples "machine learning" with "expert analysis", summarizes historical daily content and high-quality source content, performs statistical screening after word segmentation, and then selects after review by business experts. At the same time, business keywords are given weights by considering factors such as frequency of occurrence in daily reports and business relevance.

[0063] First, count the keywords in the previous daily reports, such as "segmentation", "statistics", "word frequency", etc. Then, based on the experience of business experts, weight the keywords according to the frequency of occurrence, information relevance, etc. For example:

[0064]

[0065] Secondly, business experts judge and score the source websites of the information to determine the relevance level of energy technology (the relevance ranges from 1 to 10, representing a degree of relevance to the business ranging from distant to close).

[0066] The relevance level is mainly scored based on the frequency of occurrence, frequency of publication, and amount of information published. For example:

[0067]

[0068] Finally, the ranking combines the TF-IDF algorithm and the BM25 algorithm to score each piece of news, and sorts all the news according to the scoring results. At the same time, statistics can be collected based on the information sources adopted during the compilation of the daily report, and the relevance level and the weight level of business keywords can be dynamically adjusted.

[0069] The present invention first analyzes daily newspapers and annotated news, automatically segments and counts news titles, sets high-frequency words as sorting keywords, and assigns weights; secondly, business experts judge and score the source websites of the information to determine the relevance level of energy technology (the relevance ranges from 1 to 9, representing a relationship from distant to close to the business); finally, the ranking combines the TF-IDF algorithm and the BM25 algorithm to score each news article, and sorts all the news according to the scoring results.

[0070] Use high-frequency words as ranking keywords: By analyzing daily newspapers and annotated news, the news titles are automatically segmented and counted, and high-frequency words are set as ranking keywords and given weights. This can effectively capture hot topics and key information in the field of energy technology, and improve the relevance and value of information ranking.

[0071] Use source websites to score relevance: Business experts judge and score the source websites of information to determine the relevance level of energy technology. This can effectively distinguish the authority and professionalism of different source websites in the field of energy technology, and improve the accuracy and credibility of information sorting.

[0072] Comprehensive scoring using TF-IDF algorithm and BM25 algorithm: Score each news article using TF-IDF algorithm and BM25 algorithm, and sort all news articles according to their scores. This can effectively comprehensively consider factors such as keyword frequency, document length, document frequency, etc. of news information, and improve the efficiency and quality of information sorting.

[0073] The embodiment of the present invention also provides a science and technology intelligence information push module, including:

[0074] A weight allocation unit, used to count business keywords based on previous daily reports, and to allocate weights to the business keywords based on the occurrence frequency and information relevance of the business keywords;

[0075] A data acquisition unit is used to acquire information to be pushed on that day;

[0076] A relevance setting unit, used to set a relevance level for the information to be pushed on the day according to the appearance frequency, publishing frequency and amount of information published on the website corresponding to the information to be pushed on the day;

[0077] A scoring unit, used for scoring the information to be pushed on the day according to the business keyword weight and the relevance level;

[0078] The information push unit is used to sort the information to be pushed on the day according to the scoring results and push it.

[0079] The present invention scores each piece of news information based on different factors to reflect its relevance, popularity, value and other dimensions. These factors include the keyword frequency of the news information, the relevance of the source website, the TF-IDF algorithm and the BM25 algorithm, etc. The module is also responsible for dynamically adjusting and weighing the importance of these factors according to different user groups and scenarios to improve the flexibility and adaptability of information scoring. Each piece of news information is sorted according to the score to achieve recommended sorting. According to different user groups and scenarios, the sorting rules are dynamically adjusted and optimized to improve the accuracy and personalization of information sorting.

[0080] The present invention also provides a scientific and technological intelligence information push system, comprising:

[0081] The data collection module is used to crawl and collect data sources regularly or in real time, and crawl relevant information containing target keywords from the website corresponding to the target journal;

[0082] The data optimization module is used to remove duplicate data, screen data, filter data and clean data to obtain the information to be pushed;

[0083] For example, the science and technology intelligence information push module mentioned above is used to push information to users.

[0084] Based on the above embodiments, the data collection module is responsible for periodically or in real time capturing and collecting data sources based on the information provided by the data source management module to obtain news, papers, patents and other data related to energy technology intelligence. The module is also responsible for reasonably controlling and adjusting the frequency and scope of data collection according to the robot protocol to avoid legal risks and ban risks:

[0085] For websites that are strictly prohibited from crawling, replace the data source website and search for similar data source websites to obtain relevant data;

[0086] For websites that have robots requirements but are not prohibited from crawling, relevant legal risks can be avoided by reasonably reducing the crawling frequency and refining the crawling sections. For example, the crawling frequency of journals such as "Electric Power Engineering Technology" is reduced to once a week;

[0087] For websites without robots requirements, we strictly ensure that data is stored normally.

[0088] Based on the above embodiments, the scientific and technological intelligence information push system further includes:

[0089] The data resource library module is used to store and manage the energy technology intelligence data processed by the data optimization module; this module is also responsible for providing query, update, backup and other functions of the data resource library to ensure the stability and availability of the data resource library.

[0090] Based on the above embodiments, the scientific and technological intelligence information push system further includes:

[0091] The keyword library module is used to store keywords about various sub-fields, levels and aspects of energy science and technology intelligence; it is used to screen and annotate data related to energy science and technology intelligence. And according to the development dynamics and changing trends in the field of energy science and technology, the keyword library is updated and expanded regularly or in real time to ensure the comprehensiveness and timeliness of the keyword library.

[0092] The filter word library module is used to store various filter words that are irrelevant to energy technology intelligence and will interfere with it. It is used to filter and clean data that is irrelevant or repeated to energy technology intelligence. According to the characteristics and needs of the energy technology field, the filter word library is updated and optimized regularly or in real time to ensure the accuracy and effectiveness of the filter word library.

[0093] The word library management module is used to maintain, store and manage the information of the keyword library module and the filter word library module. Including the name, type, weight, etc. of the keyword and filter word. Support users to adjust the relevant keywords and filter words according to actual business needs.

[0094] Based on the embodiment, the data deduplication, data screening, data filtering and data cleaning of the relevant information include:

[0095] Calculate the cosine similarity between any two related information, and perform data deduplication based on the cosine similarity;

[0096] The keyword library module is used to screen and annotate the relevant information after deduplication, and the filtering vocabulary module is used to filter and clean the data.

[0097] The energy technology information acquisition and briefing generation system can automatically capture news and information related to energy technology from hundreds of websites around the world every day, and classify and filter them according to users' keyword tags to provide users with the latest, most comprehensive and most valuable energy technology information.

[0098] However, due to the fact that different websites report the same event or topic repeatedly or with high similarity, there is a lot of redundant and low-quality information in the information data in the system, which affects the user's reading experience and information acquisition efficiency. Therefore, we need to deduplicate the information data, remove duplicate or highly similar information, retain the best quality information, and avoid information loss.

[0099] Cosine Similarity is a text similarity calculation method based on the Vector Space Model. Its basic idea is to represent each text as a high-dimensional sparse vector, where each element in the vector corresponds to a word or phrase, and is assigned a weight value, such as term frequency (TF), inverse document frequency (IDF) or TF-IDF. Then the cosine value of the angle between the two vectors is calculated as the similarity between the texts. The closer the cosine value is to 1, the smaller the angle is, and the more similar the two vectors are; the closer the cosine value is to 0, the larger the angle is, and the less similar the two vectors are.

[0100] In one embodiment, we perform a 1:1 comparison between the newly stored information data of the day and all the stored data within 30 days. The text similarity between two articles is calculated using the cosine similarity algorithm. If the calculation result is greater than 0.8, we consider the two articles to be duplicates, and delete the newly stored data. If the calculation result is less than or equal to 0.8, we consider it to be new information, and include it in today's recommendation. "0.8" can be optimized and adjusted according to business needs. When we increase the threshold, it is equivalent to relaxing the deduplication requirements, the possibility of content duplication will increase, and the amount of recommended data will also increase. However, when we reduce the threshold, it is equivalent to increasing the deduplication requirements, the possibility of content duplication will decrease, but at the same time the amount of recommended data will decrease.

[0101] like Figure 3 As shown, in one embodiment, the user sorts out the intelligence information data source that needs to use cloud crawling technology according to business needs. The science and technology intelligence information push system first determines whether the website has robots protocol requirements, that is, determines whether crawling is prohibited. When it allows crawlers to crawl, the science and technology intelligence information system crawls data according to the crawling strategy preset with the user. For example, crawl the keyword "offshore wind power". If there is information related to "offshore wind power" on the website, we will collect it into the database. After entering the database, we compare the article with the articles in the database. When there is no duplicate information. We will include the information in the information to be pushed. After accumulating one day's data to be pushed, we sort the information according to the relevance level and keyword weight, and push it according to the sorting. Users organize relevant daily reports based on the pushed information.

[0102] Obviously, the above embodiments are merely examples for the purpose of clear explanation and are not intended to limit the implementation methods. For those skilled in the art, other different forms of changes or modifications can be made based on the above description. It is not necessary and impossible to list all the implementation methods here. The obvious changes or modifications derived therefrom are still within the scope of protection of the present invention.

Claims

1. A method for pushing scientific and technological intelligence information, characterized in that: include: Counting business keywords based on previous daily reports, and assigning weights to the business keywords based on the occurrence frequency and information relevance of the business keywords; Get the information to be pushed on that day; Scoring the websites corresponding to the information to be pushed on the day, and setting the relevance level, according to the appearance frequency, release frequency and release volume of the websites corresponding to the information to be pushed on the day; Using the BM25 algorithm to normalize the information to be pushed on that day; Use the TF-IDF algorithm to calculate the TD-IDF value of each word in the information to be pushed on that day after normalization; Scoring the information to be pushed on that day according to the TD-IDF value, the business keyword weight and the relevance level; Sort the information to be pushed on that day according to the scoring results and push them; The calculation formula for scoring the information to be pushed on the day according to the TD-IDF value, the business keyword weight and the relevance level is as follows: S=(a1*b1+...+a n *b n )*c Among them, S represents the final score, a n Indicates the TF-IDF value of the word segmentation, b n represents the business keyword weight corresponding to the segmentation, c represents the relevance level, n=1...n, and n represents the total number of segmentations.

2. A science and technology intelligence information push module, characterized in that: include: A weight allocation unit, used to count business keywords based on previous daily reports, and to allocate weights to the business keywords based on the occurrence frequency and information relevance of the business keywords; A data acquisition unit is used to acquire information to be pushed on that day; A relevance setting unit, used to score the websites corresponding to the information to be pushed on the day and set the relevance level according to the appearance frequency, publishing frequency and information volume of the websites corresponding to the information to be pushed on the day; A scoring unit, used to normalize the information to be pushed on the day by using the BM25 algorithm; calculate the TD-IDF value of each word in the normalized information to be pushed on the day by using the TF-IDF algorithm; and score the information to be pushed on the day according to the TD-IDF value, the business keyword weight and the relevance level; The information push unit is used to sort the information to be pushed on that day according to the scoring results and push it; The calculation formula for scoring the information to be pushed on the day according to the TD-IDF value, the business keyword weight and the relevance level is as follows: S=(a1*b1+...+a n *b n )*c Among them, S represents the final score, a n Indicates the TF-IDF value of the word segmentation, b n represents the business keyword weight corresponding to the segmentation, c represents the relevance level, n=1...n, and n represents the total number of segmentations.

3. A scientific and technological intelligence information push system, characterized in that: include: The data collection module is used to crawl and collect data sources regularly or in real time, and crawl relevant information containing target keywords from the website corresponding to the target journal; The data optimization module is used to remove duplicate data, screen data, filter data and clean data to obtain the information to be pushed; The scientific and technological intelligence information push module as described in claim 2 is used to push information to users.

4. The scientific and technological intelligence information push system according to claim 3 is characterized in that: The crawling of relevant information containing target keywords from the website corresponding to the target journal includes: For websites that are strictly prohibited from crawling, find data source websites with a similarity greater than a preset threshold and crawl them after replacement; For websites that have robots protocol requirements but are not prohibited from crawling, reduce the crawling frequency; For websites without robots protocol requirements, crawling is performed normally.

5. The scientific and technological intelligence information push system according to claim 3, characterized in that: The scientific and technological intelligence information push system also includes: The data resource library module is used to store and manage energy technology intelligence data processed by the data optimization module.

6. The scientific and technological intelligence information push system according to claim 3, characterized in that: The scientific and technological intelligence information push system also includes: The keyword library module is used to store keywords about various sub-fields, levels and aspects of energy science and technology intelligence; The filter word library module is used to store various filter words that are irrelevant to energy technology intelligence and may interfere with it.

7. The scientific and technological intelligence information push system according to claim 6, characterized in that: The data deduplication, data screening, data filtering and data cleaning of the relevant information include: Calculate the cosine similarity between any two related information, and perform data deduplication based on the cosine similarity; The keyword library module is used to screen and annotate the relevant information after deduplication, and the filtering vocabulary module is used to filter and clean the data.

8. The scientific and technological intelligence information push system according to claim 6, characterized in that: The scientific and technological intelligence information push system also includes: The word library management module is used to maintain, store and manage the information of the keyword library module and the filtering word library module.

Images