Text data processing method based on word frequency fluctuation and semantic analysis and electronic equipment

Abstract

The invention provides a text data processing method based on word frequency fluctuation and semantic analysis and electronic equipment, and can be applied to the technical field of big data. The method comprises the steps that multiple pieces of text data belonging to a target field in a preset time period are obtained, and the preset time period comprises multiple continuous sub time periods; extracting a plurality of target vocabularies from the plurality of pieces of text data; determining word frequency information of each target vocabulary in each sub-time period in the plurality of sub-time periods; according to the word frequency information, determining a plurality of hotspot vocabularies from the plurality of target vocabularies, the hotspot vocabularies being used for representing technical development hotspots of the target domain in a preset time period; and performing semantic analysis on the plurality of hot vocabularies to obtain a plurality of optimized hot vocabularies.

Description

Technical Field

[0001] The present invention relates to the technical field of big data, and in particular to a text data processing method and electronic device based on word frequency fluctuation and semantic analysis. Background Art

[0002] The technology development route is very important for the technology roadmap selection (or concept solution selection) of technology-based enterprises, especially in unfamiliar fields, which is related to the success or failure of the company's investment and the direction of investment. If the investment direction is right, you only need to improve the product in the future to stand on the main channel and not be eliminated by the market. For technology consumer companies, such as the telecommunications industry, the choice of technology route is also crucial. Although the right or wrong choice of technology route cannot directly affect the success or failure of technology companies, choosing the wrong route increases the cost of subsequent migration, transformation, and upgrading.

[0003] In related technologies, the following methods can be used to find hot spots and routes of technological development in a certain field: reading a large number of documents and finding hot spots and routes of technological development through the documents; or reading technical review articles to understand the hot spots and routes of technological development in a certain field; or using a large language model to analyze hot spots and routes of technological development from a large number of documents. However, the method of reading documents or review articles not only has the problems of long reading time, high entry threshold, low analysis efficiency, and high labor cost, but also the determined hot spots and routes of technological development are lagging and easily affected by the subjective views of the analyst. In addition, the current method of using large language models for analysis is still immature, the results of the analysis are not satisfactory, and the large language model cannot currently avoid the phenomenon of "hallucination" output.

[0004] In summary, there are currently two main problems in determining technology development hotspots and trends: first, how to solve the problem of timeliness; second, how to ensure the accuracy of hotspots. Summary of the invention

[0005] In view of the above problems, the present invention provides a text data processing method and electronic device based on word frequency fluctuation and semantic analysis.

[0006] According to a first aspect of the present invention, there is provided a text data processing method based on word frequency fluctuation and semantic analysis, comprising: acquiring a plurality of text data belonging to a target domain within a preset time period, wherein the preset time period includes a plurality of continuous sub-time periods; extracting a plurality of target words from the plurality of text data; determining the word frequency information of each target word in each of the plurality of sub-time periods; determining a plurality of hot words from the plurality of target words based on the word frequency information, wherein the hot words are used to characterize the technology development hotspots of the target domain within the preset time period; and performing semantic analysis on the plurality of hot words to obtain a plurality of optimized hot words.

[0007] According to an embodiment of the present invention, multiple hot words are determined from multiple target words based on word frequency information, including: obtaining total word frequency information of each target word in a preset time period; determining high-frequency words from multiple target words based on the total word frequency information; determining the word frequency fluctuation trend of multiple medium-frequency and high-frequency words based on the word frequency information of each of the multiple high-frequency words in multiple sub-time periods; and determining multiple hot words from multiple high-frequency words based on the word frequency fluctuation trend.

[0008] According to an embodiment of the present invention, the word frequency fluctuation trend includes continuous decrease, continuous increase and continuous fluctuation. According to the word frequency fluctuation trend, multiple hot words are determined from multiple high-frequency words, including: deleting the medium-frequency words whose word frequency fluctuation trend is continuously decreasing among the multiple high-frequency words, to obtain multiple hot words.

[0009] According to an embodiment of the present invention, determining the word frequency fluctuation trend of multiple high-frequency words according to the word frequency information of each of the multiple high-frequency words in multiple sub-time periods includes: for each sub-time period, normalizing the word frequency information of each high-frequency word according to the word frequency information of the multiple high-frequency words in the sub-time period to obtain the normalized word frequency information of each high-frequency word in each sub-time period; for each high-frequency word frequency, determining the word frequency fluctuation trend of each high-frequency word according to the normalized word frequency information of the high-frequency word in multiple sub-time periods.

[0010] According to an embodiment of the present invention, after determining the word frequency information of each target word in multiple sub-time periods, it also includes: for two adjacent sub-time periods, sorting multiple co-frequency words in the previous sub-time period according to the word frequency information of the later sub-time period to determine the change trend among the multiple co-frequency words; based on the first word frequency information of each of the multiple target words in the earliest sub-time period, arranging the multiple target words in descending order or ascending order to obtain first word sequence information; according to the change trend among the multiple co-frequency words, arranging the multiple co-frequency words in the multiple target words in descending order or ascending order to obtain second word sequence information, wherein the multiple target words and the multiple co-frequency words are all arranged in descending order or in ascending order.

[0011] According to an embodiment of the present invention, the method further includes: performing a visualization operation on the plurality of target words based on the second word order information by a visualization tool to obtain a visualization graph; and displaying the visualization graph.

[0012] According to an embodiment of the present invention, multiple hot words are determined from multiple target words according to word frequency information, and the method further includes: determining a preset number of hot words from the multiple target words according to second word order information.

[0013] According to an embodiment of the present invention, performing semantic analysis on multiple hot words to obtain multiple optimized hot words also includes: determining the word vector information of each of the multiple hot words; performing clustering operations on the multiple hot words according to the word vector information to obtain at least one category cluster with similar semantics; and deleting hot words whose distance to at least one category cluster is greater than a distance threshold to obtain optimized hot words.

[0014] According to an embodiment of the present invention, multiple target words are extracted from multiple text data, including: taking the words in the keyword part of the text data as multiple target words; or extracting multiple initial words from the title and abstract of the text data; and cleaning the multiple initial words based on a stop word list to obtain multiple target words.

[0015] The second aspect of the present invention provides a text data processing device based on word frequency fluctuation and semantic analysis, including: an acquisition module, used to acquire multiple text data belonging to a target field within a preset time period, wherein the preset time period includes multiple continuous sub-time periods. An extraction module, used to extract multiple target words from multiple text data. A word frequency determination module, used to determine the word frequency information of each target word in each sub-time period in multiple sub-time periods. A hot spot determination module, used to determine multiple hot words from multiple target words based on the word frequency information, wherein the hot words are used to characterize the technology development hotspots of the target field within the preset time period. A semantic analysis module, used to perform semantic analysis on multiple hot words to obtain multiple optimized hot words.

[0016] The third aspect of the present invention provides an electronic device, comprising: one or more processors; a memory for storing one or more computer programs, which, when the one or more processors execute the one or more computer programs, implements the steps of the above-mentioned text data processing method based on word frequency fluctuation and semantic analysis.

[0017] The fourth aspect of the present invention also provides a computer-readable storage medium having a computer program or instruction stored thereon, which, when executed by a processor, implements the steps of the above-mentioned text data processing method based on word frequency fluctuation and semantic analysis.

[0018] The fifth aspect of the present invention also provides a computer program product, including a computer program or instructions, which, when executed by a processor, implement the steps of the text data processing method based on word frequency fluctuation and semantic analysis.

[0019] The embodiment of the present invention provides a new frequency ranking method for text words, which introduces a time tag to the word frequency, and uses the word frequency change information of the words in a continuous time period to quickly and accurately locate hot words. Afterwards, the determined hot words are further optimized through semantic analysis in the natural language processing method, which further improves the accuracy of the determined hot words, helps technology companies to quickly and accurately select technology development routes based on hot words, and reduces the cost of subsequent migration, transformation, and upgrading of enterprises. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] The above contents and other objects, features and advantages of the present invention will become more apparent through the following description of the embodiments of the present invention with reference to the accompanying drawings, in which:

[0021] Figure 1 The application scenario of the text data processing method based on word frequency fluctuation and semantic analysis according to an embodiment of the present invention is shown.

[0022] Figure 2 A flow chart of a text data processing method based on word frequency fluctuation and semantic analysis according to an embodiment of the present invention is shown.

[0023] Figure 3 The graph shows the frequency fluctuation trend distribution of all words with the same frequency of 10 in 2000.

[0024] Figure 4 A schematic diagram showing the sorting of multiple target words based on the second word order information is shown.

[0025] Figure 5 A structural block diagram of a text data processing device based on word frequency fluctuation and semantic analysis according to an embodiment of the present invention is shown.

[0026] Figure 6 A block diagram of an electronic device suitable for a text data processing method based on word frequency fluctuation and semantic analysis according to an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0027] Below, embodiments of the present invention will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the present invention. In the following detailed description, for ease of explanation, many specific details are set forth to provide a comprehensive understanding of embodiments of the present invention. However, it is apparent that one or more embodiments may also be implemented without these specific details. In addition, in the following description, descriptions of known structures and technologies are omitted to avoid unnecessary confusion of concepts of the present invention.

[0028] The terms used herein are only for describing specific embodiments and are not intended to limit the present invention. The terms "comprise", "include", etc. used herein indicate the existence of the features, steps, operations and / or components, but do not exclude the existence or addition of one or more other features, steps, operations or components.

[0029] All terms (including technical and scientific terms) used herein have the meanings commonly understood by those skilled in the art unless otherwise defined. It should be noted that the terms used herein should be interpreted as having a meaning consistent with the context of this specification and should not be interpreted in an idealized or overly rigid manner.

[0030] In order to facilitate technology companies to formulate technology development routes, technology companies generally do not pay much attention to the integrity of knowledge and understanding of knowledge, but are more concerned with the overall trend or direction of technology development. In related applications, PageRank sorting methods, such as TF-IDF, TextRank and other algorithms, can be used for sorting, but such methods are helpful for knowledge graph construction, but less helpful for technology development trends and hot spot analysis, which restricts the timeliness and accuracy of determining technology development hot spots and trends based on word frequency sorting.

[0031] In order to solve the problem of how to quickly and accurately determine the hot spots and trends of technological development, the present invention provides a text data processing method based on word frequency fluctuation and semantic analysis, including: acquiring multiple text data belonging to a target field within a preset time period, wherein the preset time period includes multiple continuous sub-time periods; extracting multiple target words from the multiple text data; determining the word frequency information of each target word in the multiple sub-time periods; determining multiple hot words from the multiple target words based on the word frequency information of each of the multiple target words in the multiple sub-time periods, wherein the hot words are used to characterize the hot spots of technological development in the target field within the preset time period; and performing semantic analysis on the multiple hot words to obtain multiple optimized hot words.

[0032] Figure 1 The application scenario of the text data processing method based on word frequency fluctuation and semantic analysis according to an embodiment of the present invention is shown.

[0033] like Figure 1As shown, the application scenario 100 according to this embodiment may include a first terminal device 101, a second terminal device 102, a third terminal device 103, a network 104 and a server 105. The network 104 is used to provide a medium for a communication link between the first terminal device 101, the second terminal device 102, the third terminal device 103 and the server 105. The network 104 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc. The first terminal device 101, the second terminal device 102, and the third terminal device 103 may be various electronic devices with display screens and supporting web browsing, including but not limited to smart phones, tablet computers, laptop portable computers, and desktop computers, etc. The server 105 may be a server that provides various services, such as a background management server that provides support for websites browsed by users using the first terminal device 101, the second terminal device 102, and the third terminal device 103 (only as an example).

[0034] It should be noted that the text data processing method based on word frequency fluctuation and semantic analysis provided in the embodiment of the present invention can generally be executed by the server 105, or other servers, or server clusters. Accordingly, the text data processing based on word frequency fluctuation and semantic analysis provided in the embodiment of the present invention can generally be set in the server 105, other servers, or server clusters.

[0035] For example, a user can use one of the first terminal device 101, the second terminal device 102, and the third terminal device 103 to log in to a text website, and use the search engine of the text website to obtain multiple text data belonging to the target field within a preset time period. The terminal device used by the user can send the multiple text data obtained above to the server 105 through interaction with the server 105, and the server 105 extracts multiple target words from the multiple text data; determines the word frequency information of each target word in multiple sub-time periods; determines multiple hot words from the multiple target words according to the word frequency information of each of the multiple target words in multiple sub-time periods; and performs semantic analysis on the multiple hot words to obtain multiple optimized hot words, thereby determining the technical development hotspots and technical development trends in the target field.

[0036] Alternatively, the text data processing method based on word frequency fluctuation and semantic analysis provided in the embodiment of the present invention can generally be executed by a terminal device. For example, the user can also use one of the first terminal device 101, the second terminal device 102, and the third terminal device 103 to obtain multiple text data locally, and extract target words, determine word frequency information, determine and optimize hot words based on the terminal device used by the user, and the specific operations are similar to those of the above server 105.

[0037] It should be understood that Figure 1The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0038] Figure 2 A flow chart of a text data processing method based on word frequency fluctuation and semantic analysis according to an embodiment of the present invention is shown.

[0039] like Figure 2 As shown, the method 200 includes operations S210 to S250.

[0040] In operation S210, a plurality of text data belonging to a target domain within a preset time period is acquired, wherein the preset time period includes a plurality of continuous sub-time periods.

[0041] In operation S220, a plurality of target words are extracted from the plurality of text data.

[0042] In operation S230, word frequency information of each target word in a plurality of sub-time periods is determined.

[0043] In operation S240, a plurality of hot words are determined from the plurality of target words according to the word frequency information of each of the plurality of target words in the plurality of sub-time periods, wherein the hot words are used to represent the hot spots of technical development in the target field in the preset time period.

[0044] In operation S250, semantic analysis is performed on the plurality of hot words to obtain a plurality of optimized hot words.

[0045] According to the embodiment of the present invention, the preset time period and the continuous sub-time period can be determined according to actual needs. For example, the preset time period can be from 2000 to 2002, and accordingly, the preset time period can include three continuous sub-time periods, namely 2000, 2001, and 2002.

[0046] Alternatively, the preset time period is still 2000 to 2002, and the preset time period may include 6 consecutive sub-time periods, namely the first half of 2000, the second half of 2000, the first half of 2001, the second half of 2001, the first half of 2002, and the second half of 2002.

[0047] Alternatively, the preset time period may be 5 years, 10 years, 100 years, etc., and the sub-time period may be 1 day, 2 months, 1 year, 5 years, 10 years, etc.

[0048] According to an embodiment of the present invention, the target field may be an emerging field, such as the computer field or the large model field. The target field may also be a field that the user desires to explore, such as the science and technology field or the climate change field.

[0049] In addition, the target field may not be limited to a certain broad field, but may also be a sub-field under a broad field, such as the image processing field and the natural language processing field under the computer field.

[0050] According to an embodiment of the present invention, the plurality of text data includes articles such as papers, documents, reviews, journals, etc. that can be obtained from invention channels. The text data can be texts in multiple languages, such as Chinese, English, etc.

[0051] According to an embodiment of the present invention, for operation S210, considering the comprehensiveness and timeliness of English literature and the advantage that English words do not have Chinese segmentation words, English literature can be preferentially selected as the analysis basis of the present invention, i.e., text data. Alternatively, if Chinese literature is used, a Chinese segmentation operation can be added in the process of extracting the target vocabulary in operation S220.

[0052] For example, the academic paper website (arxiv.org) created by Cornell University can be used as a resource pool for acquiring papers in the present invention.

[0053] According to an embodiment of the present invention, for operation S220, the target vocabulary extracted from the plurality of text data can be understood as a vocabulary with certain technical representative meaning in the text data. Taking the text data as a paper as an example, the target vocabulary can refer to a keyword in the paper; or, it can also be a vocabulary related to the technology in the target field extracted from the title and abstract of the paper.

[0054] It should be noted that at least one target word may be extracted from each text data, and the target words extracted from multiple text data may be the same or different.

[0055] According to an embodiment of the present invention, for operation S230, the word frequency information is used to represent the number of occurrences of the target word. The same target word may appear at least once in each sub-time period, or may not appear in a certain sub-time period.

[0056] For example, target vocabulary A appeared once in 2000, did not appear in 2001, and appeared 10 times in 2002. Then the frequency information of target vocabulary A in the sub-time period 2000 is 1, the frequency information in the sub-time period 2001 is 0, and the frequency information in the sub-time period 2002 is 10.

[0057] In practical applications, compared to the entire text data, although the target vocabulary can represent the technical content of the text data to a certain extent, not all target vocabulary are related to technology development hotspots or technology development trends. Therefore, after determining the target vocabulary, the target vocabulary is time-tagged, and the sorted target vocabulary is sorted, visualized and analyzed to screen out the hot words. Therefore, operation S240 can obtain the frequency fluctuation trend of the target vocabulary by analyzing the target vocabulary after adding the time tag, and then determine the hot words based on the frequency fluctuation trend. For operation S250, through the semantic analysis of the hot words, the hot words can be further optimized from a semantic perspective, so as to accurately screen out the technology development hotspots and technology development trends in the target field within a preset time period.

[0058] The embodiment of the present invention provides a new frequency ranking method for text words, which introduces a time tag to the word frequency, and uses the word frequency change information of the words in a continuous time period to quickly and accurately locate hot words. Afterwards, the determined hot words are further optimized through semantic analysis in the natural language processing method, which further improves the accuracy of the determined hot words, helps technology companies to quickly and accurately select technology development routes based on hot words, and reduces the cost of subsequent migration, transformation, and upgrading of enterprises.

[0059] According to an embodiment of the present invention, for operation S220, extracting multiple target words from multiple text data includes: taking the words in the keyword part of the text data as multiple target words; or extracting multiple initial words from the title and abstract of the text data; and cleaning the multiple initial words based on a stop word list to obtain multiple target words.

[0060] In a specific embodiment, keywords of the paper may be directly selected and used as target words.

[0061] However, in actual application, due to the influence of writing habits, not all papers provide keywords. Therefore, in order to ensure that as many papers in the target field as possible are analyzed, the papers can also be used as text data, and the words in the keyword part of the paper are also the keywords of the paper, and the title and abstract of the paper are also the title and abstract of the paper.

[0062] When the paper title and paper abstract are selected as the paper keyword replacement, multiple initial words can be extracted from the paper title and paper abstract through natural language processing methods. For example, for English papers, the English words in the paper title and paper abstract are directly used as the initial words; for Chinese papers, the paper title and paper abstract can be segmented first, and then the segmented words are used as the initial words.

[0063] In the present invention, it is considered that if only the paper abstract is processed, some core words will be "drowned" by the descriptive words in the paper abstract. Therefore, the present invention adds the vocabulary of the paper title so that the word frequency of the core words can be highlighted in the statistical process. Afterwards, based on the stop word list, stop words, numbers, spaces, etc. that interfere with the word frequency analysis are deleted from multiple initial words to obtain the target vocabulary. The stop word list is pre-constructed based on expert experience, and the stop words included in the stop word list are mainly function words such as pronouns, articles, and quantifiers.

[0064] In another specific embodiment, the paper keywords can be combined with the paper title and paper abstract. For example, for papers with keywords in multiple text data, the paper keywords can be directly used as target words; for papers without keywords in multiple text data, target words can be extracted from the paper title and paper abstract.

[0065] The embodiment of the present invention directly obtains the keywords of the paper to quickly, accurately and conveniently extract the target vocabulary. In addition, the paper title and the paper abstract can be combined to extract effective target vocabulary in a smaller paper abstract relative to the full paper, and the influence of the core vocabulary is increased by using the word frequency of the paper title, thereby ensuring the easy accessibility and accuracy of the target vocabulary.

[0066] Before determining the frequency information of each target word, it should be noted that the definition of hot words in the present invention is based on the following basic idea: the higher the frequency of a word, the higher its attention, the more people study it, and the more likely it is to become a trend in future technological development.

[0067] Therefore, according to an embodiment of the present invention, for operation S230, the number of occurrences of each target word in each sub-time period can be directly counted, and the word frequency information of each target word in each sub-time period can be determined based on the number of occurrences, so as to add a time tag to the target word.

[0068] Alternatively, the number of occurrences of each target word in the entire preset time period can be counted first, and the extracted target words can be counted and sorted. After completing the word frequency statistics and sorting, a time tag is added to each target word to obtain the word frequency information of each target word in each sub-time period. The time tag is also the time point when the paper where the target word is located is submitted. Adding a time tag to the word frequency is an innovative point of the present invention, which facilitates subsequent trend analysis.

[0069] According to an embodiment of the present invention, after determining the word frequency information of the target vocabulary, operation S240 includes: obtaining the total word frequency information of each target vocabulary within a preset time period; determining high-frequency vocabulary from multiple target vocabulary based on the total word frequency information; determining the word frequency fluctuation trend of each of the multiple high-frequency vocabulary based on the word frequency information of each of the multiple high-frequency vocabulary in multiple sub-time periods; and determining multiple hot words from the multiple high-frequency vocabulary based on the word frequency fluctuation trend.

[0070] For example, if the number of occurrences of each target word in the entire preset time period has been calculated in operation S230, the total word frequency information can be directly obtained. If the number of occurrences of each target word in the entire preset time period has not been calculated in operation S230, the total word frequency information of each target word in the preset time period can be obtained by counting the word frequency information in each sub-time period.

[0071] Since the basic idea of ​​the present invention is that the higher the frequency of a word, the higher its attention, the more people study it, and the easier it is to become a trend of future technological development, therefore, when analyzing the frequency fluctuation trend, the multiple target words can be sorted according to the total frequency information of the multiple target words in the preset time period, and the medium and low frequency target words can be deleted from the multiple target words. The medium and low frequency target words here are based on the entire preset time period. The medium and low frequency target words can be determined based on a predefined full-time word frequency threshold, such as the full-time word frequency threshold can be 1, 3, etc.

[0072] For low- and medium-frequency target words with low word frequency in the entire preset time period, these low- and medium-frequency target words are either niche fields or have just been proposed and are not mature, which is not very helpful for analyzing hot spots and technology development trends. Therefore, based on the total word frequency information of the entire preset time period, high-frequency words are selected from multiple target words to further improve the accuracy of the target words and reduce the amount of data for subsequent word fluctuation analysis.

[0073] In addition, after filtering out high-frequency words, some words that are obviously not related to technology or academics can be removed based on expert experience. For example, after deleting target words with total word frequency information less than or equal to 1 from 2000 to 2002 and sorting them, target words that are not related to technology can be deleted based on expert experience: 2000, abstract, etc.

[0074] In some embodiments, the word frequency fluctuation trend can be determined directly based on the word frequency information of each high-frequency word in multiple consecutive sub-time periods. For example, if the target word B appears twice in 2000, three times in 2001, and five times in 2002, the word frequency fluctuation trend can be determined based on the continuous change trend of the word frequency of the target word B from 2000 to 2002.

[0075] According to an embodiment of the present invention, the word frequency fluctuation trend of each high-frequency word is visualized by a visualization tool.

[0076] According to an embodiment of the present invention, the word frequency fluctuation trend includes continuous decrease, continuous increase and continuous fluctuation. According to the word frequency fluctuation trend, multiple hot words are determined from multiple high-frequency and medium-frequency words, including: deleting high-frequency words whose word frequency fluctuation trend is continuously decreasing from the multiple high-frequency words, to obtain multiple hot words.

[0077] In a specific embodiment, only high-frequency words with a continuously increasing word frequency fluctuation trend may be used as hot words.

[0078] For example, for a category of target words with a frequency of less than 20 in 2000, if this category of target words fluctuates in 2001, it should be paid special attention to and can be selected as hot words for further analysis to understand the hot spots and trends of technology development in the target field. Furthermore, if this category of target words continues to maintain a high frequency in 2001 and 2002, this category of target words should be the hot words that are focused on. If the frequency of a certain category of target words gradually decreases to a low frequency or disappears after three years of evolution, it means that this category of target words has withdrawn from the stage of history. If the frequency of a certain category of target words is low in 2000, high in 2001, and low in 2002, it means that the research on this type of technology has matured or this type of technology has encountered a bottleneck. At this time, it can also be focused on as a hot word, or it can be abandoned.

[0079] In another specific embodiment, for high-frequency words that continue to rise and continue to fluctuate, combined with the word frequency information of each target word in each sub-time period, multiple hot words that meet predetermined conditions are screened out from the multiple high-frequency words that continue to rise and continue to fluctuate.

[0080] For example, after selecting a plurality of high-frequency words that are continuously rising and continuously fluctuating, taking the word frequency information of the earliest sub-time period as a benchmark, a plurality of same-frequency words with the same word frequency information in the earliest sub-time period are selected.

[0081] For these same-frequency words, the same-frequency words with high frequency in the second earliest sub-time period can be used as hot words. Alternatively, for these same-frequency words, through a single screening, multiple same-frequency words with high frequency in the second earliest sub-time period are selected, that is, the same-frequency words whose frequency information in the second year is higher than that in the first year. Alternatively, through a second screening, several same-frequency words whose frequency information in the third year exceeds that in the second year are selected from the same-frequency words with the highest frequency information in the second year, and they are used as hot words. For more sub-time periods, the method of determining hot words is similar to that above and will not be repeated here. It should be noted that the same-frequency words mentioned here refer to the frequency information in the earliest sub-time period.

[0082] In order to more clearly describe the embodiment of determining hot words, the following will be combined with Figure 3 Further description.

[0083] Figure 3 The graph shows the frequency fluctuation trend distribution of all words with the same frequency of 10 in 2000.

[0084] like Figure 3 As shown in the figure, the horizontal axis is all the words with the same frequency of 10 in 2000, and the vertical axis is the word frequency information. The blue line, orange line, and gray line represent the word frequency information of the same target word in 2000, 2001, and 2002 respectively.

[0085] According to an embodiment of the present invention, for all the same-frequency words with a word frequency of 10, the same-frequency words with high word frequency in 2001 can be directly used as hot words, such as Figure 3 A preset number of co-frequency words, such as 20, are selected from the right side to the left side.

[0086] Alternatively, through a single screening, the same-frequency words whose frequency information in 2001 is higher than that in 2000 are selected, that is, the same-frequency words in the orange line that are higher than the blue line.

[0087] Or, through the second screening, for several high-frequency words in 2001 ( Figure 3 Several words with the same frequency near the peak on the right), select texts according to the word frequency in 2002, that is, Figure 3 The frequency of the words in the middle green line rises again near the peak of the orange line. These are the words that should be focused on.

[0088] In another specific embodiment, there is another type of target vocabulary whose frequency is relatively high in multiple consecutive sub-time periods, and can also be directly used as hot words. For example, in 2000, target words with a frequency of more than 100 have relatively high frequency in the next three years, and this type of target vocabulary can be used as hot words. The number of this type of target vocabulary is not large, roughly around 100, and can be directly screened based on expert experience, or can be screened using natural language processing.

[0089] In other embodiments, determining the frequency fluctuation trend of each of the multiple high-frequency words based on the frequency information of each of the multiple high-frequency words in the multiple sub-time periods includes: for each sub-time period, normalizing the frequency information of each high-frequency word based on the frequency information of the multiple high-frequency words in the sub-time period to obtain the normalized frequency information of each high-frequency word in each sub-time period; for each high-frequency word frequency, determining the frequency fluctuation trend of each high-frequency word based on the normalized frequency information of the high-frequency word in the multiple sub-time periods.

[0090] Considering that the word frequency in a certain sub-time period may surge, the surge may be related to the number of papers submitted in this field. Therefore, in order to avoid a certain sub-time period having a greater impact on the overall frequency fluctuation trend of the target vocabulary, the word frequency statistics of each sub-time period can be normalized.

[0091] As a specific embodiment, for each sub-time period, the word frequency information of each target word in the sub-time period is divided by the word frequency information of all target words in the sub-time period.

[0092] For example, the word frequency information of target vocabulary C from 2000 to 2002 was 10, 162, and 30 respectively, and the word frequency information of target vocabulary D from 2000 to 2002 was 5, 62, and 3 respectively. The word frequency information of target vocabulary C and D in 2001 was affected by the overall number of papers submitted that year, and the resulting word frequency fluctuation trend was continuous fluctuation. However, the normalized word frequency information of target vocabulary C from 2000 to 2002 was 0.66, 0.72, and 0.91 respectively, and the normalized word frequency information of target vocabulary D from 2000 to 2002 was 0.33, 0.28, and 0.09 respectively. After excluding the influence of 2001, the word frequency fluctuation trends of target vocabulary C and D were continuous rising and continuous falling respectively.

[0093] Therefore, by using the word frequency information of multiple high-frequency words in the same sub-time period to perform normalization operations, the influence of a particular sub-time period on the word fluctuation trend of high-frequency words can be minimized as much as possible, which helps to improve the accuracy of the determined hot words.

[0094] According to an embodiment of the present invention, after determining the word frequency information of each target word in the multiple sub-time periods, it also includes sorting the multiple target words. The specific sorting operation includes: for two adjacent sub-time periods, sorting the multiple co-frequency words in the previous sub-time period according to the word frequency information of the later sub-time period to determine the change trend between the multiple co-frequency words; based on the first word frequency information of each of the multiple target words in the earliest sub-time period, arranging the multiple target words in descending order or ascending order to obtain the first word sequence information; according to the change trend between the multiple co-frequency words, arranging the multiple co-frequency words in the multiple target words in descending order or ascending order to obtain the second word sequence information, wherein the multiple target words and the multiple co-frequency words are all arranged in descending order or in ascending order.

[0095] For example, still taking the preset time period of 2000 to 2002 as an example, after determining the target vocabulary, first sort all the target vocabulary that appeared in 2000 in descending order according to the word frequency and put them in an Excel table. Similarly, put the words in 2001 and 2002 in the table according to the above method. Then, find the vocabulary in the table containing 2000 from the table of 2001, and find the word frequency of these words in 2001 and put them in a column of this table in 2000, and leave the vocabulary blank. Similarly, do the same processing for the vocabulary in 2002. The above method can be implemented by applying the VLOOKUP function of Excel. After completing the above sorting task, perform secondary sorting for multiple target words with the first vocabulary order, that is, sort the same frequency words with the same word frequency in the word frequency statistics of 2000. For example, for the same-frequency words with a frequency of "2" in 2000, these same-frequency words can be sorted according to their frequency in 2001, either in ascending or descending order or by normal distribution. In this way, the frequency of the words in 2001 can help to find the trend or law of the same-frequency words in 2000. Otherwise, the words in 2001 are out of order and present a "random walk" state distribution, which is not convenient for trend analysis. Similarly, the frequency of the words in 2002 can be used to sort the same-frequency words in 2001.

[0096] The present invention takes the same-frequency words with a word frequency of "10" in 2000 as an example, and shows the sorted second word order information through Table 1.

[0097] Table 1

[0098]

[0099] The words in the first column from left to right are arranged in descending order according to the word frequency in 2001 recorded in the third column "2001vs2000" and the word frequency in 2002 recorded in the fourth column "2002vs2000".

[0100] According to an embodiment of the present invention, text data processing based on word frequency fluctuation and semantic analysis also includes: using a visualization tool, performing visualization operations on multiple target words based on second vocabulary order information to obtain visualization graphics; and displaying the visualization graphics.

[0101] For example, the visualization tool may be Excel, and a plurality of target words arranged based on the second vocabulary order information may be visualized with the help of a bar chart of Excel.

[0102] Figure 4 A schematic diagram showing the sorting of multiple target words based on the second word order information is shown.

[0103] like Figure 4 As shown in the figure, the horizontal axis is all the target words that appeared from 2000 to 2002, and the vertical axis is the word frequency information. The blue line, orange line, and gray line represent the word frequency information of the same target word in 2000, 2001, and 2002 respectively.

[0104] According to an embodiment of the present invention, for operation S240, hot words can also be selected from multiple target words by setting the number of words. Specifically, according to the word frequency information, multiple hot words are determined from the multiple target words, and it also includes: according to the second word order information, a preset number of hot words are determined from the multiple target words.

[0105] Due to the above-mentioned sorting operation on the target words, not only can multiple target words have different importance levels in the same sub-time period based on the frequency of the words, but also the words with the same frequency can have different importance levels due to the frequency of the words in the next few sub-time periods. Therefore, when selecting hot words based on the number of words, the hot words can be determined simply, quickly and conveniently.

[0106] According to an embodiment of the present invention, for operation S250, semantic analysis is performed on multiple hot words to obtain multiple optimized hot words, which also includes: determining the word vector information of each of the multiple hot words; clustering the multiple hot words according to the word vector information to obtain at least one category cluster with similar semantics; and deleting hot words whose distance to at least one category cluster is greater than a distance threshold to obtain optimized hot words.

[0107] Through the above operations S210-S240, the analysis of the word frequency fluctuation trend is realized, and the hot words are screened, but it is difficult to distinguish whether there are synonyms or near synonyms in the hot words. In addition, since the text data used in the present invention is a paper, and the words in the paper are usually obscure and difficult to understand, therefore, only obtaining hot words through word frequency may lead to misjudgment of technology development hotspots and technology development trends.

[0108] Therefore, the present invention introduces a word embedding algorithm in natural language processing technology in operation S250. The word embedding algorithm is used to extract the contextual features of text data, that is, the paper, and the contextual features are used to "interpret" the hot words and convert them into word vector information, that is, word embedding information (Word Embedding). After conversion into word vector information, the hot words are classified by a clustering method to form at least one category cluster. Each category cluster can be understood as a category cluster composed of synonyms or near-synonyms. By deleting hot words whose distance from at least one category cluster is greater than a distance threshold, semantically isolated hot words can be deleted, thereby obtaining optimized hot words.

[0109] A variety of distance criteria can be used for clustering, such as Euclidean distance, cosine similarity, etc. If the semantics of two words are similar, the distance between their word vectors is also relatively small.

[0110] The specific operation steps are as follows: Step 1, convert the hot words into word vector information through the gensim module in python. Step 2, cluster the word vector information in step 1. For example, clustering can be performed through the KMeans method of the sklearn module in python. Step 3, delete the hot words whose distance to at least one category cluster obtained by clustering is greater than the distance threshold to obtain the optimized hot words. The optimized hot words are basically the core words of future technology development trends.

[0111] In actual application, a typical word among the optimized hot words is "web", which had a frequency of 37 in 2000, 82 in 2001, and 0 in 2002. This has a lot to do with the dot-com bubble and its collapse at that time, which indirectly proves the effectiveness of the present invention.

[0112] In order to more clearly understand the implementation of the present invention, a specific embodiment is described below.

[0113] Step 1: Obtain all paper elements in the target field, such as title, abstract, and keywords.

[0114] Step 2: Extract target words from the multiple papers obtained in step 1, and clean the target words to remove non-core words such as stop words and auxiliary words.

[0115] Step 3: Perform word frequency statistics on the cleaned target vocabulary.

[0116] Step 4: Add time tags to the target words after counting the word frequency.

[0117] Step 5: sort multiple target words and determine hot words from them according to the word frequency information of the target words in each sub-time period. In step 5, high-frequency words can be selected from multiple target words in sequence, the word frequencies can be normalized, the word frequency fluctuation trend can be analyzed, and the hot words can be determined.

[0118] Step 6: Visualize the sorted word frequencies. In step 6, the word frequency information, target words or high-frequency words processed in step 5 can be normalized at any time.

[0119] Step 7: Analyze the visualization results and give hotspot analysis conclusions.

[0120] Figure 5 A structural block diagram of a text data processing device based on word frequency fluctuation and semantic analysis according to an embodiment of the present invention is shown.

[0121] like Figure 5 As shown, the text data processing device 500 based on word frequency fluctuation and semantic analysis of this embodiment includes an acquisition module 510, an extraction module 520, a word frequency determination module 530, a hotspot determination module 540 and a semantic analysis module 550.

[0122] The acquisition module 510 is used to acquire a plurality of text data belonging to a target domain within a preset time period, wherein the preset time period includes a plurality of continuous sub-time periods.

[0123] The extraction module 520 is used to extract multiple target words from multiple text data.

[0124] The word frequency determination module 530 is used to determine the word frequency information of each target word in each sub-time period of the plurality of sub-time periods.

[0125] The hotspot determination module 540 is used to determine a plurality of hotspot words from a plurality of target words according to word frequency information, wherein the hotspot words are used to represent the hotspots of technical development in the target field within a preset time period.

[0126] The semantic analysis module 550 is used to perform semantic analysis on a plurality of hot words to obtain a plurality of optimized hot words.

[0127] It should be noted that the operation of the device part of the present invention corresponds to the operation of the method part thereof, and will not be described in detail here.

[0128] According to an embodiment of the present invention, any multiple modules among the acquisition module 510, the extraction module 520, the word frequency determination module 530, the hotspot determination module 540 and the semantic analysis module 550 can be combined into one module for implementation, or any one of the modules can be split into multiple modules. Alternatively, at least part of the functions of one or more of these modules can be combined with at least part of the functions of other modules and implemented in one module.

[0129] According to an embodiment of the present invention, multiple of the acquisition module 510, the extraction module 520, the word frequency determination 530, the hot spot determination module 540 and the semantic analysis module 550 can be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on a chip, a system on a substrate, a system on a package, an application-specific integrated circuit (ASIC), or can be implemented by hardware or firmware such as any other reasonable way of integrating or packaging the circuit, or implemented in any one of the three implementation methods of software, hardware and firmware or in an appropriate combination of any of them. Alternatively, multiple of the acquisition module 510, the extraction module 520, the word frequency determination 530, the hot spot determination module 540 and the semantic analysis module 550 can be at least partially implemented as a computer program module, and when the computer program module is run, the corresponding function can be executed.

[0130] Figure 6 A block diagram of an electronic device suitable for a text data processing method based on word frequency fluctuation and semantic analysis according to an embodiment of the present invention is shown.

[0131] like Figure 6 As shown, the electronic device 600 according to an embodiment of the present invention includes a processor 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage part 608 to a random access memory (RAM) 603. The processor 601 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or a related chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 601 may also include an onboard memory for caching purposes. The processor 601 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present invention.

[0132] In RAM603, various programs and data required for the operation of electronic device 600 are stored. Processor 601, ROM602 and RAM603 are connected to each other via bus 604. Processor 601 performs various operations of the method flow according to an embodiment of the present invention by executing the programs in ROM 602 and / or RAM 603. It should be noted that the program can also be stored in one or more memories other than ROM602 and RAM603. Processor 601 can also perform various operations of the method flow according to an embodiment of the present invention by executing the programs stored in the one or more memories.

[0133] According to an embodiment of the present invention, the electronic device 600 may further include an input / output (I / O) interface 605, which is also connected to the bus 604. The electronic device 600 may further include one or more of the following components connected to the input / output I / O interface 605: an input portion 606 including a keyboard, a mouse, etc.; an output portion 607 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage portion 608 including a hard disk, etc.; and a communication portion 609 including a network interface card such as a LAN card, a modem, etc. The communication portion 609 performs communication processing via a network such as the Internet. A drive 610 is also connected to the I / O interface 605 as needed. A removable medium 611, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 610 as needed, so that a computer program read therefrom is installed into the storage portion 608 as needed.

[0134] The present invention also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiment; or may exist independently without being assembled into the device / apparatus / system. The above computer-readable storage medium carries one or more programs, and when the above one or more programs are executed, the method according to the embodiment of the present invention is implemented.

[0135] According to an embodiment of the present invention, the computer-readable storage medium may be a non-volatile computer-readable storage medium, for example, it may include but is not limited to: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present invention, the computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, an apparatus or a device. For example, according to an embodiment of the present invention, the computer-readable storage medium may include the ROM602 and / or RAM603 described above and / or one or more memories other than ROM602 and RAM603.

[0136] The embodiment of the present invention also includes a computer program product, which includes a computer program, and the computer program contains program code for executing the method shown in the flowchart. When the computer program product is run in a computer system, the program code is used to enable the computer system to implement the above method provided by the embodiment of the present invention.

[0137] The computer program executes the above functions defined in the system / device of the embodiment of the present invention when it is executed by the processor 601. According to the embodiment of the present invention, the system, device, module, unit, etc. described above can be implemented by a computer program module.

[0138] In one embodiment, the computer program may rely on tangible storage media such as optical storage devices, magnetic storage devices, etc. In another embodiment, the computer program may also be transmitted and distributed in the form of signals on a network medium, and downloaded and installed through the communication part 609, and / or installed from a removable medium 611. The program code contained in the computer program may be transmitted using any appropriate network medium, including but not limited to: wireless, wired, etc., or any suitable combination of the above.

[0139] In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 609, and / or installed from the removable medium 611. When the computer program is executed by the processor 601, the above functions defined in the system of the embodiment of the present invention are performed. According to the embodiment of the present invention, the system, device, means, module, unit, etc. described above can be implemented by a computer program module.

[0140] According to an embodiment of the present invention, the program code for executing the computer program provided by the embodiment of the present invention can be written in any combination of one or more programming languages. Specifically, these computing programs can be implemented using high-level process and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, Java, C++, python, "C" language or similar programming languages. The program code can be executed entirely on the user computing device, partially on the user device, partially on the remote computing device, or entirely on the remote computing device or server. In the case of a remote computing device, the remote computing device can be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computing device (e.g., using an Internet service provider to connect through the Internet).

[0141] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present invention. In this regard, each box in the flow chart or block diagram can represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0142] It will be appreciated by those skilled in the art that the features described in the various embodiments of the present invention may be combined and / or combined in various ways, even if such combinations or combinations are not explicitly described in the present invention. In particular, without departing from the spirit and teachings of the present invention, the features described in the various embodiments of the present invention may be combined and / or combined in various ways. All of these combinations and / or combinations fall within the scope of the present invention.

[0143] The specific embodiments described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A text data processing method based on word frequency fluctuation and semantic analysis, It is characterized in that The method comprises: Acquire a plurality of text data belonging to a target domain within a preset time period, wherein the preset time period includes a plurality of continuous sub-time periods; Extracting a plurality of target words from the plurality of text data; Determine the word frequency information of each of the target words in each of the multiple sub-time periods; Determine a plurality of hot words from the plurality of target words according to the word frequency information, wherein the hot words are used to represent the hot spots of technical development in the target field within the preset time period; and Semantic analysis is performed on the multiple hot words to obtain multiple optimized hot words.

2. The method according to claim 1, It is characterized in that Determining a plurality of hot words from the plurality of target words according to the word frequency information includes: Obtaining total word frequency information of each target word within the preset time period; Determining high-frequency words from the plurality of target words according to the total word frequency information; Determining the frequency fluctuation trend of each of the plurality of high-frequency words according to the frequency information of each of the plurality of high-frequency words in the plurality of sub-time periods; and According to the word frequency fluctuation trend, a plurality of hot words are determined from the plurality of high-frequency words.

3. The method according to claim 2, It is characterized in that The word frequency fluctuation trend includes continuous decrease, continuous increase and continuous fluctuation. The determining of multiple hot words from the multiple high-frequency words according to the word frequency fluctuation trend includes: The high-frequency words whose frequency fluctuation trend is continuously decreasing among the plurality of high-frequency words are deleted to obtain the plurality of hot words.

4. The method according to claim 2, It is characterized in that Determining the frequency fluctuation trend of each of the plurality of high-frequency words according to the frequency information of each of the plurality of high-frequency words in the plurality of sub-time periods includes: For each of the sub-time periods, the word frequency information of each of the high-frequency words is normalized according to the word frequency information of the plurality of high-frequency words in the sub-time period, so as to obtain the normalized word frequency information of each of the high-frequency words in each of the sub-time periods; For each of the high-frequency words, the word frequency fluctuation trend of each of the high-frequency words is determined according to the normalized word frequency information of the high-frequency words in the plurality of sub-time periods.

5. The method according to claim 1, It is characterized in that After determining the word frequency information of each target word in the plurality of sub-time periods, the method further includes: For two adjacent sub-time periods, a plurality of words with the same frequency in the previous sub-time period are sorted according to the word frequency information of the later sub-time period to determine the change trend between the plurality of words with the same frequency; Arrange the target words in descending order or ascending order based on the first word frequency information of each of the target words in the earliest sub-time period to obtain first word order information; According to the change trend among the multiple co-frequency words, the multiple co-frequency words in the multiple target words are arranged in descending order or ascending order to obtain second word order information, wherein the multiple target words and the multiple co-frequency words are all arranged in descending order or ascending order.

6. The method according to claim 5, It is characterized in that The method further comprises: Using a visualization tool, performing a visualization operation on the plurality of target words based on the second word order information to obtain a visualization graph; and The visualization graph is displayed.

7. The method according to claim 5, It is characterized in that The step of determining a plurality of hot words from the plurality of target words according to the word frequency information further includes: According to the second vocabulary sequence information, a preset number of hot words are determined from the plurality of target words.

8. The method according to claim 1, It is characterized in that The semantic analysis of the multiple hot words to obtain the optimized multiple hot words also includes: Determine word vector information of each of the multiple hot words; Performing a clustering operation on the multiple hot words according to the word vector information to obtain at least one category cluster with similar semantics; and The hot words whose distance from the at least one category cluster is greater than a distance threshold are deleted to obtain optimized hot words.

9. The method according to claim 1, It is characterized in that The step of extracting a plurality of target words from the plurality of text data comprises: Using the keywords in the text data as multiple target words; or Extracting a plurality of initial words from the title and abstract of the text data; and cleaning the plurality of initial words based on a stop word list to obtain a plurality of target words.

10. An electronic device, It is characterized in that The electronic device comprises: one or more processors; a memory for storing one or more computer programs, It is characterized in that the one or more processors execute the one or more computer programs to implement the steps of the method according to any one of claims 1 to 9.

Images