Similar news identification method and device and medium

Abstract

The invention discloses a similar news identification method and device and a medium, and the method comprises the steps: analyzing newly stored news, and obtaining a news identification index; the news identification indexes comprise a news title, a news abstract, release time, an enterprise unique identifier and a news unique identifier; according to a distributed search engine and the news identification index, obtaining a correlation score between the newly stored news and the old news and / or the old news group; selecting similar news of the newly stored news from the old news and / or the old news group according to the correlation score; constructing a news group according to the newly stored news and the similar news; the newly-stored news serves as a main document of the news group, and similar news is mounted in the news group in a sub-document mode through the unique identifier of the associated enterprise. And the similar news identification efficiency is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer, and particularly relates to a similar news identification method, device and medium. BACKGROUND

[0002] With the development of the Internet, the speed and coverage of news dissemination have reached an unprecedented level. However, this development has also brought new challenges: different media agencies often use similar expressions or quote the same information sources when reporting the same event, resulting in high homogeneity of news content.

[0003] Therefore, for professionals who rely on public opinion analysis to assess the health of enterprises, such as customer managers, it has become a big problem to quickly find valuable information from numerous repetitive and similar news reports.

[0004] At present, the existing similar news identification method mostly adopts simple text matching or keyword search, resulting in the problem of low identification accuracy. SUMMARY

[0005] The present application provides a similar news identification method, device and medium, which is used to solve the problem of low similar news identification accuracy.

[0006] The present application adopts the following technical scheme:

[0007] On the one hand, the present application provides a similar news identification method, which comprises: analyzing new warehouse news to obtain news identification indicators; the news identification indicators include news title, news abstract, publication time, enterprise unique identifier, news unique identifier; obtaining the relevance score of new warehouse news and old news and / or old news group according to the distributed search engine and the news identification indicators; selecting similar news of new warehouse news from old news and / or old news group according to the relevance score; constructing a news group according to the new warehouse news and the similar news; the new warehouse news is used as the main document of the news group, and the similar news is mounted in the news group in the form of a sub-document through the associated enterprise unique identifier.

[0008] In one example, the relevance score of the new-in-warehouse news and the old news and / or the old news group is obtained according to the distributed search engine and the news identification index, specifically comprising: filtering the old news and / or the old news group belonging to the same enterprise from the news database by using the term query of Elasticsearch to obtain a first query set; filtering the old news and / or the old news group meeting the time window from the first query set by using the range of Elasticsearch to limit the time window to obtain a second query set; filtering the old news and / or the old news group with similar content from the second query set by using the match query of Elasticsearch for the news title and the news abstract and connecting with the should logic, and determining the relevance score calculated by Elasticsearch for the old news and / or the old news with similar content.

[0009] In one example, the old news and / or the old news group meeting the time window is filtered from the first query set to obtain the second query set, specifically comprising: comparing the publishing time of the new-in-warehouse news with the publishing time of the old news in the first query set to determine whether the old news meets the time window; and / or comparing the publishing time of the new-in-warehouse news with the publishing time of the main document news of the old news group in the first query set to determine whether the old news group meets the time window; and the old news and / or the old news group meeting the time window is assembled into the second query set.

[0010] In one example, the old news and / or the old news group with similar content is filtered from the second query set, specifically comprising: calculating the similarity of the news title and the news abstract of the new-in-warehouse news with the news title and the news abstract of the old news in the second query set to determine whether the old news has similarity; and / or calculating the similarity of the news title and the news abstract of the new-in-warehouse news with the news title and the news abstract of the main document news of the old news group in the second query set to determine whether the old news group has similarity; and the old news and / or the old news group with similarity is assembled into the second query set.

[0011] In one example, the similar news of the new-in-warehouse news is selected from the old news and / or the old news group according to the relevance score, specifically comprising: selecting the highest relevance score; compensating the highest relevance score according to a preset score compensation coefficient to obtain a compensated relevance score; the compensation coefficient is greater than 0 and less than 1; and selecting the similar news of the new-in-warehouse news from the old news and / or the old news group according to the compensated relevance score and the highest relevance score.

[0012] In an example, the selecting the similar news of the newly-stored news from the old news and / or the old news group according to the compensation correlation score and the highest correlation score comprises: constructing a score interval according to the compensation correlation score and the highest correlation score; determining a target old news and / or a target old news group whose correlation scores are in the score interval, and determining each old news of the target old news and / or the target old news group as the similar news of the newly-stored news.

[0013] In an example, after the constructing the news group according to the newly-stored news and the similar news, the method further comprises: deleting all old news and / or old news groups that have been classified into the news group in the news database; and storing the news group into the news database.

[0014] In an example, after the storing the news group into the news database, the method further comprises: displaying a main document news of the news group when the news group is queried by an application end; and displaying the similar news of the news group when a similar news trigger condition of the news group is received.

[0015] In another aspect, the embodiments of the present application provide a similar news identification device, comprising: at least one processor; and a memory in communication connection with the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the similar news identification method according to any one of the embodiments.

[0016] In another aspect, the embodiments of the present application provide a similar news identification nonvolatile computer storage medium, which stores computer executable instructions, and the computer executable instructions can perform the similar news identification method according to any one of the embodiments.

[0017] The above at least one technical solution adopted by the embodiments of the present application can achieve the following beneficial effects:

[0018] By comprehensively analyzing multi-dimensional features such as news titles, news abstracts and publishing times, and using a distributed search engine (such as elasticsearch) for automatic query, the correlation between the newly-stored news and the old news can be automatically calculated, the news reports with high similarity can be quickly screened out, and the news reports can be classified into corresponding news groups. This mechanism greatly improves the efficiency of users obtaining key information, especially when facing massive news data, the information retrieval efficiency is improved.

[0019] In addition, the newly-stored news is the main document of the news group, so when the news is displayed on the application side, only the main document news (i.e., the latest news) in each news group needs to be displayed, which greatly reduces the repeated display of similar news and improves the user experience. At the same time, the user can view all similar news of the displayed news by clicking the similar news, ensuring the comprehensiveness and accessibility of information. BRIEF DESCRIPTION OF DRAWINGS

[0020] In order to more clearly illustrate the technical solutions of the present application, some embodiments of the present application will be described in detail below with reference to the accompanying drawings, in which:

[0021] Figure 1 A flowchart of a similar news identification method provided by an embodiment of the present application;

[0022] Figure 2 A structural diagram of a similar news identification device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0023] In order to make the purpose, technical solutions and advantages of the present application clearer, the technical solutions of the present application will be described clearly and completely below with reference to specific embodiments and corresponding drawings. Obviously, the described embodiments are only some of the embodiments of the present application, not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0024] Some embodiments of the present application will be described in detail below with reference to the accompanying drawings.

[0025] Figure 1 A flowchart of a similar news identification method provided by an embodiment of the present application. The method can be applied to different business fields, such as the Internet financial business field, the e-commerce business field, the instant messaging business field, the game business field, the public service business field, etc. Some input parameters or intermediate results in the flowchart allow manual intervention to adjust to help improve accuracy.

[0026] The implementation of the analysis method related to the embodiments of the present application can be a terminal device or a server, and the present application does not make special limitations thereon. For the convenience of understanding and description, the following embodiments are described in detail with the server as an example.

[0027] It should be noted that the server can be a single device or a system composed of multiple devices, i.e., a distributed server, and the present application does not make specific limitations thereon.

[0028] Figure 1 The flowchart in the above embodiment includes the following steps:

[0029] S101: Analyze the newly-stored news to obtain news identification indexes; the news identification indexes include news titles, news abstracts, publication times, enterprise unique identifiers, and news unique identifiers.

[0030] It should be noted that the enterprise unique identifier is used to identify a single enterprise, and the news unique identifier is used to identify a single news.

[0031] S102: Obtain a relevance score of the newly-stored news and old news and / or old news groups according to a distributed search engine and the news identification indexes.

[0032] In some embodiments of the present application, a term query of Elasticsearch is used to filter old news and / or old news groups belonging to the same enterprise from a news database to obtain a first query set.

[0033] A range query of Elasticsearch is used to filter old news and / or old news groups that meet a time window from the first query set to obtain a second query set.

[0034] A match query of Elasticsearch is used to filter old news and / or old news groups that are similar in content from the second query set, and a relevance score calculated by Elasticsearch for the old news and / or old news that are similar in content is determined.

[0035] That is, a term query is first used to filter news belonging to the same enterprise. Then, a range query is used to limit a time window, for example, to select all news of the current date one day ago. Finally, a match query is used to filter news that are similar in content by combining news titles and news abstracts and using a should logical association.

[0036] In the process of filtering old news and / or old news groups that meet a time window from the first query set to obtain the second query set, the following steps are performed:

[0037] For old news: The publication time of the newly-stored news is compared with the publication time of old news in the first query set to determine whether the old news meets the time window.

[0038] For old news groups: The publication time of the newly-stored news is compared with the publication time of the main document news of old news groups in the first query set to determine whether the old news groups meet the time window.

[0039] In the process of filtering old news and / or old news groups that are similar in content from the second query set, the following steps are performed:

[0040] For old news: similarity calculation is performed between the news title and news abstract of the newly-stored news and the news title and news abstract of the old news in the second query set to determine whether the old news has similarity;

[0041] For old news group: similarity calculation is performed between the news title and news abstract of the newly-stored news and the main document news title and news abstract of the old news group in the second query set to determine whether the old news group has similarity.

[0042] The old news and / or old news group with similarity are grouped into the second query set.

[0043] It should be noted that Elasticsearch calculates a relevance score for each old news to measure the relevance of the old news to the query condition, and automatically returns the old news or old news group with similarity and the respective relevance score.

[0044] It should be noted that when there is no first query set or second query set or old news and / or old news group with similar content, S104-S104 are no longer performed.

[0045] S103: selecting similar news of the newly-stored news from the old news and / or old news group according to the relevance score.

[0046] In some embodiments of the present application, the highest relevance score is selected.

[0047] Then, the highest relevance score is compensated according to a preset score compensation coefficient to obtain a compensated relevance score. The preset score compensation coefficient is multiplied by the highest relevance score to obtain the compensated relevance score.

[0048] It should be noted that the compensation coefficient is greater than 0 and less than 1, for example, the score coefficient is 0.95.

[0049] Finally, similar news of the newly-stored news is selected from the old news and / or old news group according to the compensated relevance score and the highest relevance score.

[0050] The process of selecting similar news of the newly-stored news from the old news and / or old news group according to the compensated relevance score and the highest relevance score is as follows:

[0051] First, a score interval is constructed according to the compensated relevance score and the highest relevance score. Old news and / or old news group with a relevance score in the score interval are determined as target old news and / or target old news group with a relevance score in the score interval.

[0052] Determine the similar news of the target old news and / or each old news of the target old news group as the newly-stored news.

[0053] S104: Construct a news group according to the newly-stored news and the similar news; the newly-stored news is the main document of the news group, and the similar news is mounted in the news group as a sub-document through the association of the enterprise unique identifier.

[0054] For example, the newly-stored news a, the old news b, the old news group D, the main document d of the old news group D, and the sub-documents d1 and d2, b, d, d1 and d2 are sub-documents of the news group A, and the newly-stored news a is the main document of the news group A.

[0055] In some embodiments of the present application, in the news database, all the old news and / or old news groups that have been classified into news groups are deleted, and the news groups are stored in the news database.

[0056] In some embodiments of the present application, when the application end is queried, the main document news of the news group is displayed. When the similar news trigger condition of the news group is received, the similar news of the news group is displayed.

[0057] It should be noted that the form of the main document of the news group displayed in elasticsearch is completely consistent with the news, and the difference is that the news group mounts several sub-documents of its similar news.

[0058] In summary, in actual application, only the news in the main document needs to be displayed to meet the needs of most cases, so the similar news of the present application will not be queried out. When the user needs to deeply understand the background information of a certain news, the trigger button of all similar news can be clicked to realize the deep mining of information. In addition, in order to maintain the neatness of the database, all the original news entries that have been classified into news groups will be deleted to avoid data redundancy.

[0059] It should be noted that although the embodiments of the present application are introduced and described in sequence with reference to Figure 1 steps S101 to S104, this does not mean that steps S101 to S104 must be executed in strict sequence. The embodiments of the present application introduce and describe steps S101 to S104 in sequence as shown in Figure 1 in order to facilitate the understanding of the technical scheme of the embodiments of the present application by those skilled in the art. In other words, in the embodiments of the present application, the sequence of steps S101 to S104 can be adjusted as needed.

[0060] By Figure 1The method can realize the efficiency and accuracy of enterprise public opinion monitoring by intelligent processing and analyzing a large amount of news data through a distributed search engine and news identification index. The concept is as follows:

[0061] By using the news identification index (such as title, abstract, and publication time), considering multi-dimensional features, and using a combination of distributed search engines (such as elasticsearch) queries, the correlation score between new database news and old news and / or old news groups can be automatically calculated, and similar old news can be classified into new news groups according to the correlation score.

[0062] In addition, simple text matching or keyword search cannot avoid the problem of data redundancy, and all database original news classified into new news groups in the present application will be deleted to reduce data redundancy, which can optimize database management.

[0063] That is, the similarity between news can be judged according to the news title, abstract, and publication time, and similar news can be automatically classified into the corresponding news group. In addition, the new database news serves as the main document of the news group, so when displaying news on the application side, only the main document news (i.e., the latest news) in each news group needs to be displayed, greatly reducing the repeated display of similar news and improving user experience. At the same time, users can view all similar news of the displayed news by clicking on similar news, ensuring the comprehensiveness and accessibility of information.

[0064] In summary, the system is particularly suitable for enterprise credit scenarios that require rapid screening and classification of massive news information, providing more accurate and efficient services for customers.

[0065] Improve information retrieval efficiency: By analyzing multi-dimensional features such as news title, news abstract, and publication time, news reports with high similarity can be quickly screened and classified into corresponding news groups. This mechanism greatly improves the efficiency of users obtaining key information, especially when facing massive news data.

[0066] Reduce data redundancy: The system automatically merges repeated or highly similar news items into a news group and displays only the latest news in each group on the front end. This not only reduces the amount of information users need to browse, but also avoids the problem of key information being ignored due to information overload, thereby improving the effective utilization of information.

[0067] Improve user experience: An intuitive and user-friendly interface design allows users to easily browse news summaries and view all similar news by simple interactive actions (such as clicking on similar news). This approach ensures the comprehensiveness of information and enhances the user's exploration experience.

[0068] In summary, the accuracy and practicability of similar news recognition are significantly improved, and a more efficient and convenient information service platform is provided for enterprise and individual users.

[0069] Based on the same idea, some embodiments of the present application also provide a device and a non-volatile computer storage medium corresponding to the above method.

[0070] Figure 2 A structural schematic diagram of a similar news recognition device provided by an embodiment of the present application includes:

[0071] at least one processor; and

[0072] a memory in communication with the at least one processor; wherein

[0073] The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the similar news recognition method of any one of the above.

[0074] Some embodiments of the present application provide a non-volatile computer storage medium for similar news recognition, which stores computer executable instructions, and the computer executable instructions can perform the similar news recognition method of any one of the above.

[0075] Each of the embodiments in the present application is described in a progressive manner, and the same or similar parts of each of the embodiments can be referred to each other. Each of the embodiments mainly describes the difference from other embodiments. In particular, the device and medium embodiments are described simply because they are basically similar to the method embodiments, and the relevant parts can be referred to the part of the description of the method embodiments.

[0076] The device and medium provided by the embodiments of the present application correspond to the method, and therefore, the device and medium also have the similar beneficial technical effects as the method. Since the beneficial technical effects of the method have been described in detail above, the beneficial technical effects of the device and medium will not be described here.

[0077] Those skilled in the art should understand that the embodiments of the present application can be provided as a method, a system, or a computer program product. Therefore, the present application can be in the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application can be in the form of a computer program product implemented on one or more computer usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer usable program code.

[0078] The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart block or blocks. Figure 1 one or more flowcharts and / or blocks in the flowcharts and / or combination thereof. Figure 1 one or more flowcharts and / or blocks in the flowcharts and / or combination thereof.

[0079] The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart block or blocks. Figure 1 one or more flowcharts and / or blocks in the flowcharts and / or combination thereof. Figure 1 one or more flowcharts and / or blocks in the flowcharts and / or combination thereof.

[0080] The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart block or blocks. Figure 1 one or more flowcharts and / or blocks in the flowcharts and / or combination thereof. Figure 1 one or more flowcharts and / or blocks in the flowcharts and / or combination thereof.

[0081] In one typical configuration, the computing device includes one or more processors (CPUs), input / output interfaces, network interfaces, and memory.

[0082] The memory can include non-persistent memory and / or volatile memory, such as random access memory (RAM) and / or cache memory, non-volatile memory, such as read-only memory (ROM), EPROM, and / or flash memory. The memory is an example of computer-readable media.

[0083] Computer-readable media includes permanent and non-permanent, movable and non-movable media that can implement information storage by any method or technology. The information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassette, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information accessible to a computing device. According to the definition herein, computer-readable media does not include transitory media such as modulated data signals and carriers.

[0084] It should also be noted that the terms "comprising", "containing", or any other variant thereof are intended to cover non-exclusive inclusions, so that a process, method, article or apparatus that includes a list of elements does not only include those elements, but also includes other elements not explicitly listed, or further includes elements inherent in such a process, method, article or apparatus. Without more limitations, the element defined by the statement "comprising a" does not exclude the presence of additional identical elements in the process, method, article or apparatus that includes the element.

[0085] The above only describes the embodiments of the present application and is not intended to limit the present application. Those skilled in the art can make various modifications and changes to the present application. Any modification, equivalent replacement, improvement, etc. made within the technical principles of the present application shall fall within the protection scope of the present application.

Claims

1. A similar news identifying method characterized by comprising: The method comprises: parsing the newly-stored news to obtain news identification indexes; the news identification indexes comprise news titles, news abstracts, publication times, enterprise unique identifiers, and news unique identifiers; obtaining the relevance scores of the newly-stored news and old news and / or old news groups according to the distributed search engine and the news identification indexes; selecting similar news of the newly-stored news from the old news and / or old news groups according to the relevance scores; constructing a news group according to the newly-stored news and the similar news; the newly-stored news serves as a main document of the news group, and the similar news is mounted in the news group in the form of a sub-document through the association of the enterprise unique identifier.

2. The method of claim 1, wherein, The method of obtaining the relevance scores of the newly-stored news and old news and / or old news groups according to the distributed search engine and the news identification indexes specifically comprises: filtering old news and / or old news groups belonging to the same enterprise from a news database by using the term query of Elasticsearch to obtain a first query set; filtering out old news and / or old news groups meeting a time window from the first query set by using the range of Elasticsearch to limit the time window to obtain a second query set; filtering out content-similar old news and / or old news groups from the second query set by using the match query of Elasticsearch to query news titles and news abstracts and connecting them by using the should logic, and determining the relevance scores of the content-similar old news and / or old news calculated by Elasticsearch.

3. The method of claim 2, wherein, The method of filtering out old news and / or old news groups meeting a time window from the first query set to obtain a second query set specifically comprises: comparing the publication time of the newly-stored news with the publication times of the old news in the first query set to determine whether the old news meets the time window; and / or, comparing the publication time of the newly-stored news with the publication time of the main document news of the old news group in the first query set to determine whether the old news group meets the time window; assembling the old news and / or old news groups meeting the time window into the second query set.

4. The method of claim 1, wherein, The method of filtering out content-similar old news and / or old news groups from the second query set specifically comprises: calculating the similarity of the news titles and news abstracts of the newly-stored news with the news titles and news abstracts of the old news in the second query set to determine whether the old news has similarity; and / or, calculating the similarity of the news titles and news abstracts of the newly-stored news with the news titles and news abstracts of the main document news of the old news group in the second query set to determine whether the old news group has similarity; assembling the old news and / or old news groups having similarity into the second query set.

5. The method of claim 1, wherein, The method of selecting similar news of the newly-stored news from the old news and / or old news groups according to the relevance scores specifically comprises: selecting the highest relevance score; compensating the highest relevance score according to a preset score compensation coefficient to obtain a compensated relevance score; the compensation coefficient is greater than 0 and less than 1. According to the compensation correlation score and the highest correlation score, similar news of the newly-stored news is selected from old news and / or old news groups.

6. The method of claim 5, wherein, The selecting similar news of the newly-stored news from old news and / or old news groups according to the compensation correlation score and the highest correlation score specifically includes: According to the compensation correlation score and the highest correlation score, a score interval is constructed; Determine the target old news and / or target old news groups whose correlation scores are in the score interval, and determine each old news of the target old news and / or target old news groups as similar news of the newly-stored news.

7. The method of claim 6, wherein, After the constructing news groups according to the newly-stored news and the similar news, the method further includes: In the news database, delete all old news and / or old news groups that have been classified into news groups; Store the news groups into the news database.

8. The method of claim 7, wherein, After the storing the news groups into the news database, the method further includes: When queried by the application end, display the main document news of the news groups; When receiving the similar news trigger condition of the news groups, display the similar news of the news groups.

9. A similar news identifying apparatus characterized by comprising: It includes: At least one processor; And The memory is in communication connection with the at least one processor; wherein The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the similar news identification method of any one of claims 1-8.

10. A non-transitory computer storage medium storing computer-executable instructions that, when executed, cause a computer to perform: identifying similar news. The computer executable instructions can execute the similar news identification method of any one of claims 1-8. ​

Images