Content-based news duplicate removal method and device and nonvolatile storage medium

Abstract

The invention discloses a content-based news duplicate removal method and device and a nonvolatile storage medium. The method comprises the steps of obtaining a to-be-processed news set, deleting to-be-processed news with text structure similarity greater than or equal to preset similarity, and obtaining residual to-be-processed news; determining a content element and a category system of each piece of residual to-be-processed news, and classifying the plurality of pieces of residual to-be-processed news with the same content element and category system into a to-be-deduplicated set; for each piece of target residual to-be-processed news in the to-be-deduplicated set, triggering a text analysis model by adopting a first cue word to output a plurality of questions related to the target residual to-be-processed news to obtain a question list, and triggering the text analysis model by adopting a second cue word to output summary information of the target residual to-be-processed news; and according to the first semantic similarity of the question list and the second semantic similarity of the summary information, determining to-be-deleted news in the to-be-deduplicated set.

Description

Technical Field

[0001] This application relates to the field of information processing technology, and more specifically, to a content-based news deduplication method and apparatus, and a non-volatile storage medium. Background Technology

[0002] In related technologies, algorithms for deduplicating news articles are mainly designed for search scenarios. That is, when the content of two articles is highly similar, they are considered duplicates. When using the above methods, if the text recorded in two news articles is not repeated, even if the semantic analysis results indicate that the content of the two news articles is the same, the calculated similarity between the two news articles is very low, which makes it impossible to perform the deduplication operation. Therefore, there is a problem that it is not possible to deduplicat news articles with the same content but different text recording methods, and it is not applicable to various news deduplication scenarios.

[0003] There is currently no effective solution to the above problems. Summary of the Invention

[0004] This application provides a content-based news deduplication method and apparatus, as well as a non-volatile storage medium, to at least solve the technical problem that related technologies cannot perform deduplication operations on news articles with different text recording methods but essentially the same content.

[0005] According to one aspect of the embodiments of this application, a content-based news deduplication method is provided, comprising: acquiring a set of news to be processed, and determining the text structure similarity of each two news items in the set; deleting news items whose text structure similarity is greater than or equal to a preset similarity to obtain remaining news items; determining the content elements and category system of each remaining news item, and classifying multiple remaining news items with identical content elements and category systems into a deduplication set, wherein the content elements are used to summarize the events recorded in the remaining news items, and the category system is used to describe the domain to which the events described in the remaining news items belong; for each target remaining news item in the deduplication set, using a first prompt word to trigger a text analysis model to output multiple questions related to the target remaining news item to obtain a question list, and using a second prompt word to trigger a text analysis model to output summary information of the target remaining news item; and determining the news items to be deleted in the deduplication set based on the first semantic similarity of the question list and the second semantic similarity of the summary information.

[0006] Optionally, determining the text structure similarity of each pair of news items in the news collection includes: for each news item, performing text processing to obtain multiple strings that make up the news item, where the strings include: text and words; for each string, encoding the string into a first feature vector, and determining the weight of the string based on the frequency of the string appearing in the news item, where frequency and weight are positively correlated; determining multiple first feature vectors and multiple weights corresponding to the news item based on the multiple strings contained in the news item, and determining the fingerprint of the news item based on the multiple first feature vectors and multiple weights, where the fingerprint is used to describe the text structure features of the news item, and the text structure features include: the characters that make up the text and the arrangement of the characters; for each pair of news items, determining the text structure similarity based on the two fingerprints corresponding to the two news items.

[0007] Optionally, the content elements of each remaining news item to be processed are determined, including: using a third prompt word to trigger the text analysis model to output the content elements of the news item to be processed, wherein the third prompt word indicates the extraction of content elements from the news item to be processed, and the content elements include: the time of the news occurrence, the location of the news occurrence, and the subject of the news description.

[0008] Optionally, determining the category system for each remaining unprocessed news item includes: processing and analyzing the remaining unprocessed news item using a text analysis model to obtain multiple optional categories output by the text analysis model; iteratively updating the candidate category set based on the multiple optional categories until the ratio of the number of target categories in the optional categories to the number of categories contained in the candidate category set is less than a preset value, and outputting the candidate category set after the last iteration update, wherein the target category is a category not included in the candidate category set; generating the category system for the remaining unprocessed news item based on the multiple categories contained in the candidate category set, wherein the category system is in a multi-level form, and categories at different levels in the same category system correspond to different ranges in the same field.

[0009] Optionally, the candidate category set is iteratively updated based on the optional categories, including: during the first iteration update, the initial category system is determined as the candidate category set, the target category is determined from the multiple optional categories output by the text analysis model, and the target category is merged into the candidate category set; during subsequent iteration updates, the output result of the previous iteration update is determined as the candidate category set, the target category is determined from the multiple optional categories output by the text analysis model, and the target category is merged into the candidate category set.

[0010] Optionally, the initial category system is generated using the following method: triggering the text analysis model to output the news title and content of the news to be processed, and generating summary information of the news to be processed based on the news title, news content, and preset template; encoding the summary information of the news to be processed into a second feature vector, wherein the second feature vector is used to summarize the event described by the news to be processed; clustering multiple news items to be processed based on the second feature vector corresponding to each news item to obtain multiple vector sets; and determining the preset categories at different levels corresponding to each vector set in multiple preset categories to obtain the initial category system.

[0011] Optionally, the news items to be deleted are determined from the set of items to be deduplicated based on the first semantic similarity of the question list and the second semantic similarity of the summary information. This includes: for each pair of target remaining unprocessed news items corresponding to two question lists, determining the semantic similarity between each first question in one question list and each second question in the other question list to obtain multiple question semantic similarities; determining the first semantic similarity based on the multiple question semantic similarities; and, if the first semantic similarity is greater than the preset question similarity and the second semantic similarity is greater than the preset summary information similarity, determining any one of the two target remaining unprocessed news items as the news item to be deleted.

[0012] According to another aspect of the embodiments of this application, a content-based news deduplication apparatus is also provided, comprising: an acquisition module, configured to acquire a set of news to be processed, determine the text structure similarity of each two news items in the set, delete news items whose text structure similarity is greater than or equal to a preset similarity, and obtain remaining news items; a preliminary screening module, configured to determine the content elements and category system of each remaining news item, and classify multiple remaining news items with identical content elements and category systems into a deduplication set, wherein the content elements are used to summarize the events recorded in the remaining news items, and the category system is used to describe the domain to which the events described in the remaining news items belong; a processing module, configured to, for each target remaining news item in the deduplication set, use a first prompt word to trigger a text analysis model to output multiple questions related to the target remaining news item, obtain a question list, and use a second prompt word to trigger a text analysis model to output summary information of the target remaining news item; and a deduplication module, configured to determine news items to be deleted in the deduplication set based on the first semantic similarity of the question list and the second semantic similarity of the summary information.

[0013] According to another aspect of the embodiments of this application, a non-volatile storage medium is also provided, in which a computer program is stored, wherein the above-described content-based news deduplication method is executed by running the computer program on the device where the non-volatile storage medium is located.

[0014] According to another aspect of the embodiments of this application, an electronic device is also provided, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to execute the above-described content-based news deduplication method through the computer program.

[0015] According to another aspect of the embodiments of this application, a computer program product is also provided, including computer instructions that, when executed by a processor, implement the steps of the above-described content-based news deduplication method.

[0016] In this embodiment, a set of news items to be processed is obtained, and the text structure similarity of each pair of news items in the set is determined. News items with a text structure similarity greater than or equal to a preset similarity are deleted to obtain the remaining news items to be processed. The content elements and category system of each remaining news item are determined, and multiple remaining news items with identical content elements and category systems are grouped into a deduplication set. The content elements are used to summarize the events recorded in the remaining news items, and the category system is used to describe the domain to which the events described in the remaining news items belong. For each target remaining news item in the deduplication set, a first prompt word triggers the text analysis model to output multiple questions related to the target remaining news item to obtain a question list, and a second prompt word triggers the text analysis model to output the target remaining news item. Abstract information; based on the first semantic similarity of the question list and the second semantic similarity of the abstract information, the method of deduplicating news is determined in the deduplication set. The accuracy of deduplication is improved by two deduplication operations: coarse screening and fine screening. In the coarse screening, news articles with the same text recording style are deduplicated. In the fine screening, a language model with text processing and analysis functions is used to determine the category system to which the news belongs and extract the elements of the news. Based on the category system and the extracted elements, the content similarity between two news articles is calculated, and deduplication is performed based on the content similarity. This achieves the goal of deduplicating news articles with different text recording styles but the same content, thereby improving the accuracy of deduplication, enriching the application scenarios of deduplication, and solving the technical problem that related technologies cannot perform deduplication operations on news articles with different text recording styles but the same content. Attached Figure Description

[0017] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0018] Figure 1 This is a hardware structure block diagram of a computer terminal for implementing a content-based news deduplication method, based on relevant technologies.

[0019] Figure 2 This is a flowchart illustrating the steps of a content-based news deduplication method according to an embodiment of this application.

[0020] Figure 3 This is a structural diagram of a content-based news deduplication apparatus according to an embodiment of this application. Detailed Implementation

[0021] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0022] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0023] In related technologies, algorithms used for news deduplication include the Simhash algorithm and the KSentence algorithm. The Simhash algorithm for news deduplication involves: segmenting the text into words to obtain N-dimensional feature vectors; assigning weights to the word segments; calculating hash values ​​for the feature vectors; and then weighting and summing all feature vectors to obtain their text fingerprints. The similarity between two text fingerprints is then calculated using Hamming distance. The KSentence algorithm is based on the assumption that "the longest K sentences in two duplicate texts should be identical." The KSentence algorithm for news deduplication involves: selecting the K longest sentences in the article, concatenating them, and calculating the hash value (MD5 value) of the concatenated result as the text fingerprint. If the text fingerprints of two texts are identical, then the two texts are considered duplicates. The aforementioned algorithm only focuses on the similarity of text structure, and can only detect news articles with similar textual styles. For two news articles where the text is not repeated but the content analyzed is the same, using the above method to calculate similarity will result in a very low similarity score, making deduplication impossible. In other words, the deduplication algorithms used in related technologies are ineffective in the following situations: 1) significant differences in textual descriptions, but identical content; 2) different details but describing the same events. There is a problem that it cannot be applied to all news deduplication scenarios. To solve this problem, this application provides a related solution, which is described in detail below.

[0024] According to an embodiment of this application, a method embodiment for content-based news deduplication is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0025] The methods and embodiments provided in this application can be executed on mobile terminals, computer terminals, or similar computing devices. Figure 1 A hardware block diagram of a computer terminal for implementing a content-based news deduplication method is shown. Figure 1As shown, the computer terminal 10 may include one or more processors 102 (shown as 102a, 102b, ..., 102n in the figure) 102 (processor 102 may include, but is not limited to, a microprocessor MCU or a programmable logic device FPGA, etc.), a memory 104 for storing data, and a transmission device 106 for communication functions. In addition, it may also include: a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of a BUS bus), a network interface, a power supply, and / or a camera. Those skilled in the art will understand that... Figure 1 The structure shown is for illustrative purposes only and does not limit the structure of the aforementioned electronic device. For example, computer terminal 10 may also include... Figure 1 The more or fewer components shown, or having the same Figure 1 The different configurations shown.

[0026] It should be noted that the aforementioned one or more processors 102 and / or other data processing circuits are generally referred to herein as "data processing circuits". These data processing circuits may be embodied, in whole or in part, in software, hardware, firmware, or any other combination thereof. Furthermore, the data processing circuits may be a single, independent processing module, or may be integrated, in whole or in part, into any other element within the computer terminal 10. As involved in the embodiments of this application, the data processing circuits serve as a form of processor control (e.g., selection of a variable resistor termination path connected to an interface).

[0027] The memory 104 can be used to store software programs and modules of application software, such as the program instructions / data storage device corresponding to the content-based news deduplication method in this embodiment. The processor 102 executes various functional applications and data processing by running the software programs and modules stored in the memory 104, thereby realizing the above-mentioned content-based news deduplication method. The memory 104 may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include memory remotely located relative to the processor 102, and these remote memories can be connected to the computer terminal 10 via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0028] The transmission device 106 is used to receive or send data via a network. Specific examples of the network described above may include a wireless network provided by the communication provider of the computer terminal 10. In one example, the transmission device 106 includes a Network Interface Controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the transmission device 106 may be a Radio Frequency (RF) module, used for wireless communication with the Internet.

[0029] The display may be, for example, a touchscreen liquid crystal display (LCD) that allows the user to interact with the user interface of the computer terminal 10.

[0030] This application provides a content-based news deduplication method that can run in the above-described operating environment. Figure 2 This is a flowchart illustrating the steps of a content-based news deduplication method provided in the embodiments of this application, as follows: Figure 2 As shown, the method includes the following steps:

[0031] Step S202: Obtain the set of news to be processed, determine the text structure similarity of each two news items in the set, delete the news items whose text structure similarity is greater than or equal to the preset similarity, and obtain the remaining news items to be processed.

[0032] Before performing deduplication, the method provided in this application first determines the data sources. These data sources provide news to be processed, and the news provided by these data sources is news information from the same target domain. In step S202, news to be processed is obtained from the aforementioned data sources, and a coarse screening is performed on the set of news to be processed composed of multiple news items. The events described in the news to be processed within the set all belong to the same target domain. The target domain is the largest domain in a domain architecture. For example, the architecture of the financial domain is "finance → financial macroeconomics, financial investment → financial macroeconomic statistics, financial macroeconomic financial statement analysis," then the target domain is finance. The target domain can also be science and technology, society, etc. In this application embodiment, the target domain to be analyzed is determined first, and then news to be processed is obtained from the data sources that provide news for the target domain, and then deduplication is performed. When the target domain is finance, the news to be processed obtained from the data sources includes: news describing macroeconomic events, news recording macroeconomic statistics, and news recording financial statement analysis. In step S202, a coarse screening operation is performed on the set of news to be processed based on the text structure similarity of the news to be processed. Specifically, news to be processed with a text structure similarity greater than a preset similarity is deleted to complete the coarse screening operation. Subsequently, a fine screening operation is performed on the news to be processed with a text structure similarity less than a preset similarity that remains after the coarse screening operation (i.e., the remaining news to be processed).

[0033] According to some optional embodiments of this application, determining the text structure similarity of each pair of news items in the news collection to be processed includes: for each news item to be processed, performing text processing on the news item to be processed to obtain multiple strings that make up the news item to be processed, wherein the strings include: text and words; for each string, encoding the string into a first feature vector, and determining the weight of the string based on the frequency of the string appearing in the news item to be processed, wherein the frequency is positively correlated with the weight; determining multiple first feature vectors and multiple weights corresponding to the news item to be processed based on the multiple strings contained in the news item to be processed, and determining the fingerprint of the news item to be processed based on the multiple first feature vectors and multiple weights, wherein the fingerprint is used to describe the text structure features of the news item to be processed, and the text structure features include: the characters that make up the text and the arrangement of the characters; for each pair of news items to be processed, determining the text structure similarity based on the two fingerprints corresponding to the two news items to be processed.

[0034] In this embodiment, when performing a coarse screening of the news collection to be processed, the text structure similarity between each pair of news items can be determined using the following methods. For example, the Simhash algorithm can be used to determine the text structure similarity. The process of determining the text structure similarity can be summarized as follows: First, for each news item in the news collection to be processed, a text analysis tool is used to perform word segmentation, decomposing each news item into multiple strings. Each string is encoded to generate a feature vector (i.e., the first feature vector). Each string obtained from word segmentation can be a character or a word. For example, the news item "Analysis of the First Quarter Report of Bank X in Year Y" is segmented to obtain the following strings: "X", "bank", "Y", "first quarter", "report", "analysis". Next, encoding the strings to generate feature vectors can be achieved using a language model (such as BERT, Sentence-BERT). During encoding, each word or phrase in the string is mapped to an n-dimensional (predefined dimension) numerical vector space. Next, weights are assigned to each string based on its importance within the news text to be processed. For example, the term frequency-inverse document frequency (IF-IVF) method is used, determining the weight of a string based on its frequency of occurrence in the text; the more frequently a string appears, the greater its weight, reflecting its relative importance within the text structure. Further, a text fingerprint of the news is calculated based on the (first) feature vector and its weight for each string in the news text. For instance, the feature vectors of all strings in the news text are weighted and summed to obtain a sum vector. For each dimension of the sum vector, if the value of that dimension is positive, the bit value of the SimHash fingerprint at that position is 1; if the value is negative, the bit value is 0. Following this rule, the sum vector is converted into a fixed-length bit string, which is the fingerprint of the news text to be processed. Finally, for any two news items A and B in the news set to be processed, their text fingerprints FA and FB are calculated respectively; and similarity measurement methods (such as cosine similarity, Euclidean distance) are used to calculate the similarity between FA and FB. The similarity between fingerprints FA and FB is the text structure similarity between news item A and news item B.

[0035] Step S204: Determine the content elements and category system of each remaining unprocessed news item, and classify multiple remaining unprocessed news items with identical content elements and category systems into a set to be deduplicated. The content elements are used to summarize the events recorded in the remaining unprocessed news items, and the category system is used to describe the field to which the events described in the remaining unprocessed news items belong.

[0036] The first step in the fine-tuning process in step S204 is to use a language model with text analysis capabilities (i.e., a text analysis model) to extract the content elements of each remaining news item to be processed, and to determine the category system of each news item to be processed. Remaining news items to be processed that are completely identical in both category system and content elements are grouped into a set (i.e., a set to be deduplicated). The content elements include information such as the time of the news occurrence, the location of the news occurrence, and the subject of the news description, which summarize the events recorded in the news item to be processed. The category system of each news item to be processed describes the detailed domain to which the event described by the news item belongs. The domain described by the category system covers a smaller range than the target domain. For example, the category system can be represented in the form of an architecture (e.g., A→AB→BC). In this architecture, domain C is within the scope of domain B, and domain B is within the scope of domain A. For example, in the category system "Finance → Finance - Macroeconomics, Finance - Investment → Finance - Macroeconomics - Statistical Data, Finance - Macroeconomics - Financial Statement Analysis", finance is domain A, macroeconomics and investment are domain B, and statistical data and financial statement analysis are domain C. The category system hierarchy (L) is predefined and represents the depth of the category system. The category system A→AB→BC has 3 levels.

[0037] In step S204, the text analysis model can be loaded into memory. For example, the raw data of the text analysis model can be loaded from non-volatile memory into volatile memory so that the processor can run the text analysis model. The raw data of the text analysis model refers to unprocessed data, which typically includes the parameters and structural data of the text analysis model. The structural data can be the computational relationships based on the parameters, such as the forward propagation computational relationships between intermediate layers and between neurons. Specifically, the structural data can include the structure-related code of the text analysis model, such as code used to perform related calculations between intermediate layers and between neurons.

[0038] In one implementation, a region can be partitioned in memory for loading the text analysis model, which may include a structure data storage region and a parameter storage region. The structure data storage region stores structure-related code, and the parameters referenced by it can be accessed via pointers pointing to the addresses of specific parameters in the parameter storage region. During the training of the text analysis model, frequent parameter updates may be required; in this case, updating the parameter values ​​in the parameter storage region is sufficient.

[0039] According to some optional embodiments of this application, determining the content elements of each remaining news item to be processed includes: using a third prompt word to trigger the text analysis model to output the content elements of the news item to be processed, wherein the third prompt word indicates the extraction of content elements from the news item to be processed, and the content elements include: the time of the news occurrence, the location of the news occurrence, and the subject of the news description.

[0040] In step S204, determining the content elements of each remaining news item to be processed is based on a text analysis model. This model is a language model with text analysis capabilities. The language model is trained unsupervised on a large amount of text data. During training, the language model learns the complex structures and patterns of language; therefore, it possesses functions such as text generation, text understanding, and question answering. In this embodiment, the text analysis model processes and analyzes the remaining news items triggered by content element prompts (i.e., the third prompt). These prompts define the types of content elements. For example, when the prompt defines the content element type as news occurrence time, news occurrence location, and news description object (i.e., the subject), the prompt can be in the following form: "Extract the following elements from each news item: news occurrence time, location, and subject." In this embodiment, to ensure consistency, both news occurrence time and news occurrence location are extracted using the same format. For example, for a news article titled "Analysis of Bank X's Q1 Report in Year Y" with the content "Bank X's financial performance in Q1 of Year Y shows that its net profit increased year-on-year...", the content elements extracted by the text analysis model include: the time of the news (Q1 of Year Y), the location of the news (M), and the subject (Bank X's financial performance).

[0041] According to some optional embodiments of this application, determining the category system for each remaining unprocessed news item includes: processing and analyzing the remaining unprocessed news item using a text analysis model to obtain multiple optional categories output by the text analysis model; iteratively updating the candidate category set based on the multiple optional categories until the ratio of the number of target categories in the optional categories to the number of categories contained in the candidate category set is less than a preset value, and outputting the candidate category set after the last iteration update, wherein the target category is a category not included in the candidate category set; generating the category system for the remaining unprocessed news item based on the multiple categories contained in the candidate category set, wherein the category system is in a multi-level form, and categories at different levels in the same category system correspond to different ranges in the same field.

[0042] The determination of the category system for the remaining unprocessed news in step S204 is also based on a text analysis model. In this embodiment, the text analysis model can process multiple remaining unprocessed news items in parallel to simultaneously determine their category systems. In this embodiment, determining the category system for the remaining unprocessed news items relies on iterative updates to the candidate category set. The candidate category set contains multiple candidate categories. Since the news provided by the data source for obtaining the unprocessed news set belongs to the same target domain, the first-level category (the category with the largest coverage) in the multiple candidate categories is the same. For example, if the target domain is finance, each first-level category in the candidate category set is finance. The category system is refined during the iterative update process, increasing the number of levels within the category system. During the iterative update, the text analysis model performs text analysis on the unprocessed news items (i.e., the remaining unprocessed news items) whose text structure similarity is less than a preset similarity, outputting multiple optional categories (such as "finance," "macroeconomics," "investment," etc.). The optional categories output by the text analysis model are executed under the trigger of category classification prompts. Category classification prompts limit the text analysis object and the category classification steps. For example, if the category classification prompts limit the text analysis object to "news title and news content" and the category classification steps to "hierarchical classification", then the category classification prompts can be expressed in the following form: "Please read the title and content of the following news and classify it according to the category system." The category classification prompts also define the steps for hierarchical classification: "1. First, based on the first-level category of the category system, classify it into a specific category under that first-level category. If it is determined not to belong to any category, then define a custom first-level category for it based on the granularity of the first-level category. 2. Based on the second-level categories under the first-level category system, classify it into a specific category under that second-level category. If it is determined not to belong to any category, then define a custom second-level category for it based on the granularity of the second-level category. 3. Based on the third-level categories under the second-level category system, classify it into a specific category under that third-level category. If it is determined not to belong to any category, then define a custom third-level category for it based on the granularity of the third-level category." Hierarchical classification is performed... The number of categories is determined according to the hierarchy (L) of the category system. For example, in this embodiment, the hierarchy L of the category system is predefined as 3 during the initialization of the category system. The step of hierarchical classification includes three steps: S1, S2, and S3. The first-level category covers the largest scope, and the third-level category covers the smallest scope. For example, in the category system "Finance → Finance - Macroeconomics, Finance - Investment → Finance - Macroeconomics - Statistical Data, Finance - Macroeconomics - Financial Statement Analysis", finance is the first-level category, macroeconomics and investment are second-level categories with different categories, and statistical data and financial statement analysis are third-level categories with different categories. The optional category for the text analysis model to process and analyze the remaining news is the third-level category with the smallest scope.Each iteration of the category system update utilizes the output of the text analysis model (i.e., selectable categories) to update the candidate category set. Specifically, new categories (i.e., target categories) that are included in the text analysis model's output but not in the iterative update object are added to the iterative update object (i.e., the candidate category set), completing the iteration update. The stopping condition for the iteration update is: the ratio of the number of selectable categories output by the text analysis model to the number of categories contained in the candidate category set is less than a preset value (e.g., 2%). After stopping the iteration update, a category system for the remaining unprocessed news is generated based on the candidate category set output by the last iteration update. Specifically, for each remaining unprocessed news item to be classified, the text analysis model is used to determine its matching degree with each level of categories in the candidate category set. Based on the matching degree and the affiliation relationship of each level of categories corresponding to the remaining unprocessed news item, a multi-level classification structure (i.e., category system) for the remaining unprocessed news item is generated.

[0043] Optionally, the candidate category set is iteratively updated based on the optional categories, including: during the first iteration update, the initial category system is determined as the candidate category set, the target category is determined from the multiple optional categories output by the text analysis model, and the target category is merged into the candidate category set; during subsequent iteration updates, the output result of the previous iteration update is determined as the candidate category set, the target category is determined from the multiple optional categories output by the text analysis model, and the target category is merged into the candidate category set.

[0044] In this embodiment, when iteratively updating the candidate category set, if it is the first iteration, the update object (i.e., the candidate category set) is the initial category system. This initial category system is generated by initializing the category system of the news collection to be processed using a text analysis model. The initial category system contains multiple categories at different levels. During the update, new categories (i.e., target categories) not included in the initial category system are determined from the optional categories output by the text analysis model. The candidate category set is updated by adding these new categories to the candidate category set. If it is not the first iteration, the update object (i.e., the candidate category set) is the output result of the previous iteration (also a category set). The update object (i.e., the candidate category set) is updated by determining new categories (i.e., target categories) not included in the output result of the previous iteration from the optional categories output by the text analysis model. These new categories are then added to the output result of the previous iteration.

[0045] According to some alternative embodiments of this application, the initial category system is generated by the following method: triggering a text analysis model to output the news title and news content of the news to be processed, and generating summary information of the news to be processed based on the news title, news content and preset template; encoding the summary information of the news to be processed into a second feature vector, wherein the second feature vector is used to summarize the event described by the news to be processed; clustering multiple news items to be processed based on the second feature vector corresponding to each news item to be processed to obtain multiple vector sets; determining the preset categories at different levels corresponding to each vector set in multiple preset categories, thereby obtaining the initial category system.

[0046] The initial category system mentioned in the previous embodiment is the result of performing category system initialization. In this embodiment, the category system initialization process is as follows: K1 news items to be processed are randomly selected from the news collection to be processed. The text analysis model is used to output the news title and news content of each news item to be processed, and a summary of no more than N characters is generated based on the extracted news title and news content. The text analysis model outputs the news title and news content when triggered by a summary prompt. The summary prompt limits the maximum number of characters and the composition of the summary information. For example, if the maximum number of characters in the summary information is N, and the summary information includes the news title and news content, then the summary prompt can be in the following form: "Please read..." The following news headlines and content are analyzed and understood, and the most important elements are extracted to generate a clear and accurate summary in no more than N words. The text analysis model, triggered by summary information prompts, outputs: "News Headline: {News Headline}, News Content: {News Content}". Next, a summary template (i.e., a preset template) consisting of the news headline field, news content field, and other information, is generated based on the text analysis model's output triggered by summary information prompts. Specifically, the news headline is filled into the news headline field of the preset template, and the news content is filled into the news content field. After obtaining the summary information for each news item in the news collection, the summary information for each news item is encoded into a feature vector (i.e., a second feature vector) to capture the semantic features of the summary information. These feature vectors (i.e., the second feature vectors) generated from the summary information will be used for subsequent cluster analysis to identify the event type or theme described in different news articles. Next, clustering algorithms (such as K-Means clustering and DBSCAN) are used to perform clustering based on the similarity of the summary feature vectors (i.e., the second feature vectors). The result of the clustering is a set of vectors, each containing news events with similar content (i.e., news to be processed). Therefore, the set of vectors can also be called the event set, and the event set E = {e1, e2, ..., em}, where e1, e2, and em represent different summary feature vectors. The initial category system is generated based on a given set of preset categories and vectors. Hierarchical clustering is performed on the preset categories and vector sets, progressively determining multiple preset categories at different levels for each vector set. The category system corresponding to a vector set is generated based on the hierarchical relationship of these multiple preset categories. The hierarchical depth of the category systems generated for different vector sets may vary, but since the news in the news collection to be processed belongs to the same target domain, the first-level categories of these category systems are the same. These multiple category systems corresponding to multiple vector sets constitute the aforementioned initial category system; that is, the initial category system is a collection composed of category systems at different hierarchical depths.When determining multiple pre-defined categories at different levels for each vector set, a step-by-step clustering refinement can be achieved from top to bottom. The process is as follows: Initial clustering and first-level category assignment: By clustering the event set {e1, e2, ..., em}, each event vector is assigned to the most relevant first-level category, such as "Finance". This is the top-level classification. Second-level category refinement clustering: Within the framework of the first-level categories, further clustering analysis is performed on the event sets within each first-level category, forming more specific second-level categories, such as "Macroeconomics" and "Investment", achieving mid-level refinement of the classification. Third-level category deep clustering: When the predefined category system is a hierarchical structure of L=3, the event vectors under the second-level categories are clustered again. This step extends the classification to a deeper level, generating third-level categories such as "Macroeconomics / Statistical Data" and "Investment / Financial Statement Analysis", achieving bottom-level concretization of the classification. Through the above method of step-by-step clustering refinement, a hierarchical category system is constructed. Based on this category system, the diversity and detail of news content can be effectively captured.

[0047] Step S206: For each remaining unprocessed news item in the deduplication set, the first prompt word triggers the text analysis model to output multiple questions related to the remaining unprocessed news item, resulting in a question list. The second prompt word triggers the text analysis model to output a summary of the remaining unprocessed news item.

[0048] The second step of the fine-screening operation in step S206 involves using a text analysis model to generate a question list and summary information for each piece of news to be processed (i.e., the target remaining news to be processed) in the above deduplication set. The number of questions in the question list is predefined; for example, if it is predefined that 10 questions will be output for each target remaining news item, then the question list for each target remaining news item will contain 10 questions. Each question is output by the text analysis model when triggered by a question prompt word (i.e., the first prompt word). The question prompt word (i.e., the first prompt word) limits the number of questions output by the text analysis model and the limiting conditions for the questions. For example, when the number of questions is 10, the limiting condition is "with..." If the text content is closely related to the core information of the news, then the first prompt word could be in the following form: "Based on the following news content, generate 10 key questions that are closely related to the text content and can reflect the core information of the news content." The summary information for each remaining news item is output by the text analysis model under the trigger of the summary prompt word (i.e., the second prompt word). The summary prompt word limits the maximum number of characters and the composition of the summary information. For example, if the maximum number of characters in the summary information is N, and the summary information includes the news title and news content, then the second prompt word could be in the following form: "Please read the title and content of the following news, analyze and understand them, extract the most important elements, and generate a clear and accurate summary in no more than N characters."

[0049] For step S208, the news items to be deleted are determined from the deduplication set based on the first semantic similarity of the question list and the second semantic similarity of the summary information.

[0050] In step S208, the third step of the fine-screening operation is performed: for each pair of target remaining unprocessed news items in the deduplication set, the semantic similarity (i.e., the first semantic similarity) of the two question lists corresponding to these two target remaining unprocessed news items is determined, and the semantic similarity (i.e., the second semantic similarity) of the two summary information items corresponding to these two target remaining unprocessed news items is determined. Based on the semantic similarity of the question lists (i.e., the first semantic similarity) and the semantic similarity of the summary information (i.e., the second semantic similarity), it is determined whether these two target remaining unprocessed news items describe the same event. When two target remaining unprocessed news items do not describe the same event, one of the target remaining unprocessed news items is determined as a news item to be deleted. In step S208, all news items to be deleted in the deduplication set are determined using the above method.

[0051] Optionally, the news items to be deleted are determined from the set of items to be deduplicated based on the first semantic similarity of the question list and the second semantic similarity of the summary information. This includes: for each pair of target remaining unprocessed news items corresponding to two question lists, determining the semantic similarity between each first question in one question list and each second question in the other question list to obtain multiple question semantic similarities; determining the first semantic similarity based on the multiple question semantic similarities; and, if the first semantic similarity is greater than the preset question similarity and the second semantic similarity is greater than the preset summary information similarity, determining any one of the two target remaining unprocessed news items as the news item to be deleted.

[0052] In this embodiment, when determining whether there is a news item to be deleted among the two remaining target news items based on the semantic similarity of the question list (i.e., the first semantic similarity) and the semantic similarity of the summary information (i.e., the second semantic similarity), the system jointly determines whether the two remaining target news items describe the same event based on the first semantic similarity, the semantic similarity threshold of the question list, the second semantic similarity, and the semantic similarity threshold of the summary information. If the two remaining target news items describe the same event, one of the remaining target news items is determined to be a news item to be deleted. Specifically, if the first semantic similarity is greater than the preset question similarity (T1) and the second semantic similarity is greater than the preset summary information similarity (T2), then the content of the two remaining target news items is considered to be the same, and one of the news items is determined to be a news item to be deleted. The first semantic similarity is calculated after converting the question into a vector, and the second semantic similarity is calculated after converting the summary information into a vector.

[0053] The semantic similarity (i.e., the first semantic similarity) of the above question list can be the one with the largest numerical value among all question similarities. In this case, the question similarity with the largest numerical value is determined by the following method: Take each two news items in the deduplication set (i.e., the target remaining news items to be processed) as the analysis object. For example, the analysis objects are news A and news B. Use a text analysis model to generate 10 key questions closely related to the content of news A and news B respectively, resulting in the question list QA = {qA1, qA2, ..., qA10} and the question list QB = {qB1, qB2, ..., qB10} of news A. Use a semantic similarity calculation model (e.g., Sentence-BERT) to calculate the semantic similarity between each question in QA and each question in QB, forming a 10x10 semantic similarity matrix. Each element in the similarity matrix represents the semantic similarity (i.e., question similarity) between question qAi (any question in QA) and qBj (any question in QB). The element with the largest numerical value in the semantic similarity matrix is ​​determined as the first semantic similarity. The semantic similarity of the question lists (i.e., the first semantic similarity) can also be the absolute value of the similarities of all questions. In this case, the average value of all elements in the semantic similarity matrix is ​​determined as the first semantic similarity. The semantic similarity of the question lists (i.e., the first semantic similarity) can also be determined by finding the best matching combination of two question lists, with the following steps: S1. Simplify the semantic similarity matrix, ensuring that each row and column contains at least one 0 element: For each row and column of the semantic similarity matrix, subtract the smallest element of that row from each row element to ensure that each row has at least one 0; subtract the smallest element of that column from each column element to ensure that each column has at least one 0. S2. Covering Zero Elements: Cover all zero elements in the matrix with the minimum number of straight lines (horizontal or vertical) and determine if the optimal solution has been reached. If the number of lines covering zero elements is equal to the order of the semantic similarity matrix, the optimal solution has been found. If the number of lines is less than the order of the semantic similarity matrix, adjust the semantic similarity matrix as follows: find all elements not covered by lines and record their minimum value as k; subtract k from the uncovered elements, and add k to the elements covered by the intersection of two lines (keeping the existing zero elements while generating new zero elements). S3. Iterative Optimization Until the Optimal Solution is Determined: Repeat S2, covering the zero elements of the adjusted matrix again with the minimum number of lines, and again checking if the number of lines is equal to the order of the semantic similarity matrix. If it is equal to the order of the semantic similarity matrix, output the corresponding best matching combination; if it is still less than n, return to S2 to continue adjusting until the optimal condition is met. The semantic similarity between the two questions in the above best matching combination is the first semantic similarity.

[0054] By combining the above steps, we can identify news articles describing the same event by combining coarse screening based on text structure similarity and fine screening based on deep semantic similarity. This avoids the limitations of deduplication algorithms in related technologies when faced with news articles that have large differences in wording but are essentially the same, and improves the accuracy of deduplication operations.

[0055] Figure 3 This is a structural diagram of a content-based news deduplication apparatus provided in the embodiments of this application, such as... Figure 3 As shown, the content-based news deduplication device includes: an acquisition module 30, used to acquire a set of news to be processed, determine the text structure similarity of each two news items in the set, delete news items with a text structure similarity greater than or equal to a preset similarity, and obtain the remaining news items to be processed; a preliminary screening module 32, used to determine the content elements and category system of each remaining news item to be processed, and classify multiple remaining news items to be processed with the same content elements and category system into a deduplication set, wherein the content elements are used to summarize the events recorded in the remaining news items to be processed, and the category system is used to describe the domain to which the events described in the remaining news items to be processed belong; a processing module 34, used to, for each target remaining news item to be processed in the deduplication set, use a first prompt word to trigger the text analysis model to output multiple questions related to the target remaining news item to be processed, obtain a question list, and use a second prompt word to trigger the text analysis model to output the summary information of the target remaining news item to be processed; and a deduplication module 36, used to, based on the first semantic similarity of the question list and the second semantic similarity of the summary information, determine the news items to be deleted in the deduplication set.

[0056] It should be noted that, Figure 3 Preferred embodiments of the shown examples can be found in [reference needed]. Figure 2 The relevant descriptions of the embodiments shown will not be repeated here.

[0057] According to another aspect of the embodiments of this application, a non-volatile storage medium is also provided, in which a computer program is stored, wherein the above-described content-based news deduplication method is executed by running the computer program on the device where the non-volatile storage medium is located.

[0058] The aforementioned non-volatile storage medium is used to store a program that performs the following functions: acquiring a set of news items to be processed, determining the text structure similarity of each pair of news items in the set, deleting news items with a text structure similarity greater than or equal to a preset similarity, and obtaining remaining news items to be processed; determining the content elements and category system of each remaining news item to be processed, and grouping multiple remaining news items with identical content elements and category systems into a deduplication set, wherein the content elements are used to summarize the events recorded in the remaining news items, and the category system is used to describe the domain to which the events described in the remaining news items belong; for each target remaining news item to be processed in the deduplication set, using a first prompt word to trigger the text analysis model to output multiple questions related to the target remaining news item to be processed, obtaining a question list, and using a second prompt word to trigger the text analysis model to output summary information of the target remaining news item to be processed; determining the news items to be deleted in the deduplication set based on the first semantic similarity of the question list and the second semantic similarity of the summary information.

[0059] According to another aspect of the embodiments of this application, an electronic device is also provided, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to execute the above-described content-based news deduplication method through the computer program.

[0060] The processor in the aforementioned electronic device is used to run a program that performs the following functions: acquiring a set of news items to be processed, determining the text structure similarity of each pair of news items in the set, deleting news items whose text structure similarity is greater than or equal to a preset similarity, and obtaining the remaining news items to be processed; determining the content elements and category system of each remaining news item to be processed, and classifying multiple remaining news items to be processed with identical content elements and category systems into a set to be deduplicated, wherein the content elements are used to summarize the events recorded in the remaining news items to be processed, and the category system is used to describe the domain to which the events described in the remaining news items to be processed belong; for each target remaining news item to be processed in the set to be deduplicated, using a first prompt word to trigger the text analysis model to output multiple questions related to the target remaining news item to be processed, obtaining a question list, and using a second prompt word to trigger the text analysis model to output summary information of the target remaining news item to be processed; determining the news items to be deleted in the set to be deduplicated based on the first semantic similarity of the question list and the second semantic similarity of the summary information.

[0061] According to another aspect of the embodiments of this application, a computer program product is also provided, including computer instructions that, when executed by a processor, implement the steps of the above-described content-based news deduplication method.

[0062] It should be noted that the modules in the above-mentioned content-based news deduplication device can be program modules (e.g., a set of program instructions to implement a certain function) or hardware modules. For the latter, it can be manifested in the following forms, but is not limited to them: each of the above modules is manifested as a processor, or the functions of each of the above modules are implemented by a processor.

[0063] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0064] In the above embodiments of this application, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0065] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units can be a logical functional division, and in actual implementation, there may be other division methods. For instance, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual coupling, direct coupling, or communication connection may be through some interfaces; the indirect coupling or communication connection between units or modules may be electrical or other forms.

[0066] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0067] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0068] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to related technologies, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.

[0069] The above description is only a preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A content-based news deduplication method, characterized in that, include: Obtain a set of news items to be processed, and determine the text structure similarity of each two news items in the set. Delete news items whose text structure similarity is greater than or equal to a preset similarity, and obtain the remaining news items to be processed. The content elements and category system of each remaining unprocessed news item are determined, and multiple remaining unprocessed news items with identical content elements and category systems are classified into a set to be deduplicated. The content elements are used to summarize the events recorded in the remaining unprocessed news items, and the category system is used to describe the field to which the events described in the remaining unprocessed news items belong. For each target remaining unprocessed news item in the set to be deduplicated, a first prompt word is used to trigger the text analysis model to output multiple questions related to the target remaining unprocessed news item, resulting in a question list. A second prompt word is then used to trigger the text analysis model to output summary information of the target remaining unprocessed news item. The news items to be deleted are determined from the set of items to be deduplicated based on the first semantic similarity of the list of questions and the second semantic similarity of the summary information.

2. The method according to claim 1, characterized in that, Determining the text structure similarity of each pair of news items in the set of news items to be processed includes: For each of the news items to be processed, text processing is performed on the news item to be processed to obtain multiple strings that make up the news item to be processed, wherein the strings include: text and words; For each string, the string is encoded into a first feature vector, and the weight of the string is determined based on the frequency of the string appearing in the news to be processed, wherein the frequency is positively correlated with the weight; Based on the multiple strings contained in the news to be processed, a plurality of first feature vectors and a plurality of weights corresponding to the news to be processed are determined, and a fingerprint of the news to be processed is determined based on the plurality of first feature vectors and the plurality of weights, wherein the fingerprint is used to describe the text structure features of the news to be processed, and the text structure features include: the characters that make up the text and the arrangement of the characters; For each pair of news items to be processed, the text structure similarity is determined based on the two fingerprints corresponding to the two news items to be processed.

3. The method according to claim 1, characterized in that, Determine the content elements of each of the remaining unprocessed news items, including: The text analysis model is triggered by a third prompt word to output the content elements of the news to be processed. The third prompt word indicates that the content elements are extracted from the news to be processed. The content elements include: the time of the news occurrence, the location of the news occurrence, and the subject of the news description.

4. The method according to claim 1, characterized in that, Determine the category system for each of the remaining unprocessed news items, including: The remaining unprocessed news is processed and analyzed using the text analysis model to obtain multiple selectable categories output by the text analysis model; The candidate category set is iteratively updated according to multiple optional categories until the ratio of the number of target categories in the optional categories to the number of categories contained in the candidate category set is less than a preset value. The candidate category set after the last iteration update is output, wherein the target category is a category not included in the candidate category set. The remaining news items to be processed are generated based on the multiple categories contained in the candidate category set. The category system is multi-level, and different levels of categories in the same category system correspond to different ranges of the same field.

5. The method according to claim 4, characterized in that, Iteratively update the candidate category set based on the available categories, including: During the first iteration update, the initial category system is determined as the candidate category set, the target category is determined from the multiple optional categories output by the text analysis model, and the target category is merged into the candidate category set; During non-first iteration updates, the output of the previous iteration update is determined as the candidate category set, the target category is determined from the multiple optional categories output by the text analysis model, and the target category is merged into the candidate category set.

6. The method according to claim 5, characterized in that, The initial category system was generated according to the following method: The text analysis model is triggered to output the news title and news content of the news to be processed, and a summary of the news to be processed is generated based on the news title, the news content and the preset template; The summary information of the news to be processed is encoded into a second feature vector, wherein the second feature vector is used to summarize the event described by the news to be processed; Clustering is performed on multiple news items to be processed based on the second feature vector corresponding to each news item to be processed, resulting in multiple vector sets; The initial category system is obtained by determining the different levels of the preset categories corresponding to each vector set in multiple preset categories, level by level.

7. The method according to claim 1, characterized in that, Based on the first semantic similarity of the question list and the second semantic similarity of the summary information, news items to be deleted are determined from the set of items to be deduplicated, including: For each pair of remaining unprocessed news items corresponding to two target items, determine the semantic similarity between each first question in one question list and each second question in the other question list to obtain multiple semantic similarities of the questions; and determine the first semantic similarity based on the multiple semantic similarities of the questions. If the first semantic similarity is greater than the preset question similarity and the second semantic similarity is greater than the preset summary information similarity, then either of the two remaining unprocessed target news items will be determined as the news item to be deleted.

8. A content-based news deduplication device, characterized in that, include: The acquisition module is used to acquire a set of news to be processed, determine the text structure similarity of each two news items in the set, delete the news items whose text structure similarity is greater than or equal to a preset similarity, and obtain the remaining news items to be processed. The preliminary screening module is used to determine the content elements and category system of each of the remaining unprocessed news items, and to classify multiple remaining unprocessed news items with the same content elements and category system into a set to be deduplicated. The content elements are used to summarize the events recorded in the remaining unprocessed news items, and the category system is used to describe the field to which the events described in the remaining unprocessed news items belong. The processing module is used to, for each target remaining unprocessed news item in the deduplication set, use a first prompt word to trigger the text analysis model to output multiple questions related to the target remaining unprocessed news item, obtain a question list, and use a second prompt word to trigger the text analysis model to output summary information of the target remaining unprocessed news item; The deduplication module is used to determine the news items to be deleted from the set of items to be deduplicated based on the first semantic similarity of the list of questions and the second semantic similarity of the summary information.

9. A non-volatile storage medium, characterized in that, The non-volatile storage medium stores a computer program, wherein the device containing the non-volatile storage medium executes the content-based news deduplication method according to any one of claims 1 to 7 by running the computer program.

10. An electronic device comprising a memory and a processor, characterized in that, The memory stores a computer program, and the processor is configured to execute the content-based news deduplication method according to any one of claims 1 to 7 through the computer program.

11. A computer program product comprising computer instructions, characterized in that, When the computer instructions are executed by the processor, they implement the steps of the content-based news deduplication method according to any one of claims 1 to 7.

Images