Information processing method, device, equipment and storage medium

By calculating the similarity between the pending text and stored text on the network platform and correcting the similarity based on release time, the efficiency and accuracy of finding duplicate information are solved, improving user experience and saving resources.

CN113536763BActive Publication Date: 2024-11-05北京中科闻歌科技股份有限公司
View PDF 19 Cites 0 Cited by

Patent Information

Application Number
CN202110821206.0
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2021-07-20
Publication Date
2024-11-05
Estimated Expiration
2041-07-20

AI Technical Summary

Technical Problem

The prior art is difficult to quickly and accurately find duplicate information on the network platform, resulting in poor user information viewing experience and occupies a lot of memory resources.

Method used

By obtaining the text similarity between the pending text and the stored text, and correcting the similarity based on the release time, if the corrected similarity is greater than or equal to the preset threshold, the stored text is used as the similar text to achieve rapid filtering.

Benefits of technology

It improves the screening efficiency and accuracy of similar texts, reduces the frequency of displaying duplicate information, improves the user information viewing experience, and saves equipment resources.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN113536763B_ABST
    Figure CN113536763B_ABST
Patent Text Reader

Abstract

The present disclosure relates to an information processing method, device, equipment and storage medium. The information processing method includes: obtaining a text to be processed; determining the text similarity between the text to be processed and the stored text; based on the release time of the text to be processed and the release time of the stored text, correcting the text similarity to obtain a corrected text similarity; when the corrected text similarity is greater than or equal to a preset similarity threshold, using the stored text as a similar text of the text to be processed. According to the embodiments of the present disclosure, the search accuracy and efficiency of duplicate information can be improved and equipment resources can be saved, so as to further improve the processing effect and efficiency of duplicate information, reduce the repetitiveness of text content, and thereby enable users to quickly and accurately find information of interest from the network platform, improve the user's experience of viewing information and save equipment resources.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present disclosure relates to the field of information processing technology, and in particular to an information processing method, device, equipment and storage medium. Background Art

[0002] With the advent of the big data era, online platforms are gradually becoming mature, and users can easily share and view information of interest on online platforms.

[0003] In order to avoid displaying duplicate information on the same page, the network platform needs to process the duplicate information. However, in the related art, the network platform cannot quickly and accurately find the duplicate information, so the repetitiveness of the displayed content is still high, resulting in users not being able to quickly and accurately find the information of interest from the network platform, affecting the user's information viewing experience, and the process of finding duplicate information takes up a lot of memory, wasting device resources. Summary of the invention

[0004] In order to solve the above technical problems or at least partially solve the above technical problems, the present disclosure provides an information processing method, apparatus, device and storage medium.

[0005] The present disclosure provides an information processing method, the method comprising:

[0006] Get the text to be processed;

[0007] Determining text similarity between the text to be processed and the stored text;

[0008] Based on the publishing time of the text to be processed and the publishing time of the stored text, the text similarity is corrected to obtain a corrected text similarity;

[0009] When the corrected text similarity is greater than or equal to a preset similarity threshold, the stored text is used as a similar text to the text to be processed.

[0010] The present disclosure provides an information processing device, the device comprising:

[0011] A module for obtaining text to be processed, used for obtaining text to be processed;

[0012] A text similarity determination module, used to determine the text similarity between the text to be processed and the stored text;

[0013] A text similarity correction module is used to correct the text similarity based on the publishing time of the text to be processed and the publishing time of the stored text to obtain the corrected text similarity;

[0014] The similar text determination module is used to use the stored text as a similar text to the text to be processed when the corrected text similarity is greater than or equal to a preset similarity threshold.

[0015] An embodiment of the present invention further provides an information processing device, the device comprising:

[0016] one or more processors;

[0017] a storage device for storing one or more programs,

[0018] When the one or more programs are executed by the one or more processors, the one or more processors implement the information processing method provided by any embodiment of the present invention.

[0019] An embodiment of the present invention further provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the information processing method provided by any embodiment of the present invention is implemented.

[0020] Compared with the prior art, the technical solution provided by the embodiments of the present disclosure has the following advantages:

[0021] An information processing method, device, equipment and storage medium of the disclosed embodiment can calculate the text similarity between the to-be-processed text and the stored text after obtaining the to-be-processed text, and correct the text similarity according to the release time of the to-be-processed text and the release time of the stored text to obtain the corrected text similarity. When the corrected text similarity is greater than or equal to the preset similarity threshold, the stored text is used as the similar text of the to-be-processed text. First, since the process of determining the text distance and the process of correcting the text distance based on the release time have a small amount of calculation and a small memory occupation, the amount of calculation for determining the corrected text similarity is very small, so the screening efficiency of similar texts can be improved and equipment resources can be saved. Secondly, the corrected text similarity can not only reflect the similarity of the content of the two texts, but also reflect the time relevance of the two texts, so the screening accuracy of similar texts can be improved. In summary, the disclosed embodiment can quickly and accurately find similar texts, i.e., repeated information, and then can reduce the repetitiveness of the displayed content when displaying content to users, so that users can quickly and accurately find information of interest from the network platform, improve the user's information viewing experience and save equipment resources. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0023] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0024] Figure 1 An architectural diagram of an information processing method provided by an embodiment of the present disclosure;

[0025] Figure 2 A flowchart of an information processing method provided by an embodiment of the present disclosure;

[0026] Figure 3 A flowchart of another information processing method provided by an embodiment of the present disclosure;

[0027] Figure 4 A flowchart of another information processing method provided by an embodiment of the present disclosure;

[0028] Figure 5 A flowchart of a method for searching for identical texts provided in an embodiment of the present disclosure;

[0029] Figure 6 A schematic diagram of a flow chart of an information clustering method provided in an embodiment of the present disclosure;

[0030] Figure 7 A flowchart of another information processing method provided by an embodiment of the present disclosure;

[0031] Figure 8 A schematic flow chart of an information clustering method of a minimum hash clustering module combined with a time decay coefficient provided in an embodiment of the present disclosure;

[0032] Fig. 9 A schematic diagram of the structure of an information processing device provided by an embodiment of the present disclosure;

[0033] Fig.10 A schematic diagram of the structure of an information processing device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0034] In order to more clearly understand the above-mentioned objectives, features and advantages of the present disclosure, the scheme of the present disclosure will be further described below. It should be noted that the embodiments of the present disclosure and the features in the embodiments can be combined with each other without conflict.

[0035] In the following description, many specific details are set forth to facilitate a full understanding of the present disclosure, but the present disclosure may also be implemented in other ways different from those described herein; it is obvious that the embodiments in the specification are only part of the embodiments of the present disclosure, rather than all of the embodiments.

[0036] In recent years, online platforms such as Weibo, Tieba and various search engines have gradually matured. Users can share opinions and insights on online platforms and use the forwarding function of online platforms to publish content they like, which improves the efficiency of information dissemination. Users can also view information of interest from the published information.

[0037] However, as the amount of information shared or published by users increases, the same information appears repeatedly on different display pages (such as Weibo pages) of the network platform, making the information displayed on the display pages highly repetitive. If users want to find interesting or valuable information from the display pages, the cost of obtaining information for users will be greatly increased, affecting the user's experience of viewing information.

[0038] In order to avoid displaying duplicate information on the same page, the network platform can perform deduplication processing on the information posted or forwarded by users to reduce the repetitiveness of the information displayed on the display page. Information deduplication is equivalent to clustering, which means that for a given text, according to the similarity between texts, the given text is grouped into a certain category, so that similar texts are clustered in the same category, and dissimilar texts are scattered in different categories.

[0039] At present, traditional clustering methods are generally used for text clustering. The traditional clustering method is: first, the semantic content of the text is vectorized, and then similar texts are aggregated by calculating the similarity between the texts. Among them, traditional clustering methods include: K-means clustering algorithm, deduplication method based on text semantic similarity, and text deduplication method based on SimHash. However, the K-means clustering algorithm requires the number of classes to be specified in advance, which cannot meet the needs of deduplication of billions of information per day; the deduplication method based on text semantic similarity can support streaming data deduplication, but the time complexity is high, and the number of devices needs to be continuously increased to support the deduplication function; the text deduplication method based on SimHash extracts keywords from the text as feature words, and uses the local sensitive hash function to map the words to fingerprints, and finally compares the fingerprints of each document. If the fingerprint difference is small, the two articles are considered similar, but SimHash is suitable for deduplication of long texts and performs poorly on short texts.

[0040] It can be seen that due to the above-mentioned problems, the network platform cannot accurately and quickly find duplicate information, and the processing effect of duplicate information is not good, so that the displayed text content is still highly repetitive, resulting in users being unable to quickly and accurately find information of interest from the network platform, affecting the user's experience of viewing information.

[0041] In order to solve the above problems, the embodiments of the present disclosure provide an information processing method, apparatus, information processing device and storage medium that can accurately and quickly find duplicate information.

[0042] Figure 1 The figure shows an architecture diagram of an information processing method provided by an embodiment of the present disclosure.

[0043] like Figure 1 As shown, the architecture diagram may include an electronic device 101 and a server 102. The electronic device 101 may establish a connection with the server 102 and exchange information through a network protocol such as Hyper Text Transfer Protocol over Secure Socket Layer (HTTPS). The electronic device 101 may include a mobile phone, a tablet computer, a desktop computer, a laptop computer, a vehicle terminal, a wearable device, an all-in-one machine, a smart home device, and other devices with communication functions, and may also include a virtual machine or a device simulated by a simulator. The server 102 may be a cloud server or a server cluster, and other devices with storage and computing functions.

[0044] Based on the above architecture, when performing information processing, the server 102 can obtain the text to be processed sent by the electronic device 101, calculate the text similarity between the text to be processed and the stored text, and correct the text similarity based on the release time of the text to be processed and the release time of the stored text to obtain the corrected text similarity. Further, when the corrected text similarity is greater than or equal to a preset similarity threshold, the stored text is used as a similar text of the text to be processed.

[0045] In other embodiments, the architecture diagram of an information processing method provided by the embodiment of the present disclosure may also include a database.

[0046] The database may also establish a connection with the server 102 and perform information exchange via a network protocol such as HyperText Transfer Protocol over Secure Socket Layer (HTTPS).

[0047] Optionally, the database may include a relational database, a non-relational database, or other data collection having a large data storage function.

[0048] Based on the above architecture, when performing information processing, the server 102 can obtain the text to be processed sent by the electronic device 101, and obtain the stored text from the database, and determine the text similarity between the text to be processed and the stored text, and based on the release time of the text to be processed and the release time of the stored text, correct the text similarity to obtain the corrected text similarity. When the corrected text similarity is greater than or equal to the preset similarity threshold, the stored text is used as the similar text of the text to be processed.

[0049] In the embodiments of the present disclosure, since similar texts are determined based on text similarity that reflects the degree of similarity between the contents of two texts and text publishing time that reflects the temporal relevance of the two texts, the accuracy of determining similar texts is improved when determining the corrected text similarity. Furthermore, since the computational complexity of the process of determining the text similarity and the process of correcting the text similarity based on the publishing time is very small, the computational complexity of determining the corrected text similarity is very small, thereby improving the efficiency and accuracy of searching for duplicate information.

[0050] Therefore, based on the above architecture, the search efficiency and accuracy of duplicate information can be improved, so as to further improve the processing effect and efficiency of duplicate information, reduce the repetitiveness of text content, meet the needs of users to quickly and accurately find information of interest from the network platform, and enhance the user's experience of viewing information.

[0051] According to the above architecture, the following Figures 2 to 8 The information processing method provided by the embodiment of the present disclosure is described. In the embodiment of the present disclosure, the information processing method can be executed by a server. In the embodiment of the present disclosure, the server can be Figure 1 The server 102 shown in FIG. The server 102 may be a cloud server or a server cluster or other device with storage and computing functions.

[0052] Figure 2 A flowchart of an information processing method provided by an embodiment of the present disclosure is shown.

[0053] S210: Obtain the text to be processed.

[0054] Specifically, after the electronic device publishes a new text, the newly published text can be used as the text to be processed, and the text to be processed can be sent to the server, so that the server obtains the text to be processed and searches for text that is repeated with the text to be processed.

[0055] Optionally, the text type of the text to be processed may include: long text type, short text type, etc., which is not limited here.

[0056] In the embodiment of the present disclosure, the server can simultaneously receive a large amount of text to be processed through multiple processing nodes or multiple processing threads, and search for similar texts of each text to be processed based on each processing node or each processing thread.

[0057] S220: Determine the text similarity between the text to be processed and the stored text.

[0058] Specifically, after the server obtains the text to be processed, it can obtain the stored text from the database to determine the text similarity between the text to be processed and the stored text.

[0059] In the embodiment of the present disclosure, the stored text may be a historically published text after similarity determination has been performed.

[0060] In the disclosed embodiment, the text similarity can represent the similarity between the text content of the to-be-processed text and the stored text. The higher the text similarity, the higher the similarity between the text content of the to-be-processed text and the stored text, and the lower the text similarity, the lower the similarity between the text content of the to-be-processed text and the stored text.

[0061] In some embodiments of the present disclosure, determining the text similarity between the to-be-processed text and the stored text may include:

[0062] Perform word segmentation on the text to be processed and the stored text respectively;

[0063] Determine the number of intersection words and the number of union words of the text to be processed and the stored text;

[0064] Divide the number of intersection words by the number of union words to get the text similarity.

[0065] The intersection words may be words that appear in both the text to be processed and the stored text.

[0066] The union vocabulary may be all the vocabulary in the text to be processed and the stored text.

[0067] In the disclosed embodiment, the server can perform word segmentation on the text to be processed based on one or more algorithms of the word graph scanning algorithm of the prefix dictionary, the hidden Markov algorithm, and the dynamic programming algorithm to obtain the vocabulary set of the text to be processed, and obtain the vocabulary set of the stored text from the database, then extract the number of intersection words and the number of union words from the vocabulary set of the text to be processed and the vocabulary set of the stored text, divide the number of intersection words and the number of union words, and obtain the text similarity, which is equivalent to the Jaccard similarity. That is, the Jaccard similarity can be calculated according to the following formula:

[0068]

[0069] Among them, J(A,B) is the Jaccard similarity, A is the vocabulary set of the text to be processed, B is the vocabulary set of the stored text, |A∩B| is the number of intersection vocabulary, and |A∪B| is the number of union vocabulary.

[0070] S230 , based on the publishing time of the text to be processed and the publishing time of the stored text, the text similarity is corrected to obtain a corrected text similarity.

[0071] Specifically, after the server obtains the text to be processed, it can obtain the text publishing time of the text to be processed and the stored text respectively, so as to correct the text similarity based on the publishing time of the text to be processed and the publishing time of the stored text, and obtain the corrected text similarity.

[0072] In the embodiment of the present disclosure, the text publishing time may be the time when the text is published on the network platform.

[0073] Taking the Weibo platform as an example, the texts published on the Weibo platform have strong timeliness. If the text publishing time of two texts is very different, the probability of discussing the same content is smaller. Therefore, after obtaining the text to be processed, the text publishing time of the text to be processed and the stored text can be obtained respectively, so as to correct the text similarity based on the text publishing time of the text to be processed and the stored text, and obtain the corrected text similarity.

[0074] Specifically, after calculating the text similarity, the server can calculate the difference in publishing time between the publishing time of the text to be processed and the publishing time of the stored text, and reduce the probability of classifying texts with similar vocabulary but discussing different contents according to the publishing time difference, so as to correct the text similarity and obtain the corrected text similarity.

[0075] In the disclosed embodiment, the corrected text similarity can be a degree of similarity between the text content of the to-be-processed text and the stored text based on timeliness relevance. The higher the corrected text similarity, the higher the degree of similarity between the text content of the to-be-processed text and the stored text based on timeliness relevance, and the lower the corrected text similarity, the lower the degree of similarity between the text content of the to-be-processed text and the stored text based on timeliness relevance.

[0076] Optionally, the corrected text similarity may be less than or equal to the text similarity.

[0077] S240: When the corrected text similarity is greater than or equal to a preset similarity threshold, the stored text is used as a similar text to the text to be processed.

[0078] In the embodiment of the present disclosure, the preset similarity threshold may be a preset minimum similarity probability that represents text similarity. Optionally, the preset similarity threshold may be a value such as 0.85, 0.9, etc., which is not limited here.

[0079] Specifically, after the server determines the corrected text similarity, it can compare the corrected text similarity with a preset similarity threshold. If the corrected text similarity is greater than or equal to the preset similarity threshold, it is determined that the stored text is similar to the text to be processed, and the stored text is used as the similar text of the text to be processed.

[0080] In the embodiment of the present disclosure, after obtaining the text to be processed, the text similarity between the text to be stored and the stored text can be calculated, and the text similarity can be corrected according to the release time of the text to be processed and the release time of the stored text to obtain the corrected text similarity. When the corrected text similarity is greater than or equal to the preset similarity threshold, the stored text is used as the similar text of the text to be processed. First, since the process of determining the text similarity and the process of correcting the text similarity based on the release time have a small amount of calculation and a small memory occupation, the amount of calculation for determining the corrected text similarity is very small, so the screening efficiency of similar texts can be improved and device resources can be saved. Secondly, the corrected text similarity can not only reflect the similarity of the content of the two texts, but also reflect the time relevance of the two texts. Therefore, the screening accuracy of similar texts can be improved. In summary, the embodiment of the present disclosure can quickly and accurately find similar texts, that is, repeated information, and then when displaying content to users, the repetitiveness of the displayed content can be reduced, so that users can quickly and accurately find information of interest from the network platform, improve the user's information viewing experience and save device resources.

[0081] In another embodiment of the present disclosure, in order to improve the calculation accuracy and calculation efficiency of the corrected text similarity, the corrected text similarity can be determined according to the following method to reduce the amount of calculation for determining the corrected text similarity, and improve the calculation accuracy of the corrected text similarity, thereby improving the efficiency and accuracy of searching for duplicate information.

[0082] In one embodiment of the present disclosure, in order to improve the calculation efficiency of text similarity and reduce the amount of calculation, the text similarity can be calculated based on the minimum hash signature of the text to be processed and the minimum hash signature of the stored text.

[0083] In some embodiments of the present disclosure, determining the text similarity between the to-be-processed text and the stored text may include:

[0084] Generate a minimal hash signature of the text to be processed;

[0085] Calculate the Hamming distance between the minimum hash signature of the text to be processed and the minimum hash signature of the stored text;

[0086] Determine text similarity based on Hamming distance.

[0087] Among them, the minimum hash signature of the text to be processed can be a text signature determined based on the MinHash algorithm (minimum hash algorithm).

[0088] Among them, the minimum hash signature of the stored text can be a text signature determined and stored in advance based on the MinHash algorithm (minimum hash algorithm).

[0089] The Hamming distance can be the number of different bits between two hash signatures of the same length, and can be used to characterize the similarity between two texts. If the Hamming distance is smaller, it means that the text content of the to-be-processed text is more similar to the text content of the stored text, that is, the text similarity is higher. If the Hamming distance is larger, it means that the text content of the to-be-processed text is less similar to the text content of the stored text, that is, the text similarity is lower.

[0090] Specifically, before calculating the minimum hash signature of the text to be processed, the server can perform word segmentation on the text to be processed based on one or more algorithms selected from the word graph scanning algorithm of the prefix dictionary, the hidden Markov algorithm, and the dynamic programming algorithm to obtain the vocabulary set of the text to be processed. Then, the minimum hash signature of the text to be processed is calculated based on the MinHash algorithm, and the minimum hash signature of the stored text is obtained. The Hamming distance between the minimum hash signature of the text to be processed and the minimum hash signature of the stored text is further calculated, and the text similarity is determined based on the Hamming distance.

[0091] In the embodiment of the present disclosure, when calculating the Jaccard similarity, the text content or one-hot encoding (One-Hot encoding) of each text may be stored.

[0092] In the disclosed embodiment, based on the MinHash algorithm, the positions of the words in the vocabulary set of the text to be processed can be evenly shuffled, and a preset number of random hash functions are used to map the one-hot vectors, and the first word in each vocabulary set in the new order is used as the characteristic value of the vocabulary set to calculate the minimum hash signature of the text to be processed. At the same time, the high-dimensional one-hot vector can be converted into a low-dimensional binary digital string, which reduces the memory usage and computing resources in the calculation process. Since the minimum hash signature belongs to a local sensitive hash, the hash signatures corresponding to relatively similar text contents are also relatively similar.

[0093] In the embodiment of the present disclosure, the text similarity determined according to the Hamming distance between the minimum hash signature of the text to be processed and the minimum hash signature of the stored text may be used as the Jaccard similarity.

[0094] Therefore, in the embodiment of the present disclosure, after the server obtains the text to be processed, it can quickly and accurately calculate the text similarity and reduce memory usage based on the minimum hash signature corresponding to the text to be processed and the stored text.

[0095] In another embodiment of the present disclosure, in order to further improve the calculation accuracy of the corrected text similarity, a time decay coefficient can be determined based on the publication time of the text to be processed and the publication time of the stored text, so as to further correct the text similarity based on the time decay coefficient to obtain the corrected text similarity.

[0096] In some embodiments of the present disclosure, based on the publishing time of the text to be processed and the publishing time of the stored text, the text similarity is corrected to obtain the corrected text similarity, which may include:

[0097] Calculate the publishing time difference between the publishing time of the text to be processed and the publishing time of the stored text;

[0098] Determine the time decay coefficient corresponding to the release time difference;

[0099] Multiply the text similarity by the time decay coefficient to get the corrected text similarity.

[0100] In the disclosed embodiment, the server may subtract the text publishing time of the stored text from the text publishing time of the to-be-processed text to obtain a publishing time difference.

[0101] The time decay coefficient can be used to reduce the probability that texts with similar vocabulary but different contents are classified into the same category. Specifically, the time decay coefficient can be used to correct the text similarity to obtain the corrected text similarity, so as to improve the screening accuracy of similar texts.

[0102] In some embodiments of the present disclosure, determining the time decay coefficient corresponding to the publishing time difference may include:

[0103] When the release time difference is less than the preset time threshold, the time decay coefficient is the preset decay coefficient;

[0104] When the release time difference is greater than or equal to a preset time threshold, a time decay coefficient is calculated according to the release time difference and a preset decay speed parameter.

[0105] The preset time threshold may be a preset maximum time difference used to characterize that two texts discuss the same content. Optionally, the preset time threshold may be 1 day, 3 days, 5 days, etc., which is not limited here.

[0106] The preset attenuation coefficient may be a preset fixed attenuation coefficient. Optionally, the preset attenuation coefficient may be a fixed value such as 1, 2, or 3.

[0107] The preset decay speed parameter may be a pre-set parameter for controlling the decay speed.

[0108] Therefore, in the embodiment of the present disclosure, after the server determines the text similarity, it determines the time decay coefficient according to the text publishing time, so as to further obtain the corrected text similarity based on the text similarity and the time decay coefficient, which can further improve the similarity calculation accuracy between the text to be processed and the stored text.

[0109] S230 is described in detail below by taking the example of calculating the time decay coefficient based on the publishing time difference between the publishing time of the text to be processed and the publishing time of the stored text, and correcting the text similarity based on the time decay coefficient to obtain the corrected text similarity.

[0110] In the embodiment of the present disclosure, optionally, the time attenuation coefficient may be determined by the following formula:

[0111]

[0112] Among them, λ is the time attenuation coefficient, 1 is the preset attenuation coefficient, Δt is the release time difference, T is the preset time threshold, β and α are the preset attenuation speed parameters respectively.

[0113] Therefore, when 0≤Δt≤T, the time decay coefficient is 1, that is, no decay coefficient is set, indicating that the release time difference between the text to be processed and the stored text is small, and the probability of discussing the same content is high, which means that the similarity does not decay in a short time range; when Δt≥T, the value of the time decay coefficient is This indicates that the publishing time difference between the text to be processed and the stored text is large, and the probability of discussing the same content is small. Moreover, as time goes by, the probability of discussing the same content gradually decreases, causing the corrected text similarity to gradually decrease.

[0114] In the embodiment of the present disclosure, optionally, the calculation formula of the modified text similarity can be:

[0115] dist(t,w)=ham min g t,w *λ t,w

[0116] Among them, dist(t,w) can be the target distance, that is, the corrected text similarity, ham min g t,w It can be the Hamming distance between the text to be processed and the stored text, that is, the text similarity, λ t,w It can be the time attenuation coefficient of the text to be processed and the stored text, t can be the text to be processed, and w can be the stored text.

[0117] Based on the above description, we can know that when 0≤Δt≤T, the text similarity ham min g t,w Multiply it by the time decay coefficient 1 to get the corrected text similarity, which is equal to the text similarity before correction. In the case of Δt ≥ T, the text similarity ham min g t,w With time decay coefficient Multiply them to get the corrected text similarity, so that the corrected text similarity is less than the text similarity before correction.

[0118] In summary, in the embodiments of the present disclosure, based on the minimum hash signature corresponding to the text to be processed and the stored text, the text similarity can be calculated quickly and accurately, and the memory usage can be reduced. Based on the calculated time decay coefficient, the text similarity is corrected to obtain the corrected text similarity, which can further improve the similarity calculation accuracy between the text to be processed and the stored text, thereby improving the search efficiency and accuracy of similar texts, so as to further improve the processing effect and efficiency of repeated information and reduce the repetitiveness of text content.

[0119] In order to cluster the text to be processed, the embodiment of the present disclosure also provides another information processing method. After finding similar texts to the text to be processed, the text to be processed can be clustered based on the similar texts, which can reduce the repetitiveness of the text content, thereby meeting the user's needs to quickly and accurately find information of interest from the network platform and improving the user's information viewing experience.

[0120] Figure 3 A flow chart of another information processing method provided by an embodiment of the present disclosure is shown.

[0121] like Figure 3 As shown, the information processing method may include the following steps.

[0122] S310: Obtain the text to be processed.

[0123] S320: Determine the text similarity between the text to be processed and the stored text.

[0124] In the disclosed embodiment, the stored text is the central text of the target text category, that is, the stored text may be the class center of the target text category, and the target text category may be the text category corresponding to the stored text.

[0125] In the disclosed embodiment, in order to improve the efficiency of text clustering and reduce storage pressure, a single-pass clustering algorithm can be used to cluster the text to be processed. The single-pass clustering algorithm is a concise and efficient text clustering algorithm. Each text only needs to traverse the existing categories once to complete the clustering. In addition, the single-pass clustering algorithm does not need to specify the number of classes in advance, and the number of clusters can be limited by setting a similarity threshold. Therefore, it is particularly suitable for streaming data clustering.

[0126] Among them, the clustering method of the single-pass clustering algorithm may include: taking the first text n1 as a seed, establishing a class c1, n1 as the class center of c1, and storing its id and signature in the database; for a new text n2, using a preset text similarity calculation method, calculating the similarity between n2 and the class centers of all existing classes, and finding the existing category with the greatest similarity to the text n2, and its similarity is recorded as s2; if s2 is greater than a preset similarity threshold, then adding the text n2 to the class with the greatest similarity, then terminating the clustering, and waiting for the next text to enter; if s2 is less than the threshold, the text n2 does not belong to any existing class, and a new category needs to be created, and n2 is used as the class center of the newly created category, waiting for the next text to enter for clustering.

[0127] S330 , based on the publishing time of the text to be processed and the publishing time of the stored text, the text similarity is corrected to obtain a corrected text similarity.

[0128] S340: When the corrected text similarity is greater than or equal to a preset similarity threshold, the stored text is used as a similar text to the text to be processed.

[0129] Among them, S310-S340 are similar to S210-S240 and will not be described in detail here.

[0130] S350: Add the text to be processed to the target text category.

[0131] Specifically, after finding similar text to the text to be processed, the server can add the text to be processed to the target text category to which the similar text belongs, that is, cluster the text to be processed according to the similar text to eliminate duplicate texts with repeated content, or, after adding the text to be processed to the target text category to which the similar text belongs, update the central text of the target text category according to the file to be processed to eliminate duplicate texts with repeated content.

[0132] In some embodiments, when the publishing time difference between the text to be processed and the similar text is greater than the time update threshold, after the text to be processed is added to the target text category to which the similar text belongs, the central text of the target text category is also updated according to the file to be processed to achieve duplicate elimination of texts with repeated content.

[0133] In other embodiments, when the publishing time difference between the text to be processed and the similar text is less than or equal to the time update threshold, the text to be processed is added to the target text category to which the similar text belongs, that is, the text to be processed is clustered according to the similar text to achieve duplicate elimination of texts with repeated content.

[0134] The time update threshold may be a preset time period for determining whether to update the core text of the target text category. Optionally, the time update threshold may be 3 days, 5 days, 7 days, etc.

[0135] In order to further improve clustering efficiency, in the embodiment of the present disclosure, after taking the stored text as the similar text of the text to be processed, the method further includes: taking the similar text corresponding to the maximum corrected text similarity as the target similar text of the text to be processed.

[0136] In some embodiments of the present disclosure, when the stored text is a massive amount of text, the server can traverse each stored text, determine the target text category corresponding to each stored text, calculate the corrected text similarity obtained in each traversal, and record the stored text whose corrected text similarity is greater than or equal to a preset similarity threshold as similar text. After traversing all the stored texts, if there are multiple similar texts in the text to be processed, the similar text corresponding to the largest corrected text similarity is used as the target similar text of the text to be processed, and the text to be processed is added to the target text category to which the target similar text belongs, so that the text to be processed and the unique similar text are clustered, which can further improve the clustering efficiency.

[0137] S360: When the corrected text similarity is less than a preset similarity threshold, the text to be stored is used as the central text of the new text category.

[0138] Specifically, when the corrected text similarity is less than the preset similarity threshold, the server can determine that the content similarity between the text to be stored and the stored text is low, and the text to be stored is directly used as the central text of the new text category; if the server compares the corrected text distance with the text distance of all stored texts, and when the corrected text distance is greater than or equal to the preset text distance threshold, the server can determine that the content similarity between the text to be stored and the stored text is low, and the text to be stored is directly used as the central text of the new text category.

[0139] Therefore, in the implementation example of the present disclosure, a single-pass clustering method is used to cluster the text to be processed, which can improve the clustering efficiency of the text to be processed, and the clustering process does not need to pre-specify the number of classes, which meets the needs of daily deduplication of hundreds of millions of microblogs. The number of clusters is limited by setting a similarity threshold, which is particularly suitable for streaming data clustering. In addition, the clustering method can be applied to cluster short texts posted on the Weibo platform and long texts posted on other network platforms, and has a wider range of applications.

[0140] Furthermore, after clustering the text to be processed, the server can send the clustering result to the electronic device, so that the clustered text information is displayed on the network platform of the electronic device, so that different information is displayed on the same display page of the network platform, and the repetitiveness of the information on the same page is reduced. In addition, the server can generate the number of repeated texts according to the clustering result, so that different information can be displayed on the same display page of the network platform, and the number of repetitions of each information can also be displayed. After receiving the user's touch operation, the repeated information is displayed accordingly.

[0141] Therefore, in the embodiment of the present disclosure, after determining similar texts, a single-pass clustering method can be used to add the text to be processed to the target text category to which the similar text belongs, or to add the text to be processed to the target text category to which the target similar text belongs, so as to cluster the text to be processed. This clustering method can improve the processing effect and efficiency of repeated information, can adapt to the needs of massive text clustering, does not need to increase the number of devices for clustering, and is suitable for the clustering needs of short texts, and has great promotion and application value.

[0142] The disclosed embodiment also provides another information processing method. Before determining the text similarity between the text to be processed and the stored text, the stored text that is identical to the text to be processed can be screened out based on the information summary signatures corresponding to the text to be processed and the stored text respectively, so as to improve the efficiency and accuracy of searching for duplicate information.

[0143] Figure 4 A flowchart of another information processing method provided by an embodiment of the present disclosure is shown.

[0144] As shown Figure 4 below, the information processing method may include the following steps.

[0145] S410. Obtain the text to be processed.

[0146] S420. Determine the information digest signature of the text to be processed.

[0147] In the embodiments of the present disclosure, the information digest signature of the text to be processed may be a signature obtained by extracting the message digest of the text to be processed based on the Message-Digest Algorithm and calculating. The Message-Digest Algorithm may be the MD5 Message-Digest Algorithm. The information digest signature may be used to determine whether the text to be processed and the stored text are exactly the same text.

[0148] Among them, the MD5 Message-Digest Algorithm is a widely used hash algorithm and can be applied to text representation. The Md5 Message-Digest Algorithm can generate a fixed-length hash value, usually 128 bits, through an irreversible string transformation algorithm. The MD5 of the same text is consistent. If any changes are made to the text, its md5 value will change greatly. Based on the above principle, the embodiments of the present disclosure may adopt the MD5 Message-Digest Algorithm for the information digest signature of the text to be processed to identify the text that is exactly the same as the text to be processed based on the information digest signature of the text to be processed.

[0149] To further improve the search efficiency and accuracy of duplicate information, before executing S420, the method includes: cleaning the text to be processed.

[0150] Correspondingly, S420 may include: determining the information digest signature of the cleaned text to be processed.

[0151] Specifically, after the server obtains the text to be processed, it may use regular expressions to convert the traditional Chinese characters in the text to be processed into simplified Chinese characters, unify the punctuation marks in the text to be processed into half-width form, convert the English letters in the text to be processed into lowercase, and remove the stop words in the text to be processed. Among them, stop words may be function words that frequently appear in various documents and carry very little semantic information. Stop words are less helpful for distinguishing text content, so they are removed when representing text. Common types of stop words include conjunctions, adverbs, prepositions, auxiliary words, and modal particles, such as "de", "le", "a", etc.

[0152] S430. When the information digest signature of the text to be processed is different from the information digest signature of the stored text, determine the text similarity between the text to be processed and the stored text.

[0153] The information digest signature of the stored text may be a signature calculated in advance based on the MD5 message digest algorithm.

[0154] In the disclosed embodiment, the stored text is the central text of the target text category, that is, the stored text may be the class center of the target text category, and the target text category may be the text category corresponding to the stored text.

[0155] In an embodiment of the present disclosure, if the server determines that the information digest signature of the text to be processed is different from the information digest signature of the stored text, it determines that the text to be processed and the stored text are not the same text, and continues to calculate the text similarity between the text to be processed and the stored text.

[0156] S440: Based on the publishing time of the text to be processed and the publishing time of the stored text, the text similarity is corrected to obtain a corrected text similarity.

[0157] S450: When the corrected text similarity is greater than or equal to a preset similarity threshold, the stored text is used as a similar text to the text to be processed.

[0158] Among them, S440-S450 are similar to S230-S240 and will not be described in detail here.

[0159] S460: When the information digest signature of the text to be processed is the same as the information digest signature of the stored text, the text to be processed is added to the target text category.

[0160] In an embodiment of the present disclosure, if the server determines that the information digest signature of the text to be processed is the same as the information digest signature of the stored text, then the text to be processed and the stored text are determined to be the same text, and the text to be processed is added to the target text category, that is, the text to be processed is clustered according to similar texts to achieve duplicate elimination of texts with repeated content, or, after adding the text to be processed to the target text category to which the similar text belongs, the central text of the target text category is also updated according to the file to be processed to achieve duplicate elimination of texts with repeated content.

[0161] In the disclosed embodiment, Figure 5 A schematic flow chart of a method for searching for identical texts provided in an embodiment of the present disclosure is shown.

[0162] Next, combine Figure 5 Explain the process of the server looking for identical text. Figure 5 As shown, the process of the same text search method may specifically include the following steps.

[0163] S501, obtaining text to be processed.

[0164] S502: Clean the text to be processed.

[0165] S503: Calculate the information digest signature of the text to be processed.

[0166] In the disclosed embodiment, the information digest signature of the text to be processed may be calculated using the MD5 algorithm.

[0167] S504: Determine whether there is an information digest signature of a stored text that is completely identical to the information digest signature of the text to be processed.

[0168] In the disclosed embodiment, the server may obtain the information digest signature of the stored text and determine whether there is an information digest signature of the stored text that is exactly the same as the information digest signature of the text to be processed. If so, execute S505; if not, execute S506.

[0169] S505: Add the text to be processed to the text category to which the stored text belongs.

[0170] In an embodiment of the present disclosure, if the server determines that the information digest signature of the text to be processed is the same as the information digest signature of the stored text, it is determined that there is a stored text that is exactly the same as the text to be processed, and the text to be processed is added to the text category to which the stored text belongs to cluster the text to be processed.

[0171] S506, end.

[0172] In the disclosed embodiment, if the server determines that the information digest signature of the text to be processed is not the same as the information digest signature of the stored text, it is determined that there is no stored text that is exactly the same as the text to be processed, and the search for the same text ends.

[0173] Therefore, in the embodiment of the present disclosure, before calculating the text similarity between the text to be processed and the stored text, the text to be processed can also be cleaned, and based on the information summary signatures corresponding to the text to be processed and the stored text respectively, the stored text that is exactly the same as the text to be processed can be screened out to improve the efficiency and accuracy of searching for duplicate information.

[0174] In some embodiments of the present disclosure, after the server uses the stored text as similar text of the text to be processed, it can also add the text to be processed to the target text category to which the similar text belongs, or add it to the target text category to which the target similar text belongs, so as to cluster the text to be processed and eliminate duplicate texts with repeated content. Alternatively, after adding the text to be processed to the target text category to which the similar text belongs, the central text of the target text category is updated according to the file to be processed to eliminate duplicate texts with repeated content.

[0175] In some other embodiments of the present disclosure, when the server determines that the corrected text similarity is less than a preset similarity threshold, the server uses the text to be stored as the central text of the new text category.

[0176] In the disclosed embodiment, Figure 6 A schematic flow chart of an information clustering method provided by an embodiment of the present disclosure is shown.

[0177] Next, combine Figure 6 Explain the process of clustering the text to be processed by the server. Figure 6 As shown, the process of the information clustering method may specifically include the following steps.

[0178] S601, obtaining text to be processed.

[0179] S602: Clean the text to be processed.

[0180] S603: Determine based on the MD5 module whether there is a stored text that is the same as the text to be processed.

[0181] In the embodiment of the present disclosure, the server can calculate the information digest signature of the text to be processed based on the MD5 module, obtain the information digest signature of the stored text, and determine whether the information digest signature of the text to be processed is the same as the information digest signature of the stored text. If there is no identical information digest signature, execute S604, otherwise, execute S606.

[0182] S604: Use a minimum hash clustering module combined with a time decay coefficient to search for similar texts to the text to be processed.

[0183] In an embodiment of the present disclosure, if the server determines that there is no stored text that is exactly the same as the text to be processed, a minimum hash clustering module combined with a time decay coefficient is used to calculate the text similarity and the time decay coefficient, and the text similarity is corrected according to the time decay coefficient to obtain the corrected text similarity, and similar text to the text to be processed is searched based on the corrected text similarity.

[0184] S605: Determine whether similar text to the text to be processed is found.

[0185] In some embodiments of the present disclosure, when the stored text is one text, if the server determines that the similarity of the revised text is greater than or equal to a preset similarity threshold, it is determined that similar text to the text to be processed is found, and S606 is executed; if the similarity of the revised text is less than the preset similarity threshold, it is determined that similar text to the text to be processed is not found, and S607 is executed.

[0186] In other embodiments of the present disclosure, when the stored text is a massive amount of text, the server can traverse each stored text, determine the target text category corresponding to each stored text, calculate the corrected text similarity obtained from each traversal, and record the stored text whose corrected text similarity is greater than or equal to a preset similarity threshold as similar text. After traversing all the stored texts, if there are multiple similar texts to the text to be processed, the similar text corresponding to the largest corrected text similarity is used as the target similar text of the text to be processed, then it is determined that similar text to the text to be processed is found, and S606 is executed. If the corrected text similarities between all stored texts and the text to be processed are less than the preset similarity threshold, then it is determined that similar text to the text to be processed is not found, and S607 is executed.

[0187] S606: Add the text to be processed to the text category to which the stored text belongs.

[0188] In an embodiment of the present disclosure, if the server determines that the information digest signature of the text to be processed is the same as the information digest signature of the stored text, the text to be processed is added to the text category to which the stored text belongs; or, if the server determines that similar text to the text to be processed is found, the text to be processed is added to the text category to which the stored text belongs, so as to cluster the text to be processed.

[0189] S607: Use the text to be stored as the central text of the new text category.

[0190] Therefore, in the embodiment of the present disclosure, before calculating the text similarity between the text to be processed and the stored text, the text to be processed is cleaned, and based on the information summary signatures corresponding to the text to be processed and the stored text respectively, the stored text that is the same as the text to be processed is screened out, and similar texts are searched in combination with the text similarity and the time decay coefficient, so that the efficiency and accuracy of searching for duplicate information are greatly improved, so as to further improve the clustering effect and efficiency of the text to be processed.

[0191] In another embodiment of the present disclosure, in order to improve the calculation accuracy and calculation efficiency of the corrected text similarity, the text length of the text to be processed can be determined, and the text similarity can be calculated when the length difference between the text length and the text length of the stored text is less than a preset length threshold, so as to further reduce the calculation amount of determining the corrected text similarity and improve the calculation accuracy of the corrected text similarity, thereby improving the efficiency and accuracy of searching for duplicate information.

[0192] Figure 7 A flowchart of another information processing method provided by an embodiment of the present disclosure is shown.

[0193] like Figure 7As shown, the information processing method may include the following steps.

[0194] S710: Obtain the text to be processed.

[0195] S720: Determine the text length of the text to be processed.

[0196] The text length of the text to be processed may be a character string length of the text to be processed.

[0197] S730: When it is determined that the length difference between the text length of the to-be-processed text and the text length of the stored text is less than a preset difference threshold, determine the text similarity between the to-be-processed text and the stored text.

[0198] The text length of the stored text may be the character string length of the stored text.

[0199] The preset length difference threshold may be a preset maximum difference data used to determine the similarity between the text to be processed and the stored text.

[0200] Specifically, after determining the text length of the text to be processed, the server obtains the text length of the stored text, and determines whether the length difference between the text length of the text to be processed and the text length of the stored text is less than a preset difference threshold. If so, it is determined that the text to be processed is similar to the stored text, and the text similarity between the text to be processed and the stored text is determined.

[0201] In some embodiments of the present disclosure, determining that the length difference between the text length of the to-be-processed text and the text length of the stored text is less than a preset difference threshold includes:

[0202] Taking the text length of the text to be processed as the quotient of the text length of the stored text;

[0203] When the quotient is less than the preset quotient value threshold, it is determined that the length difference value is less than the preset difference value threshold.

[0204] Specifically, the server can use the Jaccard length filtering algorithm to calculate the quotient of the text length of the text to be processed and the text length of the stored text, or calculate the quotient of the text length of the stored text and the text length of the text to be processed, and compare the quotient with a preset quotient value threshold. If the quotient is greater than the preset quotient value threshold, it is determined that the length difference value is less than the preset difference threshold, that is, it is determined that the text length difference between the text to be processed and the stored text is small, and the probability that the text to be processed is similar to the stored text is large, and the text similarity between the text to be processed and the stored text can be calculated.

[0205] In some embodiments, the length difference value may be calculated by the following formula. The length difference value calculation formula may be:

[0206] t len <ε*w len

[0207] Among them, t len is the length of the first text to be processed, w len is the second text length of the stored text, ε is a preset quotient threshold, and ε can be a value less than 1.

[0208] In some other embodiments, the length difference value may also be calculated by the following formula. The length difference value calculation formula may be:

[0209] w len <ε*t len

[0210] Here, ε can be a value less than 1.

[0211] S740: Based on the publishing time of the text to be processed and the publishing time of the stored text, the text similarity is corrected to obtain a corrected text similarity.

[0212] S750: When the corrected text similarity is greater than or equal to a preset similarity threshold, the stored text is used as a similar text to the text to be processed.

[0213] Among them, S740-S750 are similar to S230-S240 and will not be described in detail here.

[0214] S760: When the length difference value is greater than or equal to a preset length threshold, the stored text is used as the central text of the new category text.

[0215] Specifically, if the server determines that the quotient is less than a preset length difference threshold, it determines that the length difference value is greater than or equal to the preset difference threshold, that is, it is determined that the text length difference between the to-be-processed text and the stored text is large, and the probability that the to-be-processed text is similar to the stored text is small. The process of calculating the text similarity between the to-be-processed text and the stored text is not performed, and the stored text is used as the central text of the new category text.

[0216] Therefore, in the embodiment of the present disclosure, after the server obtains the text to be processed, it can determine the text length of the text to be processed, and based on the length difference between the text length of the text to be processed and the text length of the stored text, screen out the stored text with a higher probability of similarity to the text to be processed, so as to further calculate the text similarity, which can further reduce the calculation complexity of the text similarity, improve the calculation accuracy of the corrected text similarity, and improve the efficiency and accuracy of searching for duplicate information.

[0217] Furthermore, after the stored text is used as similar text to the text to be processed, the text to be processed can also be added to the target text category to which the stored text belongs, that is, the text to be processed is clustered according to the similar text to achieve duplicate elimination of text with repeated content, or, after the text to be processed is added to the target text category to which the similar text belongs, the central text of the target text category is updated according to the file to be processed to achieve duplicate elimination of text with repeated content.

[0218] In the disclosed embodiment, Figure 8 A flow chart of an information clustering method of a minimum hash clustering module combined with a time decay coefficient provided in an embodiment of the present disclosure is shown.

[0219] Next, combine Figure 8 Explain the process of clustering the text to be processed by the server. Figure 8 As shown, the process of the information clustering method may specifically include the following steps.

[0220] S801, determining stored text that is different from the text to be processed.

[0221] In the embodiment of the present disclosure, the server may pre-store text that is different from the text to be processed based on the MD5 module.

[0222] S802: Calculate the text length of the text to be processed.

[0223] In the embodiment of the present disclosure, the server may calculate the text length of the text to be processed.

[0224] S803, using the Jaccard length filtering algorithm to search for stored texts whose probability of similarity to the text to be processed is higher than a preset value.

[0225] In an embodiment of the present disclosure, the server may use the Jaccard length filtering algorithm to calculate the length difference between the text length of the stored text and the text length of the text to be stored, and compare the length difference with a preset length difference threshold to find stored text whose probability of similarity with the text to be processed is higher than a preset value.

[0226] S804: Calculate the time attenuation coefficient.

[0227] In an embodiment of the present disclosure, the server may calculate the publishing time difference between the publishing time of the text to be processed and the publishing time of the stored text, and determine a time decay coefficient corresponding to the publishing time difference.

[0228] S805. Calculate the minimum hash signature.

[0229] In an embodiment of the present disclosure, the server may calculate the minimum hash signatures of the text to be stored and the stored text respectively based on the minimum hash algorithm, and search for the stored text that is different from the text to be stored.

[0230] S806: Calculate the corrected text similarity.

[0231] In the disclosed embodiment, the server calculates the text similarity between the text to be stored and the stored text, and multiplies the time decay coefficient by the text similarity to obtain the corrected text similarity.

[0232] S807: Determine whether the corrected text similarity is greater than or equal to a preset similarity threshold.

[0233] In some embodiments of the present disclosure, when the stored text is one text, if the server can compare the revised text similarity with a preset similarity threshold to determine whether the revised text similarity is greater than or equal to the preset similarity threshold, if yes, execute S808, if not, execute S809.

[0234] In other embodiments of the present disclosure, when the stored text is a massive amount of text, the server can traverse each stored text, determine the target text category corresponding to each stored text, calculate the corrected text similarity obtained in each traversal, and record the stored text whose corrected text similarity is greater than or equal to a preset similarity threshold as similar text. After traversing all the stored texts, if there are multiple similar texts to the text to be processed, the similar text corresponding to the largest corrected text similarity is used as the target similar text of the text to be processed, then it is determined that similar text of the text to be processed is found, and S808 is executed. If the corrected text similarities between all stored texts and the text to be processed are less than the preset similarity threshold, then it is determined that similar text of the text to be processed is not found, and S809 is executed.

[0235] S808: Add the text to be processed to the target text category.

[0236] Among them, the stored text is the central text of the target text category.

[0237] S809: Use the text to be stored as the central text of the new text category.

[0238] Therefore, in the embodiment of the present disclosure, after the server obtains the text to be processed, it can screen out the stored text with a higher probability of similarity with the text to be processed according to the length difference between the text length of the text to be processed and the text length of the stored text, so as to improve the calculation efficiency and accuracy of the text similarity, and combine the text similarity and the time decay coefficient to find similar texts, so that the efficiency and accuracy of finding duplicate information are greatly improved, so as to further improve the clustering effect and efficiency of the text to be processed.

[0239] The present disclosure also provides an information processing device for implementing the above-mentioned information processing method. Fig. 9 In the embodiment of the present disclosure, the information processing device may be a server. The server may include a cloud server or a server cluster or other device with storage and computing functions.

[0240] Fig. 9 A schematic structural diagram of an information processing device provided by an embodiment of the present disclosure is shown.

[0241] like Fig. 9 As shown, the information processing device 900 may include: a to-be-processed text acquisition module 901 , a text similarity calculation module 902 , a text similarity correction module 903 , and a similar text determination module 904 .

[0242] Wherein, the to-be-processed text acquisition module 901 is used to acquire the to-be-processed text;

[0243] A text similarity determination module 902 is used to determine the text similarity between the text to be processed and the stored text;

[0244] A text similarity correction module 903 is used to correct the text similarity based on the publishing time of the text to be processed and the publishing time of the stored text to obtain a corrected text similarity;

[0245] The similar text determination module 904 is configured to use the stored text as a similar text to the text to be processed when the corrected text similarity is greater than or equal to a preset similarity threshold.

[0246] In the embodiment of the present disclosure, after obtaining the text to be processed, the text similarity between the text to be stored and the stored text can be calculated, and the text similarity can be corrected according to the release time of the text to be processed and the release time of the stored text to obtain the corrected text similarity. When the corrected text similarity is greater than or equal to the preset similarity threshold, the stored text is used as the similar text of the text to be processed. First, since the process of determining the text similarity and the process of correcting the text similarity based on the release time have a small amount of calculation and a small memory occupation, the process of determining the similar text of the text to be processed has a small amount of calculation and saves equipment resources, so the screening efficiency of similar texts can be improved. Secondly, the corrected text similarity can not only reflect the similarity of the content of the two texts, but also reflect the time relevance of the two texts. Therefore, the screening accuracy of similar texts can be improved. In summary, the embodiment of the present disclosure can quickly and accurately find similar texts, that is, repeated information, and then when displaying content to users, the repetitiveness of the displayed content can be reduced, so that users can quickly and accurately find information of interest from the network platform, improve the user's information viewing experience and save equipment resources.

[0247] In some embodiments of the present disclosure, the text similarity determination module 902 is specifically used to generate a minimum hash signature of the text to be processed;

[0248] Calculate the Hamming distance between the minimum hash signature of the text to be processed and the minimum hash signature of the stored text;

[0249] Determine text similarity based on Hamming distance.

[0250] In some embodiments of the present disclosure, the text similarity correction module 903 is specifically used to calculate the publishing time difference between the publishing time of the text to be processed and the publishing time of the stored text;

[0251] Determine the time decay coefficient corresponding to the release time difference;

[0252] The text similarity is multiplied by the time decay coefficient to obtain the modified text similarity.

[0253] In some embodiments of the present disclosure, the text similarity correction module 903 is specifically used to, when the publishing time difference is less than a preset time threshold, use a preset decay value as a time decay coefficient;

[0254] When the release time difference is greater than or equal to a preset time threshold, a time decay coefficient is calculated according to the release time difference and a preset decay speed parameter.

[0255] In some embodiments of the present disclosure, the stored text is the central text of the target text category;

[0256] Accordingly, the device further comprises: a clustering module;

[0257] Among them, the clustering module is used to add the text to be processed into the target text category.

[0258] In some embodiments of the present disclosure, the device further includes: a text category updating module, wherein the text category updating module is used to use the to-be-stored text as the central text of a new text category when the corrected text similarity is less than a preset similarity threshold.

[0259] In some embodiments of the present disclosure, the device also includes: a target similar text search module; wherein, when there are multiple similar texts to the text to be processed, the target similar text search module uses the similar text corresponding to the largest corrected text similarity as the target similar text of the text to be processed.

[0260] In some embodiments of the present disclosure, the device further includes: an information digest signature determination module; wherein the information digest signature determination module is used to determine the information digest signature of the text to be processed.

[0261] Accordingly, the text similarity determination module 902 is specifically used to determine the text similarity between the text to be processed and the stored text when the information digest signature of the text to be processed is different from the information digest signature of the stored text.

[0262] In some embodiments of the present disclosure, the stored text is the central text of the target text category;

[0263] Correspondingly, the clustering module is also used to: when the information summary signature of the text to be processed is the same as the information summary signature of the stored text, add the text to be processed to the target text category.

[0264] In some embodiments of the present disclosure, the device further comprises: a text length determination module; wherein the text length determination module determines the text length of the text to be processed;

[0265] Accordingly, the text similarity determination module 902 is specifically used to determine the text similarity between the to-be-processed text and the stored text when it is determined that the length difference between the text length of the to-be-processed text and the text length of the stored text is less than a preset difference threshold.

[0266] In some embodiments of the present disclosure, the text similarity determination module 902 is specifically used to calculate the quotient of the text length of the to-be-processed text and the text length of the stored text;

[0267] In a case where the quotient is smaller than the preset quotient value threshold, it is determined that the length difference value is smaller than the preset difference value threshold.

[0268] In some embodiments of the present disclosure, the text category updating module is specifically configured to use the text to be stored as the central text of a new category of text when the length difference value is greater than or equal to the preset length threshold.

[0269] It should be noted that Fig. 9 The information processing device 900 shown can execute Figures 2 to 8 The various steps in the method embodiment shown in the figure are implemented Figures 2 to 8 The various processes and effects in the method embodiment shown are not described in detail here.

[0270] The present disclosure also provides an information processing device, which may include a processor and a memory, wherein the memory may be used to store executable instructions, wherein the processor may be used to read the executable instructions from the memory and execute the executable instructions to implement the information processing method in the above embodiment.

[0271] Fig.10 A schematic diagram of the structure of an information processing device provided by an embodiment of the present disclosure is shown. Fig.10 , which shows a structural schematic diagram of an information processing device 1000 suitable for implementing the embodiments of the present disclosure.

[0272] The information processing device 1000 in the embodiment of the present disclosure may be Figure 1 The server in the embodiment of the present invention can be a cloud server or a server cluster or other devices with storage and computing functions.

[0273] It should be noted that Fig.10 The information processing device 1000 shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0274] like Fig.10 As shown, the information processing device 1000 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 1001, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1002 or a program loaded from a storage device 1008 to a random access memory (RAM) 1003. In the RAM 1003, various programs and data required for the operation of the information processing device 1000 are also stored. The processing device 1001, the ROM 1002, and the RAM 1003 are connected to each other via a bus 1004. An input / output (I / O) interface 1005 is also connected to the bus 1004.

[0275] Typically, the following devices may be connected to the I / O interface 1005: an input device 1006 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 1007 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 1008 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 1009. The communication device 1009 may allow the information processing device 1000 to communicate with other devices wirelessly or by wire to exchange data. Although Fig.10 The information processing device 1000 is shown to have various devices, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead.

[0276] The embodiments of the present disclosure further provide a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the processor implements the information processing method in the above embodiments.

[0277] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a non-transitory computer-readable medium, and the computer program contains program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device 1009, or installed from a storage device 1008, or installed from a ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the information processing method of the embodiment of the present disclosure are executed.

[0278] It should be noted that the computer-readable medium disclosed above may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction execution system, device or device. In the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0279] In some embodiments, the client and server may communicate using any currently known or future developed network protocol such as HTTP, and may be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), an internet (e.g., the Internet), and a peer-to-peer network (e.g., an ad hoc peer-to-peer network), as well as any currently known or future developed network.

[0280] The computer-readable medium may be included in the information processing device, or may exist independently without being incorporated into the information processing device.

[0281] The computer-readable medium carries one or more programs. When the one or more programs are executed by the information processing device, the information processing device executes:

[0282] Get the text to be processed;

[0283] Determining text similarity between the text to be processed and the stored text;

[0284] Based on the publishing time of the text to be processed and the publishing time of the stored text, the text similarity is corrected to obtain a corrected text similarity;

[0285] When the corrected text similarity is greater than or equal to a preset similarity threshold, the stored text is used as a similar text to the text to be processed.

[0286] In embodiments of the present disclosure, computer program code for performing the operations of the present disclosure may be written in one or more programming languages ​​or a combination thereof, including but not limited to object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on a user's computer, partially on a user's computer, as a separate software package, partially on a user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0287] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0288] The units involved in the embodiments described in the present disclosure may be implemented by software or hardware, wherein the name of a unit does not, in some cases, limit the unit itself.

[0289] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0290] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0291] The above description is only a preferred embodiment of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of disclosure involved in the present disclosure is not limited to the technical solutions formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above disclosed concept. For example, the above features are replaced with the technical features with similar functions disclosed in the present disclosure (but not limited to) by each other to form a technical solution.

[0292] In addition, although each operation is described in a specific order, this should not be understood as requiring these operations to be performed in the specific order shown or in a sequential order. Under certain circumstances, multitasking and parallel processing may be advantageous. Similarly, although some specific implementation details are included in the above discussion, these should not be interpreted as limiting the scope of the present disclosure. Some features described in the context of a separate embodiment can also be implemented in a single embodiment in combination. On the contrary, the various features described in the context of a single embodiment can also be implemented in multiple embodiments individually or in any suitable sub-combination mode.

[0293] Although the subject matter has been described in language specific to structural features and / or methodological logical actions, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. On the contrary, the specific features and actions described above are merely example forms of implementing the claims.

Claims

1. An information processing method, characterized in that: include: Get the text to be processed; Determining the text similarity between the to-be-processed text and the stored text; Based on the publishing time of the to-be-processed text and the publishing time of the stored text, the text similarity is corrected to obtain a corrected text similarity; In the case where the corrected text similarity is greater than or equal to a preset similarity threshold, the stored text is used as a similar text of the text to be processed, wherein both the text to be processed and the stored text are texts posted by users on a network platform, and the similar text is repeated information of the text to be processed, and the repeated information is used to assist in duplicate elimination processing; The step of correcting the text similarity based on the publishing time of the text to be processed and the publishing time of the stored text to obtain the corrected text similarity includes: Calculating the publishing time difference between the publishing time of the to-be-processed text and the publishing time of the stored text; Determine a time decay coefficient corresponding to the release time difference; Multiplying the text similarity by the time decay coefficient to obtain the corrected text similarity; The determining of the time attenuation coefficient corresponding to the publishing time difference includes: When the release time difference is less than a preset time threshold, a preset attenuation coefficient is used as the time attenuation coefficient; When the release time difference is greater than or equal to the preset time threshold, the time decay coefficient is calculated based on the release time difference and the preset decay speed parameter, wherein the time decay coefficient is the quotient of a first preset decay speed parameter and a preset exponential function, and the preset exponential function is determined based on the release time difference and the second preset decay speed parameter.

2. The method according to claim 1, characterized in that: The determining of the text similarity between the to-be-processed text and the stored text comprises: Generate a minimum hash signature of the text to be processed; Calculating the Hamming distance between the minimum hash signature of the text to be processed and the minimum hash signature of the stored text; The text similarity is determined according to the Hamming distance.

3. The method according to claim 1, characterized in that The stored text is the central text of the target text category; Wherein, after taking the stored text as a similar text to the text to be processed, the method further comprises: The to-be-processed text is added to the target text category.

4. The method according to claim 1, characterized in that: After the text similarity is corrected based on the publishing time of the text to be processed and the publishing time of the stored text to obtain the corrected text similarity, the method further includes: When the corrected text similarity is less than the preset similarity threshold, the text to be stored is used as the central text of the new text category.

5. The method according to claim 1, characterized in that After taking the stored text as a similar text to the text to be processed, the method further includes: In the case that there are multiple similar texts in the text to be processed, the similar text corresponding to the largest corrected text similarity is used as the target similar text of the text to be processed.

6. The method according to claim 1, characterized in that Before determining the text similarity between the to-be-processed text and the stored text, the method further includes: Determine the message digest signature of the text to be processed; Wherein, determining the text similarity between the to-be-processed text and the stored text includes: In a case where the information digest signature of the text to be processed is different from the information digest signature of the stored text, the text similarity between the text to be processed and the stored text is determined.

7. The method according to claim 6, characterized in that The stored text is the central text of the target text category; Wherein, after determining the information summary signature of the text to be processed, the method further includes: When the information digest signature of the text to be processed is the same as the information digest signature of the stored text, the text to be processed is added to the target text category.

8. The method according to claim 1, characterized in that Before determining the text similarity between the to-be-processed text and the stored text, the method further includes: Determine the text length of the text to be processed; Wherein, determining the text similarity between the to-be-processed text and the stored text includes: When it is determined that the length difference between the text length of the to-be-processed text and the text length of the stored text is less than a preset difference threshold, the text similarity between the to-be-processed text and the stored text is determined.

9. The method according to claim 8, characterized in that The determining that the length difference between the text length of the to-be-processed text and the text length of the stored text is less than a preset difference threshold comprises: Taking the text length of the to-be-processed text as a quotient of the text length of the stored text; In a case where the quotient is greater than a preset quotient value threshold, it is determined that the length difference value is less than the preset difference value threshold.

10. The method according to claim 8, characterized in that After determining the text length of the text to be processed, the method further includes: When the length difference value is greater than or equal to a preset length threshold, the text to be stored is used as the central text of the new category text.

11. An information processing device, characterized in that: include: A module for obtaining text to be processed, used for obtaining text to be processed; A text similarity determination module, used to determine the text similarity between the to-be-processed text and the stored text; A text similarity correction module, used to correct the text similarity based on the publishing time of the text to be processed and the publishing time of the stored text, to obtain a corrected text similarity; A similar text determination module, used for, when the similarity of the modified text is greater than or equal to a preset similarity threshold, using the stored text as a similar text of the text to be processed, wherein both the text to be processed and the stored text are texts posted by users on a network platform, and the similar text is repeated information of the text to be processed, and the repeated information is used to assist in duplicate elimination processing; The text similarity correction module is specifically used to calculate the publishing time difference between the publishing time of the text to be processed and the publishing time of the stored text; Determine a time decay coefficient corresponding to the release time difference; Multiplying the text similarity by the time decay coefficient to obtain the corrected text similarity; The text similarity correction module is specifically configured to use a preset attenuation coefficient as the time attenuation coefficient when the publishing time difference is less than a preset time threshold; When the release time difference is greater than or equal to the preset time threshold, the time decay coefficient is calculated based on the release time difference and the preset decay speed parameter, wherein the time decay coefficient is the quotient of a first preset decay speed parameter and a preset exponential function, and the preset exponential function is determined based on the release time difference and the second preset decay speed parameter.

12. An information processing device, characterized in that: include: processor; a memory for storing instructions executable by the processor; The processor is configured to execute the instructions to implement the information processing method according to any one of claims 1 to 10.

13. A storage medium, characterized in that: When the instructions in the storage medium are executed by a processor of a computing device, the computing device is enabled to execute the information processing method according to any one of claims 1 to 10.

Citation Information

Patent Citations

  • Method and device for representing text

    CN104778158A

  • Method and equipment for describing author of short texts

    CN106156091A

  • Data processing method and device

    CN106202530A

  • Multimedia information recommendation method and device and multimedia system

    CN106294630A

  • File storage method, apparatus, computer device, and storage medium

    CN109063105A