Text data processing method and device, electronic device and readable storage medium

Abstract

The present disclosure provides a text data processing method and device, an electronic device and a computer-readable storage medium, and the present disclosure relates to the field of Internet technology. The text data processing method includes: obtaining text data to be processed; determining the feature value of the text data to be processed based on a similarity detection algorithm; using the feature values ​​of multiple first text data stored in a first cache database and the feature value of the text data to be processed to perform similarity calculation in sequence to obtain a first similarity value; and in response to the first similarity value currently calculated being less than or equal to a first preset value, determining that the text data to be processed is the target text data.

Description

Technical Field

[0001] The present disclosure relates to the field of Internet technology, and more specifically, to a text data processing method and device, an electronic device, a readable storage medium, and a computer program product. Background Art

[0002] With the development of Internet technology, more and more articles are recommended to users based on their preferences. However, there are too many similar redundant comments under the articles and repeated comments generated by machines, which makes the user experience poor.

[0003] In the related art, for text data such as comments, an artificial neural network-based method is usually used to identify the text, for example, through supervised / unsupervised text similarity calculation, DSSM-LSTM calculation of short text similarity, etc., to determine whether the text is redundant information.

[0004] In the process of implementing the concepts disclosed herein, the inventors discovered that there are at least the following problems in the related art: the implementation of the related methods is heavy, relies on machine learning, and requires a large amount of computing resources. Summary of the invention

[0005] In view of this, the present disclosure provides a text data processing method and device, an electronic device, a readable storage medium and a computer program product.

[0006] One aspect of the present disclosure provides a text data processing method, comprising:

[0007] Get the text data to be processed;

[0008] Based on the similarity detection algorithm, determine the feature value of the above text data to be processed;

[0009] sequentially using the feature values ​​of the plurality of first text data stored in the first cache database and the feature values ​​of the text data to be processed to perform similarity calculations to obtain a first similarity value; and

[0010] In response to the first similarity value currently calculated being less than or equal to the first preset value, the text data to be processed is determined to be the target text data.

[0011] According to an embodiment of the present disclosure, the above method further includes:

[0012] In the case that all the calculated first similarity values ​​are greater than the first preset value, randomly selecting a preset number of text data from the second cache database to obtain a plurality of second text data;

[0013] Using the characteristic values ​​of the plurality of second text data to respectively calculate similarity with the characteristic values ​​of the text data to be processed, to obtain a plurality of second similarity values;

[0014] Based on the plurality of the second similarity values, determining the amount of third text data, wherein the third text data includes the second text data whose second similarity value is less than or equal to the first preset value;

[0015] When the amount of the third text data is less than the second preset value, storing the to-be-processed text data in the second cache database; and

[0016] When the amount of the third text data is greater than or equal to the second preset value, the third text data in the second cache database is deleted, and the to-be-processed text data and the third text data are determined as target text data.

[0017] According to an embodiment of the present disclosure, the above method further includes:

[0018] When storing the text data to be processed into the second cache database, a time mark is set for the text data to be processed based on the system time and the preset time length.

[0019] According to an embodiment of the present disclosure, the above method further includes:

[0020] Polling the time stamp of the text data in the second cache database;

[0021] When the time indicated by the time mark of the current text data is later than the current system time, extracting the current text data from the second cache database; and

[0022] The above current text data is stored in the database.

[0023] According to an embodiment of the present disclosure, the above method further includes:

[0024] The target text data is stored in the first cache database.

[0025] According to an embodiment of the present disclosure, the above method further includes:

[0026] Set up scheduled tasks;

[0027] In response to triggering the above-mentioned scheduled task, the text data stored in the above-mentioned first cache database and the above-mentioned second cache database are deleted.

[0028] According to an embodiment of the present disclosure, the determining of the feature value of the text data to be processed based on the similarity detection algorithm includes:

[0029] Using a word segmentation tool, decompose the above text data to be processed into word sets;

[0030] Calculate the hash value of each word in the above word set;

[0031] Based on the preset weights, performing weighted summation on the hash values ​​of the respective words in the above word set to obtain the signature value of the above text data to be processed; and

[0032] The above-mentioned signature value is subjected to dimensionality reduction processing to obtain the feature value of the above-mentioned text data to be processed.

[0033] According to an embodiment of the present disclosure, the above-mentioned sequentially using the feature values ​​of the plurality of first text data stored in the first cache database and the feature values ​​of the above-mentioned text data to be processed to perform similarity calculation to obtain a first similarity value includes:

[0034] For each of the first text data, the Hamming distance between the feature value of the first text data and the feature value of the text data to be processed is calculated in turn to obtain the first similarity value.

[0035] Another aspect of the present disclosure provides a text data processing device, including an acquisition module, a first determination module, a first calculation module, and a second determination module.

[0036] Wherein, the acquisition module is used to obtain the text data to be processed;

[0037] A first determination module is used to determine the feature value of the text data to be processed based on a similarity detection algorithm;

[0038] A first calculation module, configured to sequentially use the feature values ​​of the plurality of first text data stored in the first cache database and the feature values ​​of the text data to be processed to perform similarity calculation to obtain a first similarity value; and

[0039] The second determination module is configured to determine that the text data to be processed is target text data in response to the first similarity value currently calculated being less than or equal to a first preset value.

[0040] According to an embodiment of the present disclosure, the above-mentioned device also includes a selection module, a second calculation module, a third determination module, a first storage module and a fourth determination module.

[0041] The selection module is used to randomly select a preset number of text data from the second cache database to obtain a plurality of second text data when all the calculated first similarity values ​​are greater than the first preset value.

[0042] A second calculation module, used to use the characteristic values ​​of the plurality of second text data to perform similarity calculations with the characteristic values ​​of the text data to be processed, respectively, to obtain a plurality of second similarity values;

[0043] A third determining module, configured to determine the amount of third text data based on the plurality of second similarity values, wherein the third text data includes the second text data whose second similarity value is less than or equal to the first preset value;

[0044] A first storage module is used for storing the to-be-processed text data in the second cache database when the amount of the third text data is less than a second preset value; and

[0045] The fourth determining module is used to delete the third text data in the second cache database and determine the to-be-processed text data and the third text data as target text data when the amount of the third text data is greater than or equal to the second preset value.

[0046] According to an embodiment of the present disclosure, the above-mentioned device also includes a first setting module.

[0047] Wherein, the first setting module is used to set a time mark for the above-mentioned text data to be processed based on the system time and preset time length when storing the above-mentioned text data to be processed in the above-mentioned second cache database.

[0048] According to an embodiment of the present disclosure, the above-mentioned device also includes a polling module, an extraction module and a second storage module.

[0049] The polling module is used to poll the time stamp of the text data in the second cache database;

[0050] an extraction module, configured to extract the current text data from the second cache database when the time indicated by the time mark of the current text data is later than the current system time; and

[0051] The second storage module is used to store the above current text data in a database.

[0052] According to an embodiment of the present disclosure, the above-mentioned device also includes a third storage module.

[0053] Wherein, the third storage module is used to store the above-mentioned target text data into the above-mentioned first cache database.

[0054] According to an embodiment of the present disclosure, the above-mentioned device also includes a second setting module and a deletion module.

[0055] Wherein, the second setting module is used to set the scheduled task;

[0056] The deleting module is used to delete the text data stored in the first cache database and the second cache database in response to triggering the above-mentioned scheduled task.

[0057] According to an embodiment of the present disclosure, the first determination module includes a decomposition unit, a first calculation unit, a summation unit and a dimensionality reduction unit.

[0058] The decomposition unit is used to decompose the text data to be processed into word sets by using a word segmentation tool;

[0059] A first calculation unit, used to calculate the hash value of each word in the above word set;

[0060] A summing unit, configured to perform weighted summation on the hash values ​​of the respective words in the word set based on a preset weight value to obtain a signature value of the text data to be processed; and

[0061] The dimension reduction unit is used to perform dimension reduction processing on the above-mentioned signature value to obtain the characteristic value of the above-mentioned text data to be processed.

[0062] According to an embodiment of the present disclosure, the first computing module includes a second computing unit.

[0063] The second calculation unit is used to calculate the Hamming distance between the feature value of the first text data and the feature value of the text data to be processed for each of the first text data in turn, so as to obtain the first similarity value.

[0064] Another aspect of the present disclosure provides an electronic device, comprising: one or more processors; and a memory for storing one or more instructions, wherein when the one or more instructions are executed by the one or more processors, the one or more processors implement the method as described above.

[0065] Another aspect of the present disclosure provides a computer-readable storage medium storing computer-executable instructions, which are used to implement the above method when executed.

[0066] Another aspect of the present disclosure provides a computer program product, which includes computer executable instructions, and when the instructions are executed, are used to implement the method as described above.

[0067] According to the embodiments of the present disclosure, a technical means is adopted to identify the text to be processed by detecting the characteristic values ​​of the text data to be processed and the first text data, wherein the first text data is text stored in the first cache database and has been determined to be redundant information. When the similarity between the text to be processed and the first text data is less than a first preset value, the text to be processed can be considered to be redundant information. Therefore, the technical problems in the related technologies of heavy implementation, reliance on machine learning, and high demand for computing resources are at least partially overcome, thereby achieving the technical effect of improving the recognition efficiency of text and reducing the consumption of computing resources. BRIEF DESCRIPTION OF THE DRAWINGS

[0068] The above and other objects, features and advantages of the present disclosure will become more apparent through the following description of the embodiments of the present disclosure with reference to the accompanying drawings, in which:

[0069] Figure 1 An exemplary system architecture to which the text data processing method and apparatus of the present disclosure can be applied is schematically shown;

[0070] Figure 2 A flowchart of a text data processing method according to an embodiment of the present disclosure is schematically shown;

[0071] Figure 3 A flowchart of a method for determining target text data according to an embodiment of the present disclosure is schematically shown;

[0072] Figure 4 A flowchart of a method for storing current text data according to an embodiment of the present disclosure is schematically shown;

[0073] Figure 5 A flowchart of a method for obtaining feature values ​​of text data to be processed according to an embodiment of the present disclosure is schematically shown;

[0074] Figure 6 The overall process diagram of the simhash algorithm according to the embodiment of the present disclosure is schematically shown;

[0075] Figure 7 A flowchart of a method for deleting text data according to an embodiment of the present disclosure is schematically shown;

[0076] Figure 8 The overall flow chart of the text data processing method according to the embodiment of the present disclosure is schematically shown;

[0077] Fig. 9 A block diagram schematically shows a text data processing device according to an embodiment of the present disclosure; and

[0078] Fig.10The block diagram schematically shows a computer system suitable for implementing the text data processing method according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0079] Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the present disclosure. In the following detailed description, for ease of explanation, many specific details are set forth to provide a comprehensive understanding of the embodiments of the present disclosure. However, it is apparent that one or more embodiments may also be implemented without these specific details. In addition, in the following description, descriptions of known structures and technologies are omitted to avoid unnecessary confusion of the concepts of the present disclosure.

[0080] The terms used herein are only for describing specific embodiments and are not intended to limit the present disclosure. The terms "comprise", "include", etc. used herein indicate the existence of the features, steps, operations and / or components, but do not exclude the existence or addition of one or more other features, steps, operations or components.

[0081] All terms (including technical and scientific terms) used herein have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein should be interpreted as having a meaning consistent with the context of this specification, and should not be interpreted in an idealized or overly rigid manner.

[0082] In the case of using expressions such as "at least one of A, B, and C, etc.", it should generally be interpreted in accordance with the meaning of the expression generally understood by those skilled in the art (for example, "a system having at least one of A, B, and C" should include but is not limited to a system having A alone, B alone, C alone, A and B, A and C, B and C, and / or A, B, C, etc.). In the case of using expressions such as "at least one of A, B, or C, etc.", it should generally be interpreted in accordance with the meaning of the expression generally understood by those skilled in the art (for example, "a system having at least one of A, B, or C" should include but is not limited to a system having A alone, B alone, C alone, A and B, A and C, B and C, and / or A, B, C, etc.).

[0083] Currently, the Internet recommends more and more articles to users based on their preferences. However, similar comments under the articles are redundant, and there are too many repeated comments generated by machines, which makes the user experience poor.

[0084] In the related art, for text data such as comments, an artificial neural network-based method is usually used to identify the text, for example, through supervised / unsupervised text similarity calculation, DSSM-LSTM calculation of short text similarity, etc., to determine whether the text is redundant information.

[0085] However, the existing methods have the following disadvantages: the existing technology is relatively heavy to implement, often by storing comment data in the form of structured data or logs in MaxComputer, building heterogeneous data offline every day, using a distributed tensorflow model recognition system to identify redundant text based on the built graphs and labels; the identified redundant comments or text information are deleted and banned by ProcessCenter, and the reverse system can also perform reverse sample training. The entire implementation process has high requirements for the data model, relies on machine learning, and has many system services, and the overall complexity is high.

[0086] In addition, comments on online articles that flood the screen in a short period of time, or real-time chat records in IM, are all risk information and need to be identified and screened; similar comments or chats that are separated by a long period of time may not be risk information. The existing technology does not yet have a method to distinguish risk information for different time periods.

[0087] In order to at least partially solve the technical problems existing in the related art, the present disclosure provides a text data processing method and device, an electronic device and a computer-readable storage medium, and the present disclosure relates to the field of Internet technology. The text data processing method includes: obtaining text data to be processed; determining the feature value of the text data to be processed based on a similarity detection algorithm; using the feature values ​​of multiple first text data stored in a first cache database and the feature value of the text data to be processed to perform similarity calculation in sequence to obtain a first similarity value; and in response to the first similarity value currently calculated being less than or equal to a first preset value, determining that the text data to be processed is the target text data.

[0088] In the technical solution disclosed in the present invention, the acquisition, storage and application of user personal information involved are in compliance with the provisions of relevant laws and regulations, necessary confidentiality measures are taken, and do not violate public order and good morals.

[0089] In the technical solution of the present disclosure, the user's authorization or consent is obtained before obtaining or collecting the user's personal information.

[0090] Figure 1 The exemplary system architecture to which the text data processing method according to the embodiment of the present disclosure can be applied is schematically shown. It should be noted that: Figure 1What is shown is merely an example of a system architecture to which the embodiments of the present disclosure can be applied, in order to help those skilled in the art understand the technical content of the present disclosure, but it does not mean that the embodiments of the present disclosure cannot be used in other devices, systems, environments or scenarios.

[0091] like Figure 1 As shown, the system architecture 100 according to this embodiment may include terminal devices 101, 102, 103, a network 104 and a server 105. The network 104 is used to provide a medium for communication links between the terminal devices 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired and / or wireless communication links, etc.

[0092] Users can use terminal devices 101, 102, 103 to interact with server 105 through network 104 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 101, 102, 103, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients and / or social platform software, etc. (only as examples).

[0093] The terminal devices 101 , 102 , and 103 may be various electronic devices having a display screen and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, and desktop computers.

[0094] The server 105 may be a server that provides various services, such as a background management server (only as an example) that provides support for client applications used by users using the terminal devices 101, 102, and 103. The background management server may analyze and process the received user input data, and feed back the processing results (such as comment information generated according to the user input data) to the terminal device.

[0095] It should be noted that the text data processing method provided in the embodiment of the present disclosure can generally be executed by the server 105. Accordingly, the text data processing device provided in the embodiment of the present disclosure can generally be set in the server 105. The text data processing method provided in the embodiment of the present disclosure can also be executed by a server or server cluster that is different from the server 105 and can communicate with the terminal devices 101, 102, 103 and / or the server 105. Correspondingly, the text data processing device provided in the embodiment of the present disclosure can also be set in a server or server cluster that is different from the server 105 and can communicate with the terminal devices 101, 102, 103 and / or the server 105. Alternatively, the text data processing method provided in the embodiment of the present disclosure can also be executed by the terminal devices 101, 102, or 103, or can also be executed by other terminal devices different from the terminal devices 101, 102, or 103. Accordingly, the text data processing apparatus provided in the embodiments of the present disclosure may also be disposed in the terminal device 101 , 102 , or 103 , or in other terminal devices different from the terminal device 101 , 102 , or 103 .

[0096] For example, the user can input the text data to be processed on any one of the terminal devices 101, 102, or 103 (for example, the terminal device 101, but not limited thereto). Then, the terminal device 101 can locally execute the text data processing method provided by the embodiment of the present disclosure, or send the text data to be processed to other terminal devices, servers, or server clusters, and the other terminal devices, servers, or server clusters that receive the text data to be processed execute the text data processing method provided by the embodiment of the present disclosure.

[0097] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to the implementation requirements.

[0098] Figure 2 The flowchart of the text data processing method according to the embodiment of the present disclosure is schematically shown.

[0099] like Figure 2 As shown, the text data processing method includes operations S201 to S204.

[0100] In operation S201, text data to be processed is obtained.

[0101] According to an embodiment of the present disclosure, the text data to be processed may include comments or real-time chat data, etc.

[0102] In operation S202, based on a similarity detection algorithm, feature values ​​of the text data to be processed are determined.

[0103] According to an embodiment of the present disclosure, the similarity detection algorithm may include a TF-IDF algorithm, a BM25 algorithm, a simhash algorithm, an LSI (Latent Semantic Indexing) model, and a text matching model based on deep learning, etc.

[0104] According to an embodiment of the present disclosure, the feature value may include a binary value calculated by a similarity detection algorithm.

[0105] In operation S203, similarity calculation is performed in sequence using the feature values ​​of the plurality of first text data stored in the first cache database and the feature values ​​of the text data to be processed to obtain a first similarity value.

[0106] According to an embodiment of the present disclosure, the first cache database may be, for example, a redis database, which is a key-value non-relational database and stores data in memory.

[0107] According to an embodiment of the present disclosure, the value types supported by redis for storage may include strings, linked lists, sets, ordered sets, hash types, and the like.

[0108] According to an embodiment of the present disclosure, the first text data may include a comment text confirmed to be a redundant comment.

[0109] According to an embodiment of the present disclosure, similarity calculation methods may include minimum edit distance, Euclidean distance, cosine distance, Jaccard similarity, Hamming distance, and the like.

[0110] According to an embodiment of the present disclosure, the first similarity value may represent the similarity between the feature value of the first text data and the feature value of the text data to be processed.

[0111] In operation S204, in response to the first similarity value currently calculated being less than or equal to the first preset value, the text data to be processed is determined to be the target text data.

[0112] According to an embodiment of the present disclosure, after obtaining the characteristic value of the text data to be processed, the characteristic value of the first text data stored in the first cache database can be traversed to calculate the first similarity value between the characteristic value of the text data to be processed and the first similarity value thereof and the characteristic value of the text data to be processed. When the first similarity value is less than or equal to a first preset value, it can be determined that the text data to be processed is the target text data of redundant comments.

[0113] According to the embodiments of the present disclosure, a technical means is adopted to identify the text to be processed by detecting the characteristic values ​​of the text data to be processed and the first text data, wherein the first text data is text stored in the first cache database and has been determined to be redundant information. When the similarity between the text to be processed and the first text data is less than a first preset value, the text to be processed can be considered to be redundant information. Therefore, the technical problems in the related technologies of heavy implementation, reliance on machine learning, and high demand for computing resources are at least partially overcome, thereby achieving the technical effect of improving the recognition efficiency of text and reducing the consumption of computing resources.

[0114] Reference below Figure 3 to Figure 9 , combined with specific embodiments Figure 2 The method shown is further explained.

[0115] Figure 3 The flowchart of the method for determining target text data according to an embodiment of the present disclosure is schematically shown.

[0116] like Figure 3 As shown, the method for determining target text data includes operations S301 to S305.

[0117] In operation S301, when all calculated first similarity values ​​are greater than a first preset value, a preset number of text data are randomly selected from a second cache database to obtain a plurality of second text data.

[0118] According to an embodiment of the present disclosure, the second cache database may be, for example, a redis database.

[0119] According to an embodiment of the present disclosure, the second cache database may store non-redundant comments, the number of comments similar to the comment data, and the time for setting data update.

[0120] According to an embodiment of the present disclosure, part of the data may be randomly taken out from the second cache database for comparison and analysis with the text data to be processed.

[0121] According to an embodiment of the present disclosure, the second text data may include a preset number of non-redundant comment text data randomly selected from the second cache database.

[0122] In operation S302, the characteristic values ​​of the plurality of second text data are used to perform similarity calculations with the characteristic values ​​of the text data to be processed, so as to obtain a plurality of second similarity values.

[0123] According to an embodiment of the present disclosure, the second similarity value may represent the similarity between the feature value of the second text data and the feature value of the text data to be processed.

[0124] In operation S303 , the amount of third text data is determined based on the plurality of second similarity values, wherein the third text data includes second text data whose second similarity values ​​are less than or equal to a first preset value.

[0125] According to an embodiment of the present disclosure, the third text data may include second text data whose second similarity value is less than or equal to the first preset value.

[0126] In operation S304, when the amount of the third text data is less than the second preset value, the text data to be processed is stored in the second cache database.

[0127] According to an embodiment of the present disclosure, for example, the second preset value may be set to 10, and when the amount of the third text data is less than 10, the text data to be processed may be stored in the second cache database.

[0128] In operation S305, when the amount of the third text data is greater than or equal to the second preset value, the third text data in the second cache database is deleted, and the text data to be processed and the third text data are determined as target text data.

[0129] According to an embodiment of the present disclosure, the second preset value can be set to 10, for example. When the number of third text data is greater than or equal to 10, all third text data similar to the text data to be processed in the second cache database can be deleted, and the text data to be processed can be confirmed as the target text data of redundant comments.

[0130] According to an embodiment of the present disclosure, the second cache database has the function of clearing data at a fixed time, and can calculate text similarity and perform filtering and screening within a certain time interval.

[0131] According to an embodiment of the present disclosure, the text data processing method further includes:

[0132] The target text data is stored in the first cache database.

[0133] According to an embodiment of the present disclosure, a preset number of second text data are randomly selected from a second cache database, and characteristic values ​​of multiple second text data are used to perform similarity calculations with characteristic values ​​of the text data to be processed, to obtain a second similarity value, and then based on the second similarity value, it is determined whether the text data to be processed is a redundant comment according to the number of third text data obtained, and at the same time, relying on the characteristics of redis, the effect of filtering out redundant comments is achieved.

[0134] Figure 4 The flowchart of the method for storing current text data according to an embodiment of the present disclosure is schematically shown.

[0135] like Figure 4 As shown, the method for storing current text data includes operations S401 to S403.

[0136] In operation S401, the time stamp of the text data in the second cache database is polled.

[0137] According to an embodiment of the present disclosure, when storing the text data to be processed in the second cache database, a time mark is set for the text data to be processed based on the system time and a preset time length.

[0138] According to the embodiments of the present disclosure, polling may refer to the CPU periodically sending inquiries to sequentially inquire whether each peripheral device needs its service, and providing service if necessary. After the service is completed, the next peripheral device is inquired again, and the cycle repeats continuously.

[0139] According to an embodiment of the present disclosure, a time mark may be set for the text data to be processed based on the system time of the terminal device and a preset duration of the expected storage time of the text data to be processed.

[0140] In operation S402, when the time indicated by the time mark of the current text data is later than the current system time, the current text data is extracted from the second cache database.

[0141] In operation S403, the current text data is stored in a database.

[0142] According to an embodiment of the present disclosure, the data in the second cache database may be persistently stored in the database within a certain period of time.

[0143] According to an embodiment of the present disclosure, by storing the current text data in a database when the time represented by the time identifier of the current text data is later than the current system time, persistent storage of the data in the second cache database within a certain period of time can be achieved.

[0144] Figure 5 A flowchart of a method for obtaining feature values ​​of text data to be processed according to an embodiment of the present disclosure is schematically shown; Figure 6 The overall process diagram of the simhash algorithm according to the embodiment of the present disclosure is schematically shown.

[0145] Combination Figure 5 and Figure 6 As shown, the method for obtaining feature values ​​of text data to be processed includes operations S501 to S504.

[0146] In operation S501, a word segmentation tool is used to decompose the text data to be processed into word sets.

[0147] According to an embodiment of the present disclosure, a word segmentation tool, such as HanLP, can be used to segment the sentence S, to generate a word set {W1, W2, ...Wn}. Since each word contains different information, a weight λi can be set for each word, where i represents the i-th word, and it is assumed that the weight includes 5 levels of 1 to 5.

[0148] According to an embodiment of the present disclosure, for example, in an article comment: "Xiao Ming from Class 2, Grade 3 said that his homework has 9 exercises and needs to be completed independently as soon as possible", after word segmentation, it becomes "Xiao Ming (3) from Class 2, Grade 3 (4) said that (1) homework (2) has (1) 9 (3) exercises (5) and needs to be (1) completed independently (3) as soon as possible (4) (5)", and the numbers in brackets are the weights of the words.

[0149] In operation S502, a hash value of each word in the word set is calculated.

[0150] According to an embodiment of the present disclosure, the hash value of each word in the word set can be calculated by the simhash algorithm, which is a text similarity detection algorithm based on text hash values.

[0151] According to an embodiment of the present disclosure, the hash value of each word can be calculated by a hash algorithm. For example, the hash value of "Grade 3" calculated by the hash algorithm is 100101, and the hash value of "Class 2" calculated by the hash algorithm is 101011, thereby obtaining a hash value set {H1, H2, ...Hn}, where Hi represents the hash value of the i-th word.

[0152] In operation S503, based on a preset weight, a weighted sum is performed on the hash value of each word in the word set to obtain a signature value of the text data to be processed.

[0153] According to an embodiment of the present disclosure, based on the hash value calculated in S502 and the weight set in S501, the words can be assigned weights to obtain the signature value of each word. Assuming the weight value is λ, the hash value of this bit is recorded as +λ when it is 1, and recorded as -λ when the hash value is 0. For example, the signature value of "Grade 3 (4)" is "4 -4 -4 4 -4 4", and the signature value of "Class 2 (5)" is "5 -5 5 -55 5".

[0154] According to the embodiments of the present disclosure, the signature values ​​calculated for each word can be accumulated to obtain a sequence string {Simhl, Simh2, ...Simhn}, where Simhj = ∑ni = 1λHij, Hij represents the jth bit of the hash value of the i-th word, and λ is the weight of the word.

[0155] According to an embodiment of the present disclosure, accumulation can be performed by adding the first digit of the hash value of all words in the sentence to the first digit, the second digit to the second digit, and so on. For example, the result of combining the two words "third grade" and "second class" is "9-91-119".

[0156] In operation S504, the signature value is subjected to dimensionality reduction processing to obtain a feature value of the text data to be processed.

[0157] According to an embodiment of the present disclosure, the result of S503 can be converted into a binary value 01 string, and each bit is traversed, and if Simh1 is greater than 0, the bit is set to 1, otherwise it is set to 0. For example, the dimensionality reduction result of "9 -9 1 -1 1 9" is "101011".

[0158] According to an embodiment of the present disclosure, the text data processing method further includes:

[0159] For each first text data in turn, the Hamming distance between the feature value of the first text data and the feature value of the text data to be processed is calculated to obtain a first similarity value.

[0160] According to an embodiment of the present disclosure, the Hamming distance may refer to the number of different characters at the same position of two strings of the same length.

[0161] According to an embodiment of the present disclosure, the Hamming distance between the feature value of the first text data and the feature value of the text data to be processed may be calculated by using the binary values ​​calculated for each of the text data to be processed.

[0162] According to an embodiment of the present disclosure, the similarities between text data may be compared according to the Hamming distance to obtain a first similarity value.

[0163] According to an embodiment of the present disclosure, the number of 1s after XORing the corresponding bits of two 01 strings, that is, the number of different corresponding bits, is the Hamming distance. For example, when Simhash value 1 is 0 1 0 1 0 0 1 1 and Simhash value 2 is 0 0 0 1 0 1 01, the Hamming distance of the two Simhash values ​​is 3, that is, the similarity of the corresponding two texts is 3.

[0164] According to an embodiment of the present disclosure, two texts may be considered similar when the Hamming distance is less than or equal to 3, and a threshold may also be set by those skilled in the art according to actual needs.

[0165] According to an embodiment of the present disclosure, by using the simhash algorithm to obtain the characteristic value of the text data to be processed, and using the Hamming distance calculation to obtain the first similarity value between the two text data, the similarity between the first text data and the text data to be processed can be located, so as to subsequently determine whether the text data to be processed is a redundant comment based on the size relationship between the first similarity value and the first preset value.

[0166] Figure 7 The flowchart of the method for deleting text data according to an embodiment of the present disclosure is schematically shown.

[0167] like Figure 7 As shown, the method for deleting text data includes operations S701-S702.

[0168] In operation S701, a scheduled task is set.

[0169] According to an embodiment of the present disclosure, for example, a scheduled task may be set at 12:00 and 24:00 every day.

[0170] In operation S702 , in response to triggering the scheduled task, text data stored in the first cache database and the second cache database are deleted.

[0171] According to an embodiment of the present disclosure, a scheduled task may be triggered at 12:00 and 24:00 to clear the text data cached in the first cache database and the second cache database.

[0172] According to the embodiments of the present disclosure, redundant comments within a certain period of time can be distinguished by periodically clearing text data in the cache database.

[0173] Figure 8 The overall flow chart of the text data processing method according to the embodiment of the present disclosure is schematically shown.

[0174] According to an embodiment of the present disclosure, the new comments may be used as text data to be processed, and the text hash value of the text data to be processed may be calculated using the simhash algorithm, and the similarity may be calculated using the Hamming distance.

[0175] According to an embodiment of the present disclosure, two cache databases may be used to perform data screening.

[0176] According to an embodiment of the present disclosure, the hash values ​​of redundant comments can be stored in the first cache database (denoted as redisl). After the hash value of the text data to be processed is calculated, the hash values ​​stored in the first cache database can be traversed, and the similarity with the hash value of the text data to be processed can be calculated. A high similarity indicates that the text data to be processed is a redundant comment, and the text data to be processed can be deleted.

[0177] According to an embodiment of the present disclosure, the second cache database (redis2) can store non-redundant comments, store the number of comments similar to the comment, and set the time for data update. For example, the second preset value can be set to 10. If the number of similar comments is greater than or equal to 10, it can be indicated that the comment is a redundant comment. At this time, all data similar to the comment in the second cache database can be deleted and stored in the first cache database; if the number of similar comments is less than 10, the comment can be stored in the second cache database.

[0178] According to an embodiment of the present disclosure, the preset duration of comments stored in the second cache database can be set to 10 minutes to facilitate subsequent risk control judgment and data warehousing, etc., so as to achieve persistent storage of data in the second cache database for a certain period of time.

[0179] According to an embodiment of the present disclosure, a scheduled task may be set at 12:00 and 24:00 every day, so that the scheduled task may be triggered to clear the text data cached in the first cache database and the second cache database.

[0180] According to the embodiments of the present disclosure, the text data processing method provided by the present disclosure is characterized by lightweight implementation. By using two cache databases, real-time filtering and screening of risky or sensitive information under articles is achieved, wherein the risky or sensitive information includes multiple similar comments sent within a short time interval. The cache database can store the comment data in memory, and filtering and deletion consumes less resources and is faster.

[0181] According to an embodiment of the present disclosure, a sensitive information system may be used to process text data.

[0182] According to an embodiment of the present disclosure, the sensitive information system may include modules such as vocabulary management, policy configuration, user configuration, content processing and data statistics.

[0183] According to an embodiment of the present disclosure, vocabulary management may include sensitive word maintenance, vocabulary setting and sensitive word evaluation, etc.; sensitive word maintenance may include word maintenance, vocabulary setting, sampling setting and word addition, etc.; vocabulary setting may include vocabulary maintenance and vocabulary addition, etc.

[0184] According to an embodiment of the present disclosure, policy configuration may include policy management, business management and vocabulary management, etc.; policy management may include red line vocabulary, dangerous vocabulary and sensitive vocabulary, etc.; business management may include search business, production business and personal information, etc.; vocabulary management may include red line vocabulary, dangerous vocabulary and sensitive vocabulary, etc.

[0185] According to an embodiment of the present disclosure, user configuration may include user policy and the like.

[0186] According to an embodiment of the present disclosure, content processing may include user hits and text hits, etc.; user hits are associated with user policies, and text hits are associated with policy management.

[0187] According to an embodiment of the present disclosure, when processing text data, a sensitive information system may involve a user end and a background management end.

[0188] According to the embodiments of the present disclosure, on the user side, when the user uses the APP to search for services, produce services, fill in personal information and text information, it will go through policy routing, calculate the feature value based on the simhash algorithm, and transmit the feature value to the cache database; at the same time, the text data corresponding to the feature value can be fed back to the background management side according to the feature value, so as to remind new words.

[0189] According to an embodiment of the present disclosure, at the background management end, the system can add words to the cache database by creating and initializing the word library; at the same time, the cache database can be policy-maintained through redline thesaurus, dangerous thesaurus, and sensitive thesaurus.

[0190] It should be noted that, unless it is explicitly stated that there is a sequence of execution between different operations shown in the flowchart in the embodiments of the present disclosure, or different operations have a sequence of execution in technical implementation, otherwise, the execution order of multiple operations may not be prioritized, and multiple operations may also be executed simultaneously.

[0191] Fig. 9 The block diagram of a text data processing device according to an embodiment of the present disclosure is schematically shown.

[0192] like Fig. 9 As shown, the text data processing device 900 includes an acquisition module 901 , a first determination module 902 , a first calculation module 903 and a second determination module 904 .

[0193] The acquisition module 901 is used to acquire text data to be processed.

[0194] The first determination module 902 is used to determine the feature value of the text data to be processed based on a similarity detection algorithm.

[0195] The first calculation module 903 is used to sequentially use the feature values ​​of the plurality of first text data stored in the first cache database and the feature values ​​of the text data to be processed to perform similarity calculation to obtain a first similarity value.

[0196] The second determination module 904 is configured to determine that the text data to be processed is target text data in response to the first similarity value currently calculated being less than or equal to a first preset value.

[0197] According to the embodiments of the present disclosure, a technical means is adopted to identify the text to be processed by detecting the characteristic values ​​of the text data to be processed and the first text data, wherein the first text data is text stored in the first cache database and has been determined to be redundant information. When the similarity between the text to be processed and the first text data is less than a first preset value, the text to be processed can be considered to be redundant information. Therefore, the technical problems in the related technologies of heavy implementation, reliance on machine learning, and high demand for computing resources are at least partially overcome, thereby achieving the technical effect of improving the recognition efficiency of text and reducing the consumption of computing resources.

[0198] According to an embodiment of the present disclosure, the above-mentioned text data processing device 900 further includes a selection module, a second calculation module, a third determination module, a first storage module and a fourth determination module.

[0199] The selection module is used to randomly select a preset number of text data from the second cache database to obtain a plurality of second text data when all the calculated first similarity values ​​are greater than the first preset value.

[0200] The second calculation module is used to use the characteristic values ​​of the plurality of second text data to perform similarity calculations with the characteristic values ​​of the text data to be processed, so as to obtain a plurality of second similarity values.

[0201] The third determination module is configured to determine the amount of third text data based on the plurality of second similarity values, wherein the third text data includes the second text data whose second similarity value is less than or equal to the first preset value.

[0202] The first storage module is used for storing the to-be-processed text data into the second cache database when the amount of the third text data is less than a second preset value.

[0203] The fourth determining module is used to delete the third text data in the second cache database and determine the to-be-processed text data and the third text data as target text data when the amount of the third text data is greater than or equal to the second preset value.

[0204] According to an embodiment of the present disclosure, the text data processing device 900 further includes a first setting module.

[0205] The first setting module is used to set a time mark for the text data to be processed based on the system time and a preset time length when storing the text data to be processed in the second cache database.

[0206] According to an embodiment of the present disclosure, the text data processing device 900 further includes a polling module, an extraction module and a second storage module.

[0207] The polling module is used to poll the time stamp of the text data in the second cache database.

[0208] The extraction module is used to extract the current text data from the second cache database when the time represented by the time mark of the current text data is later than the current system time.

[0209] The second storage module is used to store the above current text data in a database.

[0210] According to an embodiment of the present disclosure, the text data processing device 900 further includes a third storage module.

[0211] The third storage module is used to store the target text data into the first cache database.

[0212] According to an embodiment of the present disclosure, the first determination module 902 includes a decomposition unit, a first calculation unit, a summation unit and a dimension reduction unit.

[0213] The decomposition unit is used to decompose the above text data to be processed into word sets by using a word segmentation tool.

[0214] The first calculation unit is used to calculate the hash value of each word in the above word set.

[0215] The summing unit is used to perform weighted summation on the hash values ​​of the various words in the above-mentioned word set based on preset weights to obtain the signature value of the above-mentioned text data to be processed.

[0216] The dimension reduction unit is used to perform dimension reduction processing on the above-mentioned signature value to obtain the characteristic value of the above-mentioned text data to be processed.

[0217] According to an embodiment of the present disclosure, the first calculation module 903 includes a second calculation unit.

[0218] The second calculation unit is used to calculate the Hamming distance between the feature value of the first text data and the feature value of the text data to be processed for each of the first text data in turn, so as to obtain the first similarity value.

[0219] According to an embodiment of the present disclosure, the text data processing device 900 further includes a second setting module and a deletion module.

[0220] The second setting module is used to set the scheduled tasks.

[0221] The deleting module is used to delete the text data stored in the first cache database and the second cache database in response to triggering the above-mentioned scheduled task.

[0222] According to the embodiments of the present invention, any one or more of the modules, submodules, units, and subunits, or at least part of the functions of any one of them can be implemented in one module. According to the embodiments of the present invention, any one or more of the modules, submodules, units, and subunits can be split into multiple modules for implementation. According to the embodiments of the present invention, any one or more of the modules, submodules, units, and subunits can be at least partially implemented as hardware circuits, such as field programmable gate arrays (FPGAs), programmable logic arrays (PLAs), systems on chips, systems on substrates, systems on packages, application specific integrated circuits (ASICs), or can be implemented by hardware or firmware in any other reasonable way of integrating or packaging the circuit, or implemented in any one of the three implementation methods of software, hardware, and firmware, or in any appropriate combination of any of them. Alternatively, according to the embodiments of the present invention, one or more of the modules, submodules, units, and subunits can be at least partially implemented as computer program modules, and when the computer program modules are run, the corresponding functions can be performed.

[0223] For example, any multiple of the acquisition module 901, the first determination module 902, the first calculation module 903 and the second determination module 904 can be combined in one module / unit / subunit for implementation, or any one of the modules / units / subunits can be split into multiple modules / units / subunits. Alternatively, at least part of the functions of one or more modules / units / subunits in these modules / units / subunits can be combined with at least part of the functions of other modules / units / subunits and implemented in one module / unit / subunit. According to an embodiment of the present disclosure, at least one of the acquisition module 901, the first determination module 902, the first calculation module 903 and the second determination module 904 can be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on a chip, a system on a substrate, a system on a package, an application specific integrated circuit (ASIC), or can be implemented by hardware or firmware such as any other reasonable way of integrating or packaging the circuit, or by any one of the three implementation methods of software, hardware and firmware or by a suitable combination of any of them. Alternatively, at least one of the acquisition module 901, the first determination module 902, the first calculation module 903 and the second determination module 904 may be at least partially implemented as a computer program module, and when the computer program module is executed, a corresponding function may be executed.

[0224] It should be noted that the text data processing device part in the embodiment of the present disclosure corresponds to the text data processing method part in the embodiment of the present disclosure. The description of the text data processing device part specifically refers to the text data processing method part, which will not be repeated here.

[0225] Fig.10 The block diagram schematically shows a computer system suitable for implementing the text data processing method according to an embodiment of the present disclosure. Fig.10 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0226] like Fig.10 As shown, the computer electronic device 1000 according to the embodiment of the present disclosure includes a processor 1001, which can perform various appropriate actions and processes according to the program stored in the read-only memory (ROM) 1002 or the program loaded from the storage part 1008 to the random access memory (RAM) 1003. The processor 1001 may include, for example, a general-purpose microprocessor (such as a CPU), an instruction set processor and / or a related chipset and / or a special-purpose microprocessor (for example, an application-specific integrated circuit (ASIC)), etc. The processor 1001 may also include an onboard memory for caching purposes. The processor 1001 may include a single processing unit or multiple processing units for performing different actions of the method flow according to the embodiment of the present disclosure.

[0227] In RAM 1003, various programs and data required for the operation of electronic device 1000 are stored. Processor 1001, ROM 1002 and RAM 1003 are connected to each other via bus 1004. Processor 1001 performs various operations of the method flow according to the embodiment of the present disclosure by executing the program in ROM 1002 and / or RAM 1003. It should be noted that the program can also be stored in one or more memories other than ROM 1002 and RAM 1003. Processor 1001 can also perform various operations of the method flow according to the embodiment of the present disclosure by executing the program stored in the one or more memories.

[0228] According to an embodiment of the present disclosure, the electronic device 1000 may further include an input / output (I / O) interface 1005, which is also connected to the bus 1004. The electronic device 1000 may further include one or more of the following components connected to the I / O interface 1005: an input portion 1006 including a keyboard, a mouse, etc.; an output portion 1007 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage portion 1008 including a hard disk, etc.; and a communication portion 1009 including a network interface card such as a LAN card, a modem, etc. The communication portion 1109 performs communication processing via a network such as the Internet. A drive 1010 is also connected to the I / O interface 1005 as needed. A removable medium 1011, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 1010 as needed, so that a computer program read therefrom is installed into the storage portion 1008 as needed.

[0229] According to an embodiment of the present disclosure, the method flow according to an embodiment of the present disclosure can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a computer-readable storage medium, and the computer program contains a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 1009, and / or installed from the removable medium 1011. When the computer program is executed by the processor 1001, the above-mentioned functions defined in the system of the embodiment of the present disclosure are executed. According to an embodiment of the present disclosure, the system, equipment, device, module, unit, etc. described above can be implemented by a computer program module.

[0230] The present disclosure also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or may exist independently without being assembled into the device / apparatus / system. The above computer-readable storage medium carries one or more programs, and when the above one or more programs are executed, the method according to the embodiment of the present disclosure is implemented.

[0231] According to an embodiment of the present disclosure, the computer-readable storage medium may be a non-volatile computer-readable storage medium. For example, it may include, but is not limited to: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction execution system, apparatus, or device.

[0232] For example, according to an embodiment of the present disclosure, the computer-readable storage medium may include the ROM 1002 and / or the RAM 1003 described above and / or one or more memories other than the ROM 1002 and the RAM 1003 .

[0233] The embodiments of the present disclosure also include a computer program product, which includes a computer program, which contains program code for executing the method provided by the embodiments of the present disclosure. When the computer program product runs on an electronic device, the program code is used to enable the electronic device to implement the text data processing method provided by the embodiments of the present disclosure.

[0234] When the computer program is executed by the processor 1001, the above functions defined in the system / device of the embodiment of the present disclosure are executed. According to the embodiment of the present disclosure, the system, device, module, unit, etc. described above can be implemented by a computer program module.

[0235] In one embodiment, the computer program may rely on tangible storage media such as optical storage devices, magnetic storage devices, etc. In another embodiment, the computer program may also be transmitted and distributed in the form of signals on a network medium, and downloaded and installed through the communication part 1009, and / or installed from the removable medium 1011. The program code contained in the computer program may be transmitted using any appropriate network medium, including but not limited to: wireless, wired, etc., or any suitable combination of the above.

[0236] According to an embodiment of the present disclosure, the program code for executing the computer program provided by the embodiment of the present disclosure can be written in any combination of one or more programming languages. Specifically, these computing programs can be implemented using high-level process and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, Java, C++, python, "C" language or similar programming languages. The program code can be executed entirely on the user computing device, partially on the user device, partially on the remote computing device, or entirely on the remote computing device or server. In the case of a remote computing device, the remote computing device can be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computing device (for example, using an Internet service provider to connect through the Internet).

[0237] The flowcharts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flowchart or block diagram may represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of the code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box may also occur in an order different from that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flowchart, and the combination of boxes in the block diagram or flowchart, can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions. It can be understood by those skilled in the art that the features recorded in the various embodiments and / or claims of the present disclosure can be combined and / or combined in a variety of ways, even if such a combination or combination is not explicitly recorded in the present disclosure. In particular, without departing from the spirit and teaching of the present disclosure, the features described in the various embodiments and / or claims of the present disclosure may be combined and / or combined in a variety of ways. All of these combinations and / or combinations fall within the scope of the present disclosure.

[0238] The embodiments of the present disclosure are described above. However, these embodiments are only for illustrative purposes and are not intended to limit the scope of the present disclosure. Although the embodiments are described above separately, this does not mean that the measures in the various embodiments cannot be used in combination to advantage. The scope of the present disclosure is defined by the attached claims and their equivalents. Without departing from the scope of the present disclosure, those skilled in the art may make a variety of substitutions and modifications, which should all fall within the scope of the present disclosure.

Claims

1. A text data processing method, comprising: Get the text data to be processed; Based on a similarity detection algorithm, determining a feature value of the text data to be processed; Sequentially use the feature values ​​of the plurality of first text data stored in the first cache database and the feature value of the text data to be processed to perform similarity calculation to obtain a first similarity value; In response to the first similarity value currently calculated being less than or equal to the first preset value, determining that the text data to be processed is the target text data; In the case that all the calculated first similarity values ​​are greater than the first preset value, randomly selecting a preset number of text data from the second cache database to obtain a plurality of second text data; Using the characteristic values ​​of the plurality of second text data to respectively calculate similarity with the characteristic values ​​of the text data to be processed, to obtain a plurality of second similarity values; Based on the plurality of second similarity values, determining the amount of third text data, wherein the third text data includes the second text data whose second similarity value is less than or equal to the first preset value; When the amount of the third text data is less than a second preset value, storing the to-be-processed text data in the second cache database; as well as When the amount of the third text data is greater than or equal to the second preset value, the third text data in the second cache database is deleted, and the to-be-processed text data and the third text data are determined to be target text data.

2. The method according to claim 1, further comprising: When storing the text data to be processed into the second cache database, a time mark is set for the text data to be processed based on the system time and a preset time length.

3. The method according to claim 2, further comprising: Polling the time stamp of the text data in the second cache database; When the time indicated by the time mark of the current text data is later than the current system time, extracting the current text data from the second cache database; and The current text data is stored in a database.

4. The method according to any one of claims 1 to 3, further comprising: The target text data is stored in the first cache database.

5. The method according to any one of claims 1 to 3, further comprising: Set up scheduled tasks; In response to triggering the scheduled task, the text data stored in the first cache database and the second cache database are deleted.

6. The method according to claim 1, wherein: The determining of the feature value of the text data to be processed based on the similarity detection algorithm includes: Using a word segmentation tool, decomposing the text data to be processed into word sets; Calculating a hash value for each word in the word set; Based on a preset weight, performing weighted summation on the hash values ​​of each word in the word set to obtain a signature value of the text data to be processed; and Perform dimensionality reduction processing on the signature value to obtain the feature value of the text data to be processed.

7. The method according to claim 6, wherein: The method of sequentially using the feature values ​​of the plurality of first text data stored in the first cache database and the feature values ​​of the text data to be processed to perform similarity calculation to obtain a first similarity value includes: For each of the first text data in turn, the Hamming distance between the feature value of the first text data and the feature value of the text data to be processed is calculated to obtain the first similarity value.

8. A text data processing device, comprising: An acquisition module, used to acquire text data to be processed; A first determination module, used for determining the feature value of the text data to be processed based on a similarity detection algorithm; A first calculation module, used to sequentially use the feature values ​​of the plurality of first text data stored in the first cache database and the feature values ​​of the text data to be processed to perform similarity calculation to obtain a first similarity value; A second determination module, configured to determine that the text data to be processed is target text data in response to the first similarity value currently calculated being less than or equal to a first preset value; A selection module, configured to randomly select a preset number of text data from a second cache database to obtain a plurality of second text data when all calculated first similarity values ​​are greater than the first preset value; A second calculation module, used to use the characteristic values ​​of the second text data to perform similarity calculations with the characteristic values ​​of the text data to be processed, respectively, to obtain a plurality of second similarity values; A third determining module, configured to determine the amount of third text data based on the plurality of second similarity values, wherein the third text data includes the second text data whose second similarity value is less than or equal to the first preset value; A first storage module is configured to store the to-be-processed text data in the second cache database when the amount of the third text data is less than a second preset value; and The fourth determining module is used to delete the third text data in the second cache database when the amount of the third text data is greater than or equal to the second preset value, and determine the to-be-processed text data and the third text data as target text data.

9. An electronic device, comprising: one or more processors; a memory for storing one or more instructions, Wherein, when the one or more instructions are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 7. 10 . A computer-readable storage medium having executable instructions stored thereon, wherein when the executable instructions are executed by a processor, the processor is enabled to implement the method according to any one of claims 1 to 7.

11. A computer program product, comprising computer executable instructions, wherein the computer executable instructions are used to implement the method according to any one of claims 1 to 7 when executed.

Images