Text recognition method and device, storage medium and electronic device

Abstract

The present invention discloses a text recognition method and device, a storage medium and an electronic device. The method comprises: obtaining a target word list corresponding to a target text to be recognized; using the target word list to generate a target text vector corresponding to the target text; searching for a target text clustering cluster matching the target text vector in a stored text clustering cluster set, the text clustering cluster set will be updated regularly, and the text clustering cluster set includes an object text clustering cluster configured with a text type label, the text type label is used to indicate that the object text clustering cluster is a malicious clustering cluster or a non-malicious clustering cluster; when the target text clustering cluster is found in the text clustering cluster set, and the text type label corresponding to the target text clustering cluster indicates that the target text clustering cluster is a malicious clustering cluster, the target text is determined as an abnormal text. The present invention solves the technical problem of low text recognition efficiency caused by the inability to timely discover junk text due to offline processing.

Description

Technical Field

[0001] The present invention relates to the field of the Internet, and in particular to a text recognition method and device, a storage medium and an electronic device. Background Art

[0002] In many application scenarios, text information is often needed to complete the interaction process. For example, the text information here can be external link forwarding text, shared text in the community sharing space, comment text, comment text, greeting text used when binding account relationships, etc.

[0003] However, some illegal entities often push junk text messages on a large scale to achieve some malicious purposes. For example, in the forwarding text of the external link forwarding scenario, there will be a large number of fake red envelope text messages with the nature of inducing transmission, in the conversation text of the group conversation scenario, there will be a large number of gambling-related texts, in the shared text, there will be malicious texts related to the black five categories, in the comment text, there will be a large number of diversion-related texts, and in the greeting text, there will be a large number of harassing contents. Among them, these junk text messages have the following characteristics: highly similar content; huge quantity.

[0004] For the above-mentioned junk text, the commonly used identification method is to use the traditional hash algorithm to calculate the text similarity between the text to be identified and the known junk text. When it is determined that the text to be identified is similar to the known junk text, the source end that sent the text to be identified is directly blocked. The text similarity calculated by the traditional hash algorithm in an offline state not only has poor accuracy, but also cannot detect junk text in time due to offline processing, resulting in low text recognition efficiency.

[0005] To address the above-mentioned problems, no effective solution has been proposed yet. Summary of the invention

[0006] The embodiments of the present invention provide a text recognition method and device, a storage medium and an electronic device to at least solve the technical problem of low text recognition efficiency caused by the inability to timely detect junk text due to offline processing.

[0007] According to one aspect of an embodiment of the present invention, a text recognition method is provided, comprising: obtaining a target word list corresponding to a target text to be recognized; generating a target text vector corresponding to the target text using the target word list; searching for a target text clustering cluster matching the target text vector in a stored text clustering cluster set, wherein the text clustering cluster set will be updated regularly, and the text clustering cluster set includes an object text clustering cluster configured with a text type label, and the text type label is used to indicate that the object text clustering cluster is a malicious clustering cluster or a non-malicious clustering cluster; when the target text clustering cluster is found in the text clustering cluster set, and the text type label corresponding to the target text clustering cluster indicates that the target text clustering cluster is a malicious clustering cluster, the target text is determined as an abnormal text.

[0008] According to another aspect of an embodiment of the present invention, a text recognition device is also provided, including: an acquisition unit, used to acquire a target word list corresponding to a target text to be recognized; a generation unit, used to generate a target text vector corresponding to the target text using the target word list; a search unit, used to search for a target text clustering cluster matching the target text vector in a stored text clustering cluster set, wherein the text clustering cluster set will be updated regularly, and the text clustering cluster set includes an object text clustering cluster configured with a text type label, and the text type label is used to indicate that the object text clustering cluster is a malicious clustering cluster or a non-malicious clustering cluster; an identification unit, used to determine the target text as an abnormal text when the target text clustering cluster is found in the text clustering cluster set and the text type label corresponding to the target text clustering cluster indicates that the target text clustering cluster is a malicious clustering cluster.

[0009] According to another aspect of the embodiments of the present invention, a computer-readable storage medium is provided, in which a computer program is stored, wherein the computer program is configured to execute the above-mentioned text recognition method when running.

[0010] According to another aspect of an embodiment of the present invention, there is provided an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to execute the text recognition method through the computer program.

[0011] In an embodiment of the present invention, after obtaining a target word list corresponding to a target text to be identified, a target text vector corresponding to the target text is generated using the target word list. Then, a target text cluster matching the target text vector is searched in a stored text cluster set, where the text cluster set will be updated regularly. The text cluster set includes an object text cluster configured with a text type label, and the text type label is used to indicate whether the object text cluster is a malicious cluster. If a target text cluster is found in the text cluster set, and the text type label corresponding to the target text cluster indicates a malicious cluster, the target text is identified as abnormal text. That is to say, after obtaining the target text vector corresponding to the target text to be identified, a search is performed in the set of text clustering clusters that are regularly updated and configured with text type labels to determine whether there is a target text clustering cluster corresponding to the target text vector of the target text, thereby directly using the text clustering clusters of the determined text type to identify the text type of the target text, without the need to perform offline recognition for each text separately, thereby achieving the purpose of improving text recognition efficiency and overcoming the problem in related technologies that junk text cannot be discovered in a timely manner due to offline processing, resulting in low text recognition efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of this application. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0013] Figure 1 is a schematic diagram of a hardware environment of an optional text recognition method according to an embodiment of the present invention;

[0014] Figure 2 is a schematic diagram of a hardware environment of another optional text recognition method according to an embodiment of the present invention;

[0015] Figure 3 is a flowchart of another optional text recognition method according to an embodiment of the present invention;

[0016] Figure 4 is a recognition logic architecture diagram of another optional text recognition method according to an embodiment of the present invention;

[0017] Figure 5 is a flowchart of another optional text recognition method according to an embodiment of the present invention;

[0018] Figure 6 is a schematic diagram of an optional text recognition method according to an embodiment of the present invention;

[0019] Figure 7is a flowchart of another optional text recognition method according to an embodiment of the present invention;

[0020] Figure 8 is a schematic diagram of another optional text recognition method according to an embodiment of the present invention;

[0021] Fig. 9 is a flowchart of another optional text recognition method according to an embodiment of the present invention;

[0022] Fig.10 is a schematic diagram of another optional text recognition method according to an embodiment of the present invention;

[0023] Fig.11 is a schematic structural diagram of an optional text recognition device according to an embodiment of the present invention;

[0024] Fig.12 It is a schematic diagram of the structure of an optional electronic device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0025] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0026] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0027] It should be noted that the text recognition method provided in the embodiments of the present application will involve the following technical terms:

[0028] Natural language processing (NLP) is an important direction in the fields of computer science and artificial intelligence. It studies various theories and methods that can achieve effective communication between people and computers using natural language. Natural language processing is a science that integrates linguistics, computer science, and mathematics. Therefore, research in this field will involve natural language, that is, the language people use in daily life, so it is closely related to the study of linguistics. Natural language processing technology usually includes text processing, semantic understanding, machine translation, robot question answering, knowledge graph and other technologies.

[0029] Machine Learning (ML) is a multi-disciplinary subject that involves probability theory, statistics, approximation theory, convex analysis, algorithm complexity theory and other disciplines. It specializes in studying how computers simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to make computers intelligent. Its applications are spread across all areas of artificial intelligence. Machine learning and deep learning usually include artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and self-learning.

[0030] Text vector: Extract the semantic information of the text and represent it in vector form. The vector is used instead of the text during algorithm processing.

[0031] Real-time text clustering: In real-time text data streams, similar texts are clustered together to form a cluster structure.

[0032] Bidirectional Encoder Representations from Transformers (Bert): a natural language processing model based on Transformer modeling.

[0033] Hierarchical Navigable Small World graphs (HNSW) are a data structure used for nearest-neighbor similarity search.

[0034] Key-Value (kv) distributed storage system has fast query speed, large data storage volume, and supports high concurrency. It is suitable for querying by primary key, but cannot perform complex conditional queries.

[0035] According to one aspect of an embodiment of the present invention, a text recognition method is provided. Optionally, as an optional implementation, the text recognition method can be but is not limited to being applied to Figure 1In the text recognition system in the hardware environment shown, the text recognition system may include but is not limited to a terminal device 102, a network 104, and a server 106. The terminal device 102 includes a human-computer interaction screen 1022, a processor 1024, and a memory 1026. The human-computer interaction screen 1022 is used to obtain a target text to be recognized through a human-computer interaction interface. The processor 1024 is used to encode and preprocess the target text to be recognized, and the memory 1026 is used to store the target text to be recognized.

[0036] In addition, the server 106 includes a database 1062 and a processing engine 1064. The database 1062 is used to store text cluster sets. The processing engine 1064 is used to process the target text received from the terminal device 102 and search the text cluster set for the target text cluster corresponding to the target text vector of the target text.

[0037] As an optional implementation, assume that the text recognition process provided in this embodiment is performed on the comment text in the comment scene. The specific process can be as follows: Steps S102-S104, where the target text to be recognized is obtained in the terminal device 102, and the target text is sent to the server 106 via the network 104. Then the server 106 performs the following operations:

[0038] S106, performing text preprocessing on the target text to obtain a target word list;

[0039] S108, using the target word list to generate a target text vector corresponding to the target text;

[0040] S110, searching, through the processing engine 1064, a target text cluster that matches the target text vector in the text cluster set stored in the database;

[0041] S112: When a target text cluster is found and the text type label corresponding to the target text cluster indicates a malicious cluster, the target text is determined as an abnormal text.

[0042] Optionally, the recognition result of the target text may be returned to the terminal device 102 via the network as in step S114. Figure 1 The interactive process shown is an example, and the recognition result can also be directly saved to the server for reference in the next recognition process.

[0043] As an optional implementation, assuming that the above Figure 1Each device in the text recognition system shown performs the text recognition process provided in this embodiment on the comment text in the comment scene. The specific process can be as follows: Steps S202-S206, the target text to be recognized is obtained in the terminal device 102, and the target text is preprocessed in the terminal device 102 to obtain a target word list; the target text vector corresponding to the target text is generated using the target word list. Then, as in step S208, the target text is sent to the server 106 via the network 104. The server 106 then performs the following operations:

[0044] S210, searching, through the processing engine 1064, a target text cluster that matches the target text vector in the text cluster set stored in the database;

[0045] S212: When a target text cluster is found and the text type label corresponding to the target text cluster indicates a malicious cluster, the target text is determined as an abnormal text.

[0046] Optionally, the recognition result of the target text may be returned to the terminal device 102 via the network as in step S214. Figure 2 The interactive process shown is an example, and the recognition result can also be directly saved to the server for reference in the next recognition process.

[0047] That is to say, in this embodiment, the process of converting the target text to be recognized into the target text vector can be completed in the terminal device 102 or in the server 106, and no limitation is made here.

[0048] It should be noted that, in this embodiment, after obtaining the target word list corresponding to the target text to be identified, the target text vector corresponding to the target text is generated using the target word list. Then, the target text cluster cluster matching the target text vector is searched in the stored text cluster cluster set, where the text cluster cluster set will be updated regularly, and the text cluster cluster set includes an object text cluster cluster configured with a text type label, and the above text type label is used to indicate whether the object text cluster cluster is a malicious cluster cluster. If the target text cluster cluster is found in the text cluster cluster set, and the text type label corresponding to the target text cluster cluster indicates a malicious cluster cluster, the above target text recognition is determined as abnormal text. That is to say, in the text cluster cluster set that is regularly updated and configured with a text type label, it is found whether there is a target text cluster cluster corresponding to the target text, so as to directly use the text cluster cluster of the determined text type to identify the text type of the target text, and it is no longer necessary to perform offline recognition for each text separately, so as to achieve the purpose of improving the text recognition efficiency, thereby overcoming the problem of low text recognition efficiency caused by the inability to timely discover junk text due to offline processing in the related art. In addition, if malicious texts can be identified quickly and efficiently, it will also help purify the environment of network platforms that display or share content based on user-generated content, avoid a large amount of illegal text information from interfering with the user's browsing or viewing process, provide a sunny network environment for the majority of users, and ensure the safety of their browsing.

[0049] Optionally, in this embodiment, the terminal device may be a terminal device configured with a target client, which may include but is not limited to at least one of the following: a mobile phone (such as an Android phone, an iOS phone, etc.), a laptop, a tablet computer, a PDA, a MID (Mobile Internet Devices), a PAD, a desktop computer, a smart TV, etc. The target client may be a community shared space application client, an instant messaging session client, a video playback application client, a content distribution application client, etc. The network may include but is not limited to: a wired network, a wireless network, wherein the wired network includes: a local area network, a metropolitan area network and a wide area network, and the wireless network includes: Bluetooth, WIFI and other networks that implement wireless communication. The server may be a single server, or a server cluster consisting of multiple servers, or a cloud server. The above is only an example, and this embodiment does not impose any limitation on this.

[0050] Optionally, as an optional implementation, as Figure 3 As shown, the above text recognition method includes:

[0051] S302, obtaining a target word list corresponding to the target text to be recognized;

[0052] S304, generating a target text vector corresponding to the target text using the target word list;

[0053] S306, searching for a target text clustering cluster matching the target text vector in the stored text clustering cluster set, wherein the text clustering cluster set will be updated regularly, and the text clustering cluster set includes an object text clustering cluster configured with a text type label, and the text type label is used to indicate that the object text clustering cluster is a malicious clustering cluster or a non-malicious clustering cluster;

[0054] S308: When a target text cluster is found in the text cluster set and the text type label corresponding to the target text cluster indicates that the target text cluster is a malicious cluster, the target text is determined as abnormal text.

[0055] Optionally, in this embodiment, the above text recognition method can be applied to, but not limited to, application scenarios for content display or promotion based on user-generated content (UGC), where UGC refers to displaying or providing one's own original content to other users through an Internet platform. For example, friend social networks, video / photo / knowledge sharing networks, communities / forums and other scenarios. Optionally, the text information in the above application scenarios corresponding to the text recognition method provided in this embodiment applied to the above application scenarios may include but is not limited to at least one of the following: forwarding text information in external link forwarding scenarios, shared text information in community shared spaces, comment text information on displayed content, and greeting text information used when binding account relationships (such as registration, adding friend accounts, etc.). The recognition process here is mainly to identify junk texts that are published by users through network platforms in different above application scenarios, appear frequently, and have repeated content, such as text information sent by users in the form of "drift bottles", which may be repeated content or even involve illegal sensitive information (such as pornographic gambling and drug information). These spam texts are usually malicious texts triggered by malicious entities without achieving a certain purpose. They cause great information interference to users who are browsing web pages or watching videos and other content they are interested in, making the Internet browsing environment of the majority of users unsafe.

[0056] Therefore, in order to overcome the above problems, it is necessary to timely identify and clean up the malicious text in the online business flow. However, in the related art, the commonly used identification method is to use the traditional hash algorithm to perform offline identification on the text in the business flow, which not only has poor identification accuracy, but also has poor real-time performance, and cannot timely discover the malicious text appearing in the online data. In the embodiment of the present application, after obtaining the target text vector corresponding to the target text to be identified, the target text cluster cluster corresponding to the target text vector will be searched in the text cluster cluster set that is regularly updated and configured with the text type label, so as to directly use the text cluster cluster of the determined text type to identify the text type of the target text, and no longer need to perform offline identification for each text separately, so as to achieve the purpose of improving the text recognition efficiency, thereby overcoming the problem of low text recognition efficiency caused by the inability to timely discover junk text due to offline processing in the related art. In addition, in the case of quickly and efficiently identifying malicious text, it will also be beneficial to purify the environment of the network platform for content display or sharing based on user-generated content, avoid a large amount of illegal text information from causing information interference to the user's browsing or viewing process, provide a sunny network environment for the majority of users, and ensure the browsing security of users.

[0057] Optionally, in this embodiment, the above text recognition method can be applied to, but is not limited to, Figure 4 In the recognition logic architecture shown, the recognition logic architecture may include but is not limited to: a text vector generation module 402, a vector search module 404, a real-time clustering module 406 and a label assignment module 408. Among them, the text vector generation module 402 is used to convert the target word list of the target text into a target text vector; the vector search module 404 stores a text clustering cluster set, and the module is used to find a target text clustering cluster matching the target text vector in the stored text clustering cluster set; the real-time clustering module 406 is used to cluster the target text that meets the clustering condition when the target text clustering cluster matching the target text vector is not found in the stored text clustering cluster set, so as to generate a new text clustering cluster, and add the new text clustering cluster that meets the addition condition to the text clustering cluster set. The label assignment module 408 is used to assign a text type label to the new text clustering cluster. It should be noted that in this embodiment, for a text cluster, by assigning a text type label (marking) to any text in it, the purpose of batch marking all texts in the text cluster can be achieved without repeated operations, thereby achieving the effect of improving marking efficiency.

[0058] Optionally, in this embodiment, the text vector corresponding to the target text generated by the target word list can be implemented by, but not limited to, a model obtained by distilling the model using a Bert-based language processing model. That is to say, the above-mentioned text vector generation module can include, but is not limited to, a model based on TinyBert processed by the model distillation technology. Among them, the Bert-based language processing model is a super-large-scale neural network, which uses massive corpus data for unsupervised learning and can be applied to natural language processing tasks such as text classification, question and answer, similarity, sentiment analysis, sequence labeling, etc. In addition, the model structure of TinyBert after distillation is the same as that of Bert, but the number of network layers becomes smaller, and the number of parameters involved in the operation in the network becomes less. In this way, by reducing the size of the model, the process of converting text into text vectors will be accelerated, which will further help improve the recognition efficiency of the text.

[0059] Optionally, in the present embodiment, the stored text clustering cluster set may include, but is not limited to, a plurality of object text clustering clusters. The object text clustering cluster here may be a clustering structure formed by clustering similar texts in a real-time text data stream. For each clustering structure, there is a cluster center vector, and text vectors whose distance from the cluster center vector is less than a threshold value may be, but is not limited to, classified as the clustering structure, that is, determining that the text vector hits the object text clustering cluster corresponding to the clustering structure, i.e., the text vector is similar to each text included in the object text clustering cluster.

[0060] Furthermore, when a target text clustering cluster matching the target text vector is found in the stored text clustering cluster set, it means that the target text hits the target text clustering cluster in the text clustering cluster set. Then, based on the text type label configured in the target text clustering cluster, the type label of the target text can be determined, thereby achieving the effect of quickly and efficiently identifying whether the target text is junk text that is regarded as malicious text.

[0061] It should be noted that the above-mentioned text clustering cluster set can be updated regularly, and the update cycle of the regular update here can be but not limited to 1 week, 1 day or 1 hour, etc., and the specific cycle value can be but not limited to be set according to the actual scenario. That is to say, by timely updating the text clustering cluster set, it will be possible to further realize real-time recognition of text and achieve the purpose of timely discovering malicious text. In addition, in this embodiment, the update scenario of the above-mentioned text clustering cluster set is to obtain the object text cluster cluster obtained by new clustering. That is to say, in the case where the current target text does not find the target text cluster cluster in the stored text clustering cluster set, the target text can be temporarily cached. In the case where it meets the clustering condition, the target text can be used to generate a new object text cluster cluster in real time, and in the case where the adding condition is met, the new object text cluster cluster is added to the text clustering cluster set, and the relevant attribute information is adaptively adjusted and updated.

[0062] Optionally, in this embodiment, after obtaining a new object clustering module through the real-time clustering process, a new cluster identifier (labeling) can be assigned to it. It should be noted that in this embodiment, the first N popular object text clustering clusters in the current period can be pulled from the stored text clustering cluster set at a fixed time (such as every hour), but not limited to, and one text is randomly selected from each of the N object text clustering clusters for labeling to determine whether the type of text in the object text clustering cluster is junk text that is regarded as malicious text, and N is a positive integer.

[0063] In addition, in this embodiment, the above-mentioned recognition logic architecture may also include but is not limited to a supervisory control module, which is used to: analyze and count the text clustering cluster set based on the analysis function of some data analysis tools to obtain analysis results, and display the analysis results in real time. For example, in the case where the target text is identified as malicious text, the content in the target text clustering cluster where the target text is located can be tracked in real time, and a trend curve showing its magnitude change can be drawn. Furthermore, after the trend curve is detected based on the anomaly detection algorithm, automatic warning information can be issued in time before the malicious text breaks out to avoid unnecessary losses to users.

[0064] Specific combination Figure 5The example shown is used for illustration: it is assumed that the online part includes a logic center 500, which includes a preprocessing logic 500-2, a reporting logic 500-4 and a processing logic 500-6. Here, the online part obtains the target text to be recognized (the new text as shown in the figure), and the new text is preprocessed in the preprocessing logic 500-2 in the logic center 500. For example, the preprocessing operations include: word segmentation, removal of stop words, conversion of traditional Chinese to simplified Chinese, removal of special characters, etc., to obtain a space-separated word list. Then, the word list is input into the text vector generation module 502, and the above word list is converted into a target text vector through the model after distillation processing in the text vector generation module 502. Then, the vector search module 504 searches for the target text clustering cluster that matches the target text vector in the stored text clustering cluster set, that is, determines whether the target text vector hits the clustering cluster in the text clustering cluster set.

[0065] In the case of a hit, that is, the target text cluster is found, the target text cluster can be determined through the index list, and the text type label configured therein can be obtained to determine the type of the target text. Then, through the processing logic 500-6 in the logic center 500, the target text is processed accordingly. If it is determined to be a malicious text, the source account or source IP of the target text is blocked or warned. If it is determined to be a non-malicious text, the target text is allowed to interact normally.

[0066] If it is determined that there is no hit, that is, the target text cluster is not found, the real-time clustering module 506 can be used to consider real-time clustering of the target text. If the target text meets the clustering condition, it is clustered, and the new object text cluster is added to the text cluster set, and the information recorded in the index list 508 is updated. Then the label assignment module 510 assigns a text type label to the new object text cluster to identify the object text cluster.

[0067] In the process of clustering the target text, it may include but not be limited to first determining the frequency of occurrence of the target text in the device; when the frequency of occurrence is greater than a threshold, clustering it into a text cluster, and reporting it to the data analysis library 512 of the offline part through the reporting logic 500-4, so that the data analysis library 512 aggregates the new text clusters reported by each device. When the aggregation result indicates that the frequency of the new text cluster is greater than the threshold, it is determined to be a new object text cluster to be added to the text cluster set.

[0068] In the above process, the offline monitoring control module 514 will implement real-time monitoring to track the magnitude change of malicious texts and draw a trend curve showing the magnitude change. Furthermore, after detecting the trend curve based on the anomaly detection algorithm, automatic warning information can be issued in time before the malicious text breaks out to avoid unnecessary losses to users.

[0069] Through the embodiments provided by the present application, after obtaining the target text vector corresponding to the target text to be identified, a target text cluster corresponding to the target text vector will be searched in the set of text clusters that are regularly updated and configured with text type labels, thereby directly using the text clusters of the determined text type to identify the text type of the target text, without the need to perform offline identification for each text separately, thereby achieving the purpose of improving text recognition efficiency, and thus overcoming the problem of low text recognition efficiency caused by the inability to timely detect junk text due to offline processing in related technologies. In addition, when malicious text is quickly and efficiently identified, it will also be beneficial to purify the environment of network platforms that display or share content based on user-generated content, avoid a large amount of illegal text information from causing information interference to the user's browsing or viewing process, provide a sunny network environment for the majority of users, and ensure the browsing security of users.

[0070] As an optional solution, searching for a target text cluster that matches the target text vector in a stored text cluster set includes:

[0071] S1, searching in a clustering graph constructed based on a set of text clustering clusters, wherein in the clustering graph, each text vector is set as an element, and connecting lines are set between elements corresponding to text vectors in the same object text clustering cluster;

[0072] S2, when the target cluster center vector is found in the clustering graph, the object text cluster cluster indicated by the target cluster center vector is determined as the target text cluster cluster, wherein the distance between the target cluster center vector and the target text vector is greater than the target distance threshold.

[0073] Optionally, in this embodiment, the search for target text clusters can be, but is not limited to, a vector retrieval technology based on HNSW. Here, HNSW is a graph-based vector retrieval method provided by related technologies. Specifically, it forms a graph structure by building edges for similar vectors in a vector set, thereby increasing the query speed. This method uses mechanisms such as highways, discarded lists, and dynamic lists to ensure efficient query speed. On this basis, a hierarchical graph structure is also used to further accelerate the retrieval speed. A schematic diagram of a vector similarity retrieval algorithm based on HNSW can be shown as follows: Figure 6 As shown. Figure 6In (a), each hollow circle represents a vector, and the lines between the hollow circles represent the similarity relationship between the vectors. Based on this graph structure, you can start the query from the entry point. Figure 6 As shown in (b), the graph results here can also be multi-level, which is more conducive to improving the efficiency of search.

[0074] In the embodiment of the present application, each object text cluster in the text cluster set will construct a corresponding graph structure, so as to determine the target text that matches the target text vector.

[0075] Specifically, the following example can be used: Assume that an online similarity vector search service is built based on HNSW, and a simsvr service is established to reduce the memory overhead, search request volume, and index update synchronization time of the above service. Based on the text vectors in each object text cluster in the text cluster set, a clustering graph structure for querying similar texts is constructed, such as using text vectors as elements to set connections between similar texts in the same object text cluster.

[0076] In the case of obtaining the target text vector of the target text, the cluster cluster that is most similar to the target text vector can be searched in the above cluster graph structure. That is, traverse each object text cluster cluster, and compare the cosine similarity between the cluster center vector of each object text cluster cluster and the target text vector in turn. In the case of finding a target cluster center vector whose cosine similarity exceeds the target threshold, it is determined that the target text vector hits the existing text cluster cluster set, and the object text cluster cluster corresponding to the above target cluster center vector is determined as the target text cluster cluster, and the cluster ID of the target text cluster cluster is obtained. In the case of not finding a target cluster center vector whose cosine similarity exceeds the target threshold, it is determined that the target text vector does not hit the existing text cluster cluster set, and then continue to enter the subsequent processing operation.

[0077] Through the embodiments provided in the present application, based on the graph structure characteristics of the vector graph, the speed of searching for the target text clustering cluster matching the target text vector in the clustering graph constructed by combining the text clustering clusters will be accelerated, thereby improving the search efficiency and further improving the efficiency of recognizing the target text.

[0078] As an optional solution, searching for a target text vector in a clustering graph constructed based on a text clustering cluster set includes:

[0079] S1, according to the index relationship recorded in the index list corresponding to the text clustering cluster set, determine the candidate text clustering clusters where each word in the target word list is located from the text clustering cluster set, wherein the index relationship is the mapping relationship between each word in the text clustering cluster set and the target text clustering cluster;

[0080] S2, search in the candidate clustering graph corresponding to the candidate text clustering cluster.

[0081] Optionally, in this embodiment, the above-mentioned index list may include but is not limited to a two-level index list, wherein the two-level index list includes: a first-level index list (also referred to as a forward index list) and a second-level index list (also referred to as a reverse index list). Here, the first-level index list is used to record the index relationship between the cluster index identifier corresponding to each object text cluster in the text cluster cluster set and the position pointer of each word in the text vector contained therein in the second-level index list. That is, the mapping relationship between each object text cluster and one or more words contained therein. Here, the second-level index list is used to record the index relationship between each word in the text cluster cluster set and the cluster identifier corresponding to the object text cluster cluster to which it belongs. That is, the mapping relationship between each word and the object text cluster to which it belongs.

[0082] It should be noted that the above cluster index identifier may be a cluster identifier (ie, cluster ID) allocated to the object text cluster for global distinction, or may be an identifier set solely for building an index list, which is not limited in this embodiment.

[0083] Optionally, in order to reduce the search volume and improve the search efficiency, in this embodiment, the search range may be pre-screened to determine a candidate text cluster with a smaller range from the text cluster set.

[0084] Among them, when the target text vector is obtained, the object text clustering clusters where each word involved in the target word list corresponding to the above target text vector is located can be found based on the index relationship in the index list corresponding to the text clustering cluster set, such as the secondary index list (also called the inverted index list). For example, assuming that the text clustering cluster set includes N object text clustering clusters, after analyzing the target word list of the target text vector, it is found that the sentences of the target text include the words: W1, W2, W3, and W4. Then, it can be directly found in the inverted index list that the object text clustering cluster where word W1 is located is cluster B1, the object text clustering cluster where word W2 is located is cluster B2, the object text clustering cluster where word W3 is located is also cluster B2, and the object text clustering cluster where word W4 is located is cluster B3. Then, the candidate set {B1, B2, B3} can be determined from the overall text clustering cluster set, and it can be determined as a candidate text clustering cluster, and searched in the candidate clustering graph corresponding to the candidate text clustering cluster.

[0085] Through the embodiments provided in the present application, by determining candidate text clusters, the search scope is narrowed, thereby facilitating the reduction of the search amount for searching the target text vector, thereby achieving the effect of improving the search efficiency.

[0086] As an optional solution, after searching for a target text cluster matching the target text vector in the stored text cluster set, the method further includes:

[0087] S1, when the target text cluster is not found in the text cluster set, a new object text cluster is created using the target text vector;

[0088] S2, update the text clustering cluster set according to the new object text clustering cluster.

[0089] It should be noted that, in this embodiment, when no text clustering cluster is found in the existing text clustering cluster set, real-time clustering processing will be adopted for the target text to be identified. That is to say, the target text is judged in real time, and when the judgment result indicates that the target text meets the clustering condition, the target text is clustered into a new object text cluster in real time. Thereby, the junk text that has not been clustered can be found online in time, and the junk text triggered by maliciousness can be integrated in time and stored in the text clustering cluster set, so as to directly use the text clustering cluster set to identify malicious text in the future, thereby improving the recognition efficiency of text.

[0090] As an optional solution, updating the text clustering cluster set according to the new object text clustering cluster includes:

[0091] 1) When the number of currently stored object text clustering clusters in the text clustering cluster set has not yet reached the target value, directly add the new object text clustering cluster to the text clustering cluster set;

[0092] 2) When the number of object text clustering clusters currently stored in the text clustering cluster set reaches the target value, at least one object text clustering cluster is removed from the text clustering cluster set, and a new object text clustering cluster is added to the text clustering cluster set.

[0093] It should be noted that in this embodiment, an upper limit value (ie, target value) is configured for the text cluster set to avoid unlimited increase in the number of clusters, thereby reducing the workload required for traversal, search and comparison, thereby achieving the purpose of improving text recognition efficiency.

[0094] Specific combination Figure 7 The process shown is explained as follows: Assuming that the target text clustering cluster that matches the target text vector is not found in the text clustering cluster set, you can use Figure 5 The real-time clustering module 506 is shown to determine whether to create a new object text cluster based on the target text vector.

[0095] As in step S702, a text vector q corresponding to a new text is obtained (i.e., the target text vector shown in the figure), and as in step S704, a candidate set consisting of M object text clusters in which the M phrases appearing in the text vector q are respectively located is obtained according to the inverted index list. Then, it is determined whether the above text vector q is similar to any text cluster in the candidate set (such as taking the cosine distance between the text vector q and the cluster center vector of the text cluster as the similarity between the two). In the case where it is determined that the text vector q is similar to a text cluster (such as the similarity is greater than a specified threshold), then as in step S706-1, the above text vector q is attributed to the text cluster (clustered into an existing text cluster).

[0096] In the case where it is determined that the text vector q is not similar to a text cluster (such as the similarity is less than or equal to the specified threshold), the number of current text clusters is obtained as in step S706-2, and it is determined whether the number reaches the preset upper limit. In the case where it is determined that the upper limit is not reached, it is determined whether a new object text cluster is created based on the text vector q as in step S708-1, and the new object text cluster is added to the existing text cluster set as in step S710-1; in the case where it is determined that the upper limit is reached, it is necessary to determine the old class to be eliminated from the existing text clusters as in steps S708-2 and S710-2, and eliminate it, so as to create a new object text cluster based on the above text vector q, thereby achieving the purpose of adding a new text cluster. It should be noted that, in the case where the old class to be eliminated cannot be determined, the current target text vector can also be discarded to complete the current recognition process.

[0097] In the process of adding new text clusters or removing old text clusters, step S712 will be executed to update the corresponding index list, so that in the next query process, the updated index list can be directly used to assist in completing the search and identification process of the text vector in the text cluster set.

[0098] Through the embodiments provided in the present application, an upper limit value (i.e., a target value) is set for the number of object text clustering clusters stored in the text clustering cluster set, thereby avoiding unlimited storage of too many text clustering clusters in the text clustering cluster set, thereby reducing the workload required for comparison during the search traversal process, and improving the speed of completing traversal searches in existing text clustering set clusters, thereby further achieving the effect of improving text recognition efficiency.

[0099] As an optional solution, updating the text clustering cluster set according to the new object text clustering cluster includes:

[0100] When a new object text clustering cluster is added to the text clustering cluster set, the two-level index list corresponding to the text clustering cluster set is updated, wherein the two-level index list includes a primary index list and a secondary index list, wherein the primary index list records the cluster index identifier corresponding to each object text clustering cluster in the text clustering cluster set, and the index relationship between the position pointer of each word contained in the object text clustering cluster indicated by the cluster index identifier and the position in the secondary index list, and the secondary index list records the index relationship between each word appearing in the text clustering cluster set and the cluster identifier corresponding to the object text clustering cluster where the word is located.

[0101] It should be noted that the index list in the above text clustering set may include but is not limited to a magnitude index list, which not only ensures query efficiency but also supports functions such as fast addition and deletion.

[0102] For example, Figure 8 The secondary index list shown in the table records the index relationship between each word and the cluster identifier of the text cluster to which it belongs. The secondary index list here can be an inverted index or a forward index. The inverted index (inverted index table, also called Inverted Index) here is a data structure called a hash map, which consists of two parts: a key and a value. The value is represented by a doubly linked list. Specifically for this application, its structural diagram is as follows: Figure 8 As shown in the upper part, the index cluster identifiers corresponding to word A include: cluster 1, cluster 2, and cluster 5; the index cluster identifiers corresponding to word B include: cluster 2 and cluster 3.

[0103] Further, if Figure 8 The first-order index list (forward index list) shown in the table records the index relationship of each object text cluster and the position pointer of each word contained in the second-level index list. The forward index is also a hash table structure, and the value is in the form of a one-way linked list. The key here is the cluster ID, and the value list stores pointers (Pointer, abbreviated as Ptr) to the nodes in the value list of the inverted index.

[0104] It should be noted that the linked list in the above index list has the following characteristics: the modification, addition, and deletion of nodes are very fast, but to find a certain node in it, it is necessary to traverse the entire linked list, that is, the search is very slow. Compared with ordinary one-way linked lists, bidirectional linked lists support traversal in two directions. In addition, for the inverted index structure: it is possible to quickly query which clusters a certain word appears in, but if you want to delete or modify a certain cluster from the inverted index, the efficiency is very low, and it is necessary to traverse all the linked list nodes corresponding to all the words in the cluster. For the forward index structure: it is possible to quickly query the position of a certain cluster in the inverted index, and then it is very convenient to modify the clustering information in the inverted index, which makes up for the defects of the inverted index.

[0105] As an optional implementation, updating the two-level index list corresponding to the text cluster set includes:

[0106] 1) In the case of removing at least one object text cluster, find the position pointer of each word corresponding to at least one object text cluster in the secondary index list in the primary index list; remove the object text cluster where each word indicated by the position pointer is located;

[0107] 2) When at least one object text cluster is added, an index relationship between each word appearing in at least one object text cluster and the cluster identifier corresponding to the object text cluster where the word is located is added in the secondary index list; an index relationship between a cluster index identifier of at least one object text cluster is created in the primary index list and a position pointer of each word contained in the object text cluster indicated by the cluster index identifier at the position in the secondary index list.

[0108] For example, in the case of removing an old object text cluster, first determine the cluster index identifier (such as cluster index identifier K-2) of the old object text cluster to be removed, then find the words contained in the text cluster identified by cluster index identifier K-2 based on the primary index list, such as W1 and W9, and their corresponding position pointers are P1 and P9. Further, find the cluster identifier ID-1 corresponding to P1 and the cluster identifier ID-2 corresponding to P2 based on the secondary index list, then perform the removal process on the text cluster identified by cluster identifier ID-1 and the text cluster identified by cluster identifier ID-2, and simultaneously delete the related index relationship recorded in the index list.

[0109] For example, when adding a new object text cluster, first determine the multiple words contained in the new object text cluster, and create an index relationship for each word in the secondary index list. Then, based on the position numbers of the positions of these words in the secondary index list, create an index relationship between the position pointer and the cluster index identifier in the primary index list.

[0110] It should be noted that the above-mentioned cluster index identifier (Key, abbreviated as K) can be a cluster identifier (i.e., cluster ID) assigned to the object text clustering cluster for global distinction, or it can be a global identifier assigned solely for building an index list. This embodiment does not impose any limitation on this.

[0111] Through the embodiments provided in the present application, object text clustering clusters are managed based on the above two-level index lists, such as query, add, delete and other operations, which will greatly improve management efficiency.

[0112] As an optional solution, before adding a new object text cluster to the text cluster set, include:

[0113] S1, when the occurrence frequency of the new object text cluster is greater than a first threshold, reporting the new object text cluster to the server, so that the server aggregates the new object text cluster reported by each clustering device;

[0114] S2: when the aggregation result of the server indicates that the cluster size of the new object text cluster is greater than a second threshold, assign a cluster identifier to the new object text cluster.

[0115] Optionally, in this embodiment, when creating a new object text cluster based on the target text vector, certain clustering conditions need to be met, wherein the clustering conditions here may include but are not limited to two clustering determinations reaching corresponding threshold conditions. It should be noted that in the clustering process, the actual online traffic is distributed among multiple clustering devices. In order to achieve unified clustering numbering and statistics, in this embodiment, the clustering results of a single device reported by each clustering device can be summarized, but not limited to, and the latest clustering results are updated to the vector search service.

[0116] For example, Fig. 9 As shown, in the process of multiple devices (such as three clustering devices) performing text recognition in parallel, clustering processing can be performed in parallel in each device as in step S902, such as obtaining text vectors in each clustering device, and clustering and reporting text vectors whose frequency of occurrence in the current clustering device is greater than a first threshold (also referred to as a low threshold as shown in the figure) to the aggregation device. For example, each clustering device executes steps S904-1, S904-2, and S904-3 (i.e., low threshold trigger reporting) respectively.

[0117] The aggregation device executes step S906 to aggregate the results of the above-mentioned reported content (i.e., secondary clustering). After the size of the cluster of the current secondary clustering reaches the second threshold (also referred to as the high threshold as shown in the figure), as in step S908, the result of the above-mentioned secondary clustering is reported to the offline database 902 (i.e., the high threshold triggers reporting). The database 902 includes a clustering report 9022 and a global clustering table 9024, where the clustering report 9022 is used to implement clustering control, such as step S910, to fuse the clustering results to obtain a cluster. The global clustering table 9024 is used to assign a new global ID to the new object text clustering cluster, such as steps S912-S914, requesting to obtain the existing clustering cluster for comparison, and adding a clustering cluster to the global clustering table 9024. Then, it is synchronized and updated to the online vector search service 900 at a regular time, such as step S916, and updated regularly, so that the vector search service 900 can quickly search for the latest target text clustering cluster that matches the target text vector.

[0118] Through the embodiments provided in the present application, text vectors that meet the clustering conditions are clustered in real time, and secondary clustering is used to filter and manage the cluster clusters reported by different devices, thereby avoiding the allocation of global identifiers to clusters with low frequency of occurrence or small cluster size, or occupying global resources, thereby reducing unnecessary waste of resources.

[0119] As an optional solution, obtaining a target word list corresponding to a target text to be recognized includes: performing a preprocessing operation on the target text to obtain a target word list, wherein the preprocessing operation includes: a word segmentation operation, a redundant character removal operation, and a format conversion operation; using the target word list to generate a target text vector corresponding to the target text includes: using a distilled language vector conversion model to convert the target text into a target text vector, wherein the number of network layers in the distilled language vector conversion model is less than the number of network layers in the original language vector conversion model, and the language vector conversion model is a coding representation model for language processing.

[0120] Optionally, in this embodiment, the text-to-text vector conversion process can be implemented by using, but not limited to, TinyBert obtained by distilling the Bert-based language processing model. The model structure of TinyBert here is the same as that of Bert, except that the number of network layers is smaller and the parameters are fewer. Fig.10 The following is a model structure of Bert, where Fig.10 (a) is the conversion structure. Fig.10 (b) is the Bert structure.

[0121] After the language processing model is trained using sample data, text vectors can be generated by the following method: preprocessing the text, such as word segmentation, stop word removal, traditional Chinese to simplified Chinese, special character removal, etc., to obtain a space-delimited word list; convert the above word list into a digital ID form, and then input it into the above online model; the model will output the text vector corresponding to the above text.

[0122] Through the embodiments provided in this application, a Bert-based language processing model and model distillation technology are used to generate text vectors, thereby achieving the goal of using a model with fewer network layers to accelerate the text conversion process, thereby facilitating rapid identification of the text type based on the converted text vectors, thereby achieving the effect of improving text recognition efficiency.

[0123] As an optional solution, after determining the target text as an abnormal text, the following is further included:

[0124] S1, regularly pull multiple object text clusters that are popular in the target time period from the text cluster set;

[0125] S2, configure text type labels for multiple object text clusters.

[0126] It should be noted that after the text clustering cluster set is updated, the object text clustering clusters therein can be regularly assigned text type labels (i.e., labeled). For example, the top N popular object text clustering clusters in the current time period (1 hour) are pulled regularly (such as every hour), imported into the labeling system, and labeled in the system. For example, each object text clustering cluster in the top N popular object text clustering clusters is determined to be maliciously triggered junk text based on the results of actual operations. If it is junk text, the labeling result is to configure a text type label for it: malicious clustering cluster; if it is not junk text, the labeling result is to configure a text type label for it: non-malicious clustering cluster. The labeling results are then submitted and fed back to the online in a timely manner to achieve the purpose of real-time clustering and ensure real-time performance.

[0127] In addition, in this embodiment, the above-mentioned labeling operation may be, but is not limited to, configuring a text type label for any text in an object text cluster, so that the object text clusters of this type share the text type label without repeating the labeling operation.

[0128] Through the embodiments provided in the present application, by updating the online real-time clustering labeling results in real time, it will help to speed up the identification process of maliciously triggered junk text (abnormal text) and enhance the effective crackdown on maliciously triggered junk text.

[0129] As an optional solution, after determining the target text as abnormal text, it also includes: performing at least one of the following operations on the source account or source IP address that sends the abnormal text: sending an alarm message, terminating the use permission.

[0130] It should be noted that, in the present embodiment, further punishment processing may be performed on the above-mentioned identified abnormal text (i.e., maliciously triggered spam text), such as sending an alarm message to the source account or source IP address that triggers the abnormal text to prompt its abnormal status; or, for example, banning the source account or source IP address that triggers the abnormal text, i.e., disabling its usage rights to prevent it from maliciously triggering spam text again and causing information interference to users.

[0131] As an optional solution, after determining the target text as an abnormal text, the following is further included:

[0132] S1, obtaining clustering information of abnormal texts identified within a target period, wherein the clustering information at least includes magnitude change information of the abnormal texts;

[0133] S2, pushes the clustering information to each client in the form of a chart for display.

[0134] Optionally, in this embodiment, the data analysis function of a mature data analysis tool (such as Clickhouse) can be used to analyze the content of the above-identified abnormal text, and the analysis results can be displayed in real time. For example, the current text clustering situation can be displayed (displaying the current most popular text content), and the magnitude change trend curve of the suspicious cluster cluster can be drawn; further, for cluster clusters with abnormal magnitude changes, automatic warning can be performed based on the curve anomaly detection algorithm. For malicious clusters found in the monitoring system, it also supports operators to perform operations in the monitoring system.

[0135] Through the embodiments provided in the present application, while ensuring that abnormal text is discovered and identified in a timely manner, the real-time monitoring results of abnormal text will be further improved, and malicious clusters discovered in a timely manner will be warned to avoid unnecessary losses.

[0136] The following examples are used to illustrate this:

[0137] Assume that a video content sharing platform is used as an example. When user account ID-A shares a video through the platform, fan accounts ID-a to ID-c that follow user account ID-A publish comments on the video. Assume that fan account ID-a publishes a comment text a: "Great content!"; fan account ID-b publishes a comment text b: "Very creative!"; fan account ID-c publishes multiple repeated texts c: "Do you need a high loan?". Here, the content published by fan account ID-c is repeated, and the content itself is not conducive to user account ID-A receiving feedback on the video published by the fan account, nor is it conducive to other user accounts watching. In fact, it causes certain information interference to user account ID-A and other user accounts.

[0138] By using the text recognition method provided in this embodiment, the junk text in the above text a, text b and text c can be identified by comparing and searching with the stored text clustering cluster set. That is to say, text c can be accurately identified as abnormal text (i.e., junk text). When it is determined that the frequency of the operation of the fan account ID-c posting similar abnormal texts has reached the threshold, the background server can directly send an alarm message to the above fan account ID-c, or terminate its use authority, thereby achieving the purpose of protecting the browsing environment of the majority of users.

[0139] It should be noted that, for the above-mentioned method embodiments, for the sake of simplicity, they are all described as a series of action combinations, but those skilled in the art should know that the present invention is not limited by the described action sequence, because according to the present invention, certain steps can be performed in other sequences or simultaneously. Secondly, those skilled in the art should also know that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present invention.

[0140] According to another aspect of the embodiments of the present invention, a text recognition device for implementing the above-mentioned text recognition method is also provided. Fig.11 As shown, the device comprises:

[0141] 1) An acquisition unit 1102 is used to acquire a target word list corresponding to a target text to be recognized;

[0142] 2) a generating unit 1104, configured to generate a target text vector corresponding to the target text using the target word list;

[0143] 3) a search unit 1106, configured to search for a target text clustering cluster matching the target text vector in the stored text clustering cluster set, wherein the text clustering cluster set will be updated regularly, and the text clustering cluster set includes an object text clustering cluster configured with a text type label, and the text type label is used to indicate that the object text clustering cluster is a malicious clustering cluster or a non-malicious clustering cluster;

[0144] 4) The identification unit 1108 is used to determine the target text as abnormal text when the target text cluster is found in the text cluster set and the text type label corresponding to the target text cluster indicates that the target text cluster is a malicious cluster.

[0145] Optionally, in this embodiment, the above-mentioned text recognition device can be applied to, but not limited to, application scenarios for content display or promotion based on user-generated content (UGC), wherein UGC refers to displaying or providing one's own original content to other users through an Internet platform. For example, friend social networks, video / photo / knowledge sharing networks, communities / forums and other scenarios. Optionally, the text information in the above-mentioned application scenarios corresponding to the text recognition method provided in this embodiment applied to the above-mentioned application scenarios may include but is not limited to at least one of the following: forwarding text information in the external link forwarding scenario, shared text information in the community shared space, comment text information on the displayed content, and greeting text information used when binding account relationships (such as registration, adding friend accounts, etc.). The recognition process here is mainly to identify junk texts that are published by users through network platforms in different above-mentioned application scenarios, with high frequency and repeated content, such as text information sent by users in the form of "drift bottles", which may be repeated content, or even involve illegal sensitive information (such as pornographic gambling and drug information). These spam texts are usually malicious texts triggered by malicious entities without achieving a certain purpose. They cause great information interference to users who are browsing web pages or watching videos and other content they are interested in, making the Internet browsing environment of the majority of users unsafe.

[0146] For the specific implementation example, reference may be made to the above method implementation example, which will not be described in detail here.

[0147] According to another aspect of the embodiments of the present invention, an electronic device for implementing the above-mentioned text recognition method is also provided. The electronic device may be Figure 1 The server shown. Fig.12 As shown, the electronic device includes a memory 1202 and a processor 1204. The memory 1202 stores a computer program, and the processor 1204 is configured to execute the steps in any of the above method embodiments through the computer program.

[0148] Optionally, in this embodiment, the electronic device may be located in at least one network device among a plurality of network devices of a computer network.

[0149] Optionally, in this embodiment, the processor may be configured to perform the following steps through a computer program:

[0150] S1, obtain the target word list corresponding to the target text to be recognized;

[0151] S2, using the target word list to generate the target text vector corresponding to the target text;

[0152] S3, searching for a target text clustering cluster matching the target text vector in the stored text clustering cluster set, wherein the text clustering cluster set will be updated regularly, and the text clustering cluster set includes an object text clustering cluster configured with a text type label, and the text type label is used to indicate that the object text clustering cluster is a malicious clustering cluster or a non-malicious clustering cluster;

[0153] S4, when a target text cluster is found in the text cluster set and the text type label corresponding to the target text cluster indicates that the target text cluster is a malicious cluster, the target text is determined as an abnormal text.

[0154] Alternatively, a person skilled in the art may understand that: Fig.12 The structure shown is for illustration only, and the electronic device may also be a smart phone (such as an Android phone, an iOS phone, etc.), a tablet computer, a PDA, a mobile Internet device (MID), a PAD, and other terminal devices. Fig.12 The electronic device and the electronic equipment described above are not limited in structure. Fig.12 More or fewer components (such as network interfaces, etc.) as shown in, or with Fig.12 Different configurations are shown.

[0155] Among them, the memory 1202 can be used to store software programs and modules, such as the program instructions / modules corresponding to the text recognition method and device in the embodiment of the present invention. The processor 1204 executes various functional applications and data processing by running the software programs and modules stored in the memory 1202, that is, to implement the above-mentioned text recognition method. The memory 1202 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 1202 may further include a memory remotely arranged relative to the processor 1204, and these remote memories may be connected to the terminal via a network. Examples of the above-mentioned networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof. Among them, the memory 1202 can be specifically, but not limited to, used to store information such as target text to be recognized and recognition results. As an example, such as Fig.12 As shown, the memory 1202 may include but is not limited to the extracting unit 1102, determining unit 1104, generating unit 1106 and processing unit 1108 in the text recognition device. In addition, it may also include but is not limited to other module units in the text recognition device, which will not be repeated in this example.

[0156] Optionally, the transmission device 1206 is used to receive or send data via a network. Specific examples of the network may include a wired network and a wireless network. In one example, the transmission device 1206 includes a network adapter (Network Interface Controller, NIC), which can be connected to other network devices and routers via a network cable so as to communicate with the Internet or a local area network. In one example, the transmission device 1206 is a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0157] In addition, the electronic device further includes: a display 1208 for displaying the target text to be recognized and the recognition result; and a connection bus 1210 for connecting various module components in the electronic device.

[0158] In other embodiments, the terminal device or server may be a node in a distributed system, wherein the distributed system may be a blockchain system, and the blockchain system may be a distributed system formed by connecting the multiple nodes through network communication. Among them, the nodes may form a peer-to-peer (P2P, Peer To Peer) network, and any form of computing device, such as a server, terminal and other electronic devices, may become a node in the blockchain system by joining the peer-to-peer network.

[0159] According to one aspect of the present application, a computer program product or computer program is provided, the computer program product or computer program comprising computer instructions, the computer instructions being stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device performs the above-mentioned text recognition method. The computer program is configured to perform the steps of any one of the above-mentioned method embodiments when it is run.

[0160] Optionally, in this embodiment, the computer-readable storage medium may be configured to store a computer program for performing the following steps:

[0161] S1, obtain the target word list corresponding to the target text to be recognized;

[0162] S2, using the target word list to generate the target text vector corresponding to the target text;

[0163] S3, searching for a target text clustering cluster matching the target text vector in the stored text clustering cluster set, wherein the text clustering cluster set will be updated regularly, and the text clustering cluster set includes an object text clustering cluster configured with a text type label, and the text type label is used to indicate that the object text clustering cluster is a malicious clustering cluster or a non-malicious clustering cluster;

[0164] S4, when a target text cluster is found in the text cluster set and the text type label corresponding to the target text cluster indicates that the target text cluster is a malicious cluster, the target text is determined as an abnormal text.

[0165] Optionally, in this embodiment, a person of ordinary skill in the art may understand that all or part of the steps in the various methods of the above embodiments may be completed by instructing hardware related to the terminal device through a program, and the program may be stored in a computer-readable storage medium, and the storage medium may include: a flash drive, a read-only memory (ROM), a random access memory (RAM), a disk or an optical disk, etc.

[0166] The serial numbers of the above embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.

[0167] If the integrated units in the above embodiments are implemented in the form of software functional units and sold or used as independent products, they can be stored in the above computer-readable storage medium. Based on such an understanding, the technical solution of the present invention, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for enabling one or more computer devices (which can be personal computers, servers or network devices, etc.) to perform all or part of the steps of the methods described in various embodiments of the present invention.

[0168] In the above embodiments of the present invention, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0169] In the several embodiments provided in the present application, it should be understood that the disclosed client can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0170] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0171] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0172] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A text recognition method, characterized in that: include: Obtain a target word list corresponding to the target text to be recognized; Generating a target text vector corresponding to the target text using the target word list; Searching for a target text clustering cluster matching the target text vector in a stored text clustering cluster set, wherein the text clustering cluster set will be updated regularly, and the text clustering cluster set includes an object text clustering cluster configured with a text type label, and the text type label is used to indicate that the object text clustering cluster is a malicious clustering cluster or a non-malicious clustering cluster; In the case where the target text cluster cluster is found in the text cluster cluster set, the target text cluster cluster is determined through an index list, and the text type label corresponding to the target text cluster cluster is obtained, wherein the index list includes a two-level index list corresponding to the text cluster cluster set, and the two-level index list includes a primary index list and a secondary index list, wherein the primary index list records the cluster index identifier corresponding to each object text cluster cluster in the text cluster cluster set, and the index relationship between the position pointer of each word contained in the object text cluster cluster indicated by the cluster index identifier at the location of the secondary index list, and the secondary index list records the index relationship between each word appearing in the text cluster cluster set and the cluster identifier corresponding to the object text cluster cluster where the word is located; When the target text cluster is found in the text cluster set and the text type label corresponding to the target text cluster indicates that the target text cluster is a malicious cluster, the target text is determined as an abnormal text.

2. The method according to claim 1, characterized in that: The step of searching for a target text cluster that matches the target text vector in the stored text cluster set includes: Searching is performed in a clustering graph constructed based on the text clustering cluster set, wherein in the clustering graph, each text vector is set as an element, and connecting lines are set between elements corresponding to text vectors in the same object text clustering cluster; When a target cluster center vector is found in the clustering graph, the object text cluster indicated by the target cluster center vector is determined as the target text cluster, wherein the distance between the target cluster center vector and the target text vector is greater than a target distance threshold.

3. The method according to claim 2, characterized in that Searching for the target text vector in a clustering graph constructed based on the text clustering cluster set includes: According to the index relationship recorded in the two-level index list corresponding to the text clustering cluster set, the candidate text clustering clusters where each word in the target word list is located are determined from the text clustering cluster set, wherein the index relationship is a mapping relationship between each word in the text clustering cluster set and the target text clustering cluster; Search is performed in the candidate clustering graph corresponding to the candidate text clustering cluster.

4. The method according to claim 1, characterized in that: After searching the stored text clustering cluster set for a target text clustering cluster matching the target text vector, the method further includes: In the case where the target text cluster is not found in the text cluster set, creating a new object text cluster using the target text vector; The text clustering cluster set is updated according to the new object text clustering cluster.

5. The method according to claim 4, characterized in that The updating of the text clustering cluster set according to the new object text clustering cluster comprises: When the number of the object text clustering clusters currently stored in the text clustering cluster set has not yet reached the target value, directly adding the new object text clustering cluster to the text clustering cluster set; When the number of the object text clustering clusters currently stored in the text clustering cluster set reaches the target value, at least one object text clustering cluster is removed from the text clustering cluster set, and then the new object text clustering cluster is added to the text clustering cluster set.

6. The method according to claim 5, characterized in that The updating of the text clustering cluster set according to the new object text clustering cluster comprises: When the new object text cluster is added to the text cluster set, the two-level index list corresponding to the text cluster set is updated.

7. The method according to claim 6, characterized in that The updating of the two-level index list corresponding to the text clustering set includes: In the case of removing at least one object text cluster, finding the position pointer of each word corresponding to the at least one object text cluster in the first-level index list at the position of the second-level index list; removing the object text cluster where each word indicated by the position pointer is located; In the case of adding at least one object text clustering cluster, an index relationship between each word appearing in the at least one object text clustering cluster and a cluster identifier corresponding to the object text clustering cluster where the word is located is added to the secondary index list; an index relationship between a cluster index identifier of the at least one object text clustering cluster and a position pointer of each word contained in the object text clustering cluster indicated by the cluster index identifier at the location of the secondary index list is created in the primary index list.

8. The method according to claim 5, characterized in that Before adding the new object text cluster to the text cluster set, the method includes: When the occurrence frequency of the new object text cluster is greater than a first threshold, reporting the new object text cluster to the server, so that the server aggregates the new object text cluster reported by each clustering device; When the aggregation result of the server indicates that the cluster size of the new object text cluster is greater than a second threshold, a cluster identifier is allocated to the new object text cluster.

9. The method according to any one of claims 1 to 8, characterized in that The step of obtaining a target word list corresponding to the target text to be recognized includes: performing a preprocessing operation on the target text to obtain the target word list, wherein the preprocessing operation includes: a word segmentation operation, a redundant character removal operation, and a format conversion operation; The method of generating a target text vector corresponding to the target text using the target word list includes: using a distilled language vector conversion model to convert the target text into the target text vector, wherein the number of network layers in the distilled language vector conversion model is less than the number of network layers in the original language vector conversion model, and the language vector conversion model is a coding representation model for language processing.

10. The method according to any one of claims 1 to 8, characterized in that After determining the target text as an abnormal text, the method further includes: Periodically extracting multiple object text clusters that are in a hot state within a target time period from the text cluster set; The text type labels are configured for the plurality of object text clusters.

11. The method according to any one of claims 1 to 8, characterized in that After determining the target text as an abnormal text, the method further includes: Perform at least one of the following operations on the source account or source IP address that sends the abnormal text: send an alarm message, terminate the use permission.

12. The method according to any one of claims 1 to 8, characterized in that After determining the target text as an abnormal text, the method further includes: Acquire clustering information of the abnormal text identified within a target period, wherein the clustering information at least includes magnitude change information of the abnormal text; The clustering information is pushed to each client in the form of a chart for display.

13. A text recognition device, characterized in that: include: A first acquisition unit, used to acquire a target word list corresponding to a target text to be recognized; A generating unit, configured to generate a target text vector corresponding to the target text using the target word list; A search unit, used for searching a target text clustering cluster matching the target text vector in a stored text clustering cluster set, wherein the text clustering cluster set will be updated regularly, and the text clustering cluster set includes an object text clustering cluster configured with a text type label, and the text type label is used to indicate that the object text clustering cluster is a malicious clustering cluster or a non-malicious clustering cluster; A second acquisition unit is used to determine the target text cluster cluster through an index list and acquire the text type label corresponding to the target text cluster cluster when the target text cluster cluster is found in the text cluster cluster set, wherein the index list includes a two-level index list corresponding to the text cluster cluster set, and the two-level index list includes a primary index list and a secondary index list, wherein the primary index list records the cluster index identifier corresponding to each object text cluster cluster in the text cluster cluster set and the index relationship between the position pointer of each word contained in the object text cluster cluster indicated by the cluster index identifier at the location of the secondary index list, and the secondary index list records the index relationship between each word appearing in the text cluster cluster set and the cluster identifier corresponding to the object text cluster cluster where the word is located; The identification unit is used to determine the target text as abnormal text when the target text cluster is found in the text cluster set and the text type label corresponding to the target text cluster indicates that the target text cluster is a malicious cluster.

14. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored program, wherein the program executes the method described in any one of claims 1 to 12 when executed.

15. An electronic device comprising a memory and a processor, characterized in that: A computer program is stored in the memory, and the processor is configured to execute the method according to any one of claims 1 to 12 through the computer program.

Images