Text similarity detection method, device, electronic device and storage medium

Abstract

The embodiment of the present application provides a method, device, electronic device and computer-readable storage medium for detecting text similarity, which relates to the field of natural text type processing technology. The method includes: obtaining at least two texts to be detected; segmenting the texts, and obtaining a target sequence of the texts based on the segmentation results; merging each target element with a subsequent preset number of target elements in turn to obtain at least one merged element corresponding to the target element, and obtaining a merged element sequence of the text based on all target elements and at least one merged element; obtaining a frequency vector for each text; for any two texts, obtaining the text similarity of any two texts based on the frequency vectors of any two texts. The embodiment of the present application is more suitable for the scenario of online name review, and can effectively mine the behavior of malicious batch registration of nicknames and batch group gatherings, and issue risk warnings.

Description

Technical Field

[0001] The present application relates to the technical field of natural text type processing. Specifically, the present application relates to a text similarity detection method, device, electronic device, computer-readable storage medium and computer program product. Background Art

[0002] In the field of natural text processing, it is often necessary to calculate the similarity of different texts. Related technologies often use cosine similarity to calculate the similarity between two texts. The process can be summarized as follows: 1. Word segmentation; 2. List all words; 3. Word segmentation encoding; 4. Word frequency vectorization; 5. Apply the cosine function to calculate the similarity between two sentences.

[0003] Related technologies generally do not consider sentence order when segmenting sentences. For example, the similarity between "you love me" and "I love you" calculated by related methods is 1, but the two sentences express different meanings. When these technologies are applied to registered name screening scenarios, they often fail to identify malicious registrations and batch registrations. Summary of the Invention

[0004] The present application provides a method, apparatus, electronic device, computer-readable storage medium, and computer program product for detecting text similarity, which can solve the above-mentioned problems of the prior art. The technical solution is as follows:

[0005] According to one aspect of an embodiment of the present application, a method for detecting text similarity is provided, the method comprising:

[0006] Get at least two texts to be detected;

[0007] For each text, the text is segmented and a target sequence of the text is obtained based on the segmentation results; the target element in the target sequence is used to represent the segmentation at the corresponding position in the segmentation result or at least one item of the attribute information of the segmentation;

[0008] For each target sequence of the text, merging each target element with a subsequent preset number of target elements in sequence according to the order of the target elements in the target sequence to obtain at least one merged element corresponding to the target element, and obtaining a merged element sequence of the text based on all the target elements and the at least one merged element;

[0009] Encode the merged element sequence of each text, and obtain a frequency vector for each text based on the encoding results of the merged element sequence of each text; the features of each dimension in the frequency vector are used to represent the frequency of the corresponding element in the merged element sequence of the text in the total merged sequence. The total merged sequence is obtained by sorting the target element and the merged element in the merged element sequences of all texts, and there are no repeated elements in the total merged sequence;

[0010] For any two texts, the text similarity of the two texts is obtained according to the frequency vectors of the two texts.

[0011] According to another aspect of an embodiment of the present application, a device for detecting text similarity is provided, comprising:

[0012] A text acquisition module, used to acquire at least two texts to be detected;

[0013] The target sequence acquisition module is used to segment each text and obtain a target sequence of the text based on the segmentation results; the target element in the target sequence is used to represent the segmentation at the corresponding position in the segmentation result or at least one of the attribute information of the segmentation;

[0014] a merging module configured to, for each target sequence of text, sequentially merge each target element with a subsequent preset number of target elements according to the order of the target elements in the target sequence to obtain at least one merged element corresponding to the target element, and obtain a merged element sequence of the text based on all the target elements and the at least one merged element;

[0015] The frequency vector module is used to encode the merged element sequence of each text and obtain the frequency vector of each text based on the encoding results of the merged element sequence of each text. The features of each dimension in the frequency vector are used to represent the frequency of the corresponding element in the total merged sequence in the merged element sequence of the text. The total merged sequence is obtained by sorting the target element and the merged element in the merged element sequence of all texts. There are no repeated elements in the total merged sequence.

[0016] The similarity calculation module is used to obtain the text similarity between any two texts based on the frequency vectors of the any two texts.

[0017] As an optional embodiment, the target sequence acquisition module includes:

[0018] The word segmentation sequence submodule is used to obtain the word segmentation sequence of the text according to the word segmentation results;

[0019] The target sequence submodule is used to obtain the target sequence of the text based on the word segmentation sequence of the text.

[0020] As an optional embodiment, the word segmentation sequence submodule includes:

[0021] The initial word segmentation unit is used to obtain the initial word segmentation sequence of the text according to the word segmentation results;

[0022] a continuous word segmentation judgment module, configured to determine the initial word segmentation sequence as a word segmentation sequence if there are no continuous word segments of the first target type in the initial word segmentation sequence;

[0023] If there are continuous segmentation words of the first target type in the initial segmentation sequence, the continuous segmentation words of the target type are replaced as a whole with a preset number of preset segmentation words, and the replaced initial segmentation sequence is used as the segmentation sequence

[0024] As an optional embodiment, the continuous word segmentation judgment module is specifically used to:

[0025] If there are consecutive first-type segmentations in the initial segmentation sequence, the consecutive first-type segmentations are replaced with first target segmentations, where the first target segmentations are used to represent the number of consecutive first-type segmentations.

[0026] If there are consecutive segmentations of the second specified type in the initial segmentation sequence, the consecutive segmentations of the second specified type are replaced with second target segmentations, where the second target segmentations are a combination of consecutive segmentations of the second specified type.

[0027] As an optional embodiment, the target sequence submodule includes:

[0028] A segmentation type unit, used to determine the type of each segmentation in the segmentation sequence;

[0029] The word type sequence unit is used to obtain the word type sequence of the text according to the type of each word segmentation. Each element in the word type sequence is used to represent the type of the word segmentation at the corresponding position in the word segmentation sequence.

[0030] The target sequence unit is used to obtain the target sequence of the text according to the word type sequence of the text. The attribute information of the word segmentation includes the type of the word segmentation.

[0031] As an optional embodiment, the target sequence unit includes:

[0032] a type determination unit configured to, for each element in the word type sequence, determine, if the type represented by the element belongs to the second target type, the order of the element among the type elements belonging to the second target type in the word type sequence; and, if the type represented by the element does not belong to the second target type, determine the order of the element among the elements not belonging to the second target type in the word type sequence;

[0033] The order unit is used to obtain the target sequence according to the order of each element in the word type sequence. The attribute information of the word segmentation also includes the order of the elements corresponding to the word segmentation.

[0034] As an optional embodiment, the word segmentation type unit is specifically used to:

[0035] For each word in the word segmentation sequence, if the word segmentation belongs to multiple candidate types, then for each candidate type, the word frequency of the word segmentation belonging to the candidate type in the word segmentation sequence is determined, and the type of the word segmentation is determined according to the candidate type corresponding to the highest word frequency.

[0036] As an optional embodiment, the word segmentation type unit includes:

[0037] A first case determination unit is configured to use the candidate type corresponding to the highest word frequency as the type of the segmentation word if the candidate type corresponding to the highest word frequency is unique;

[0038] The second case determination unit is used to determine, if the candidate type corresponding to the highest word frequency is not unique, the candidate type with the highest priority and the highest word frequency as the type of the word segmentation according to a predetermined type priority.

[0039] According to another aspect of an embodiment of the present application, an electronic device is provided, which includes: a memory, a processor, and a computer program stored in the memory, and the processor executes the computer program to implement the steps of the above-mentioned text similarity detection method.

[0040] According to another aspect of the embodiments of the present application, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of the above-mentioned text similarity detection method are implemented.

[0041] According to one aspect of an embodiment of the present application, a computer program product is provided, including a computer program, which implements the steps of the above-mentioned text similarity detection method when executed by a processor.

[0042] The beneficial effects of the technical solution provided by the embodiments of the present application are:

[0043] By obtaining at least two texts to be detected, each text is segmented, and a target sequence of the text is obtained according to the segmentation results. The target elements in the target sequence can be the segmentations at the corresponding positions in the segmentation results, or the attribute information of the segmentations. The target elements can be detected from different angles and multiple similarities. For the target sequence of each text, each target element is merged with a subsequent preset number of target elements according to the order of each target element in the target sequence to obtain at least one merged element corresponding to the target element. The merged element sequence of the text is obtained according to all the target elements and at least one merged element, which can represent the target elements in the text in a more specific and fine-grained manner. Sequential relationship, and at the same time, it can also identify the subtle differences between texts with reduplication, laying the foundation for more accurate analysis of the semantics of the text, encoding the merged element sequence of each text, and obtaining the frequency vector of each text according to the encoding result of the merged element sequence of each text; the features of each dimension in the frequency vector are used to represent the frequency of the corresponding element in the total merged sequence in the merged element sequence of the text. For any two texts, the text similarity of any two texts is obtained according to the frequency vectors of any two texts, which has higher accuracy and is more suitable for the network name review scenario, and can effectively mine the behavior of batch malicious registration of nicknames and batch group gathering, and issue risk warnings. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for describing the embodiments of the present application.

[0045] Figure 1 Schematic diagram of the principle of cosine similarity in related technology;

[0046] Figure 2 A schematic diagram of an implementation environment for a solution provided in one embodiment of the present application;

[0047] Figure 3 A flowchart of a method for detecting text similarity provided in an embodiment of the present application;

[0048] Figure 4 A schematic diagram of an interface for modifying a user name provided in an embodiment of the present application;

[0049] Figure 5 A schematic diagram of a process for obtaining a merged element sequence based on a target sequence provided in an embodiment of the present application;

[0050] Figure 6 A schematic diagram of a process for determining the position of an element provided in an embodiment of the present application;

[0051] Figure 7a A schematic diagram of an initial interface for setting a detection method provided in an embodiment of the present application;

[0052] Figure 7b A schematic diagram of an interface for setting a second detection mode provided in an embodiment of the present application;

[0053] Figure 7c A schematic diagram of an interface after a detection party operates a confirmation control provided in an embodiment of the present application;

[0054] Figure 8 A schematic diagram of an interface for displaying text similarity when determining two detection methods provided in an embodiment of the present application;

[0055] Figure 9 This is a structural diagram of a text similarity detection system applicable to this scenario embodiment of the present application;

[0056] Figure 10 A schematic diagram of the structure of a text similarity detection device provided in an embodiment of the present application;

[0057] Figure 11 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0058] The following describes the embodiments of the present application in conjunction with the accompanying drawings. It should be understood that the embodiments described below in conjunction with the accompanying drawings are exemplary descriptions for explaining the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions of the embodiments of the present application.

[0059] Those skilled in the art will understand that, unless otherwise stated, the singular forms "a", "an" and "the" used herein may also include plural forms. It should be further understood that the terms "including" and "comprising" used in the embodiments of the present application mean that the corresponding features can be implemented as the presented features, information, data, steps, operations, elements and / or components, but do not exclude implementation as other features, information, data, steps, operations, elements, components and / or combinations thereof supported by the present technical field. It should be understood that when we say that an element is "connected" or "coupled" to another element, the element can be directly connected or coupled to the other element, or it can refer to that the element and the other element establish a connection relationship through an intermediate element. In addition, the "connection" or "coupling" used here can include wireless connection or wireless coupling. The term "and / or" used here indicates at least one of the items defined by the term, for example, "A and / or B" can be implemented as "A", or as "B", or as "A and B".

[0060] In order to make the objectives, technical solutions and advantages of this application clearer, the implementation methods of this application will be further described in detail below with reference to the accompanying drawings.

[0061] First, several terms involved in this application are introduced and explained:

[0062] Cosine similarity measures the similarity between two vectors by plotting their coordinates in a vector space, most commonly two-dimensional space. Cosine similarity measures the similarity between two vectors by measuring the cosine of the angle between them. The cosine of an angle of 0 degrees is 1, while the cosine of any other angle is no greater than 1, and its minimum value is -1. Therefore, the cosine of the angle between two vectors determines whether they point in roughly the same direction.

[0063] See Figure 1 , which exemplarily shows the principle diagram of cosine similarity, in which the vector Cosine of the angle between two vectors The numerator is the inner product of the two vectors, and the denominator is the product of their moduli. The larger the angle between the two vectors and the farther they are from each other, the smaller the cosine similarity. The maximum distance is when the angle between the two vectors is 180°. The smaller the angle between the two vectors and the closer they are from each other, the greater the cosine similarity. The minimum distance is when the angle between the two vectors is 0°, meaning they completely overlap.

[0064] Natural language processing (NLP) is a key area of ​​research in computer science and artificial intelligence. It studies the theories and methods that enable effective communication between humans and computers using natural language. Natural language processing (NLP) integrates linguistics, computer science, and mathematics. It has applications in machine translation, public opinion monitoring, automatic summarization, opinion extraction, text classification, question answering, text semantic comparison, speech recognition, and Chinese optical character recognition (OCR).

[0065] Word segmentation refers to the process of recombining continuous word sequences into semantically independent word sequences according to certain specifications.

[0066] The process of using cosine similarity to calculate the similarity between two texts in related technologies can be summarized as follows: 1. Word segmentation; 2. List all words; 3. Word segmentation encoding; 4. Word frequency vectorization; 5. Apply the cosine function to measure the similarity between two sentences.

[0067] The following is an example:

[0068] Sentence A: I want to concentrate on writing patents;

[0069] Sentence B: I want to focus on writing patents;

[0070] 1. Participle: Phrases are not considered here, and they are divided according to single words

[0071] listA = ['I', 'want', 'focus', 'heart', 'write', 'specialize', 'profit'];

[0072] listB = ['我', '要', '专', '注', '写', '专', '利'];

[0073] 2. List all the words, and put listA and listB in a set, obtaining:

[0074] set = {'我', '要', '专', '心', '注', '写', '利'};

[0075] Convert the above set to a dictionary dict, where the key is the word in the set and the value is the position of the word in the set, starting from 0.

[0076] dict = {'我': 0, '要': 1, '专': 2, '心': 3, '注': 4, '写': 5, '利': 6}. It can be seen that the word '我' ranks 1st in the set, with an index of 0. <000系列产品的设计和制造,包括但不限于以下方面:1. 外观设计:

[0077] 3. Encoding listA and listB, converting each character to the position where it appears in the set. After conversion:

[0078] listAcode = [0, 1, 2, 3, 5, 2, 6];

[0079] listBcode = [0, 1, 2, 3, 4, 2, 6];

[0080] 4. Conduct an element frequency count on listAcode and listBcode, which is to calculate the number of times each word segment appears. The final result is as follows:

[0081] listAcodeOneHot = [1, 1, 2, 1, 0, 1, 1];

[0082] listBcodeOneHot = [1, 1, 2, 1, 1, 0, 1];

[0083] That is, list all the words, encode the word segments, and calculate the word frequencies;

[0084] 5. After obtaining the frequency vectors of the two sentences, it becomes calculating the cosine value of the angle between the two vectors. The larger the value, the higher the similarity.

[0085]

[0086] The related technologies have the following problems when calculating similarity:

[0087] 1) When segmenting sentences, the order of the sentences is not considered. For example, for the sentences "You love me" and "I love you", if calculated according to the above scheme, the similarity is 1, but the meanings expressed by the above two sentences are different;

[0088] 2) The length and position of reduplicated words are not considered. For example, for "haha" and "hahaha", the similarity is 1.

[0089] 3) The language type is not considered;

[0090] When the text similarity determination method in the related art is applied to detect registration names, due to the small number of characters in the registration names, there are often multiple language types and the presence of reduplicated words (such as a mixture of Chinese and English, a mixture of text and numbers). The related art cannot accurately identify batch malicious registrations and group rejection behaviors.

[0091] The text similarity detection method, device, electronic device, computer-readable storage medium, and computer program product provided by this application aim to solve the above technical problems in the prior art.

[0092] The technical solutions of the embodiments of this application and the technical effects produced by the technical solutions of this application will be described below through the description of several exemplary embodiments. It should be noted that the following embodiments can be referenced, borrowed, or combined with each other. For the same terms, similar features, and similar implementation steps in different embodiments, they will not be described repeatedly.

[0093] Please refer to Figure 2 , which shows a schematic diagram of the solution implementation environment provided by an embodiment of this application. This solution implementation environment can be implemented as a video processing system. This solution implementation environment can include: terminal device 10 and server 20.

[0094] The text similarity detection system realizes the detection of text similarity through terminal device 10 and server 20.

[0095] The terminal device 10 can be an electronic device such as a mobile phone, a tablet computer, a PC (Personal Computer), a wearable device, a vehicle-mounted terminal device, a VR (Virtual Reality) device, and an AR (Augmented Reality) device, and this application does not limit this. A client that runs a target application can be installed in the terminal device 10. For example, the target application can be any type of application that requires a registered network name, such as a game application, an audio and video playback application, a forum application, a chat application, etc., and can also be an application that performs lyrics / subtitle detection. This type of application obtains the lyrics / subtitles to be disclosed and compares them with the already disclosed lyrics / subtitles corresponding to the same multimedia file. If the text similarity is too high, a prompt is given or the disclosure is canceled, thereby protecting the rights and interests of the copyright holder of the already disclosed lyrics / subtitles. This application does not limit the type of target application.

[0096] Server 20 can be a standalone physical server, a server cluster or distributed system consisting of multiple physical servers, or a cloud server providing cloud computing services. Server 20 can be the backend server of the target application described above, providing backend services to the client of the target application. Server 20 is used to provide text similarity detection results to the text publisher.

[0097] The present application provides a method for detecting text similarity. Figure 3 As shown, the method includes:

[0098] S101, obtaining at least two texts to be detected;

[0099] The text of the embodiments of the present application may come from documents, information, web pages, etc. In some embodiments, the text may also be a registered online name, such as a blog nickname, a chat application user name, an audio and video playback application user name, etc.

[0100] See Figure 4, which exemplarily shows a schematic diagram of an interface for modifying personal information in an embodiment of the present application. Users can edit user avatars, user names, personal signatures and other information through the interface. The interface includes a text box 101 to be filled in with the user name. When the object wants to register or modify the user name on the application, the new user name is filled in the text box 101. In response to the confirmation control 102 being triggered, the new user name is used as text and a text similarity check is performed. Specifically, the text similarity of the new user name can be compared with the user name that has been identified as a malicious registration. If the text similarity check passes, it is deemed that the new user name is not a malicious registration, and a pop-up window 103 is displayed on the above interface to prompt the user that the user name editing is successful.

[0101] S102: For each text, perform word segmentation on the text, and obtain a target sequence of the text according to the word segmentation result.

[0102] The method for segmenting the text is not specifically limited in this application, and may be, for example, an algorithm based on dictionary segmentation, an algorithm based on statistical machine learning, etc. The segmentation results of this application may be all individual characters in the text, and the character types may be simplified Chinese, traditional Chinese, English letters, Japanese, numbers, special characters, emoticons, etc., and are not specifically limited in this embodiment of the application.

[0103] After obtaining the word segmentation results, the target sequence of the text can be obtained based on the word segmentation results. The elements in the target sequence of the embodiment of the present application are called target elements, which are used to represent the word segmentation at the corresponding position in the word segmentation results or at least one of the attribute information of the word segmentation.

[0104] Taking the text G “Lost King 2022!” as an example, after word segmentation processing, the word segmentation result X can be expressed as:

[0105] X=['Lost','Fall','Of','K','i','n','g',”,'2','0','2','2','!','!','!']

[0106] If the target original identifier in the target sequence corresponds to the segmentation position, then the target sequence Y is the same as the segmentation result X above.

[0107] The present embodiment does not specifically limit the attribute information of the word segmentation. For example, it can be the language type of the word segmentation. The above text G contains simplified Chinese, English, numbers, and special characters. Therefore, when the target element is identified by the language type of the word segmentation, the target sequence is:

[0108] Y=['L1','L1','L1','L2','L2','L2','L2','L4','L3','L3','L3','L3','L4','L4','L4']

[0109] Among them, L1 represents Simplified Chinese, L2 represents English, L3 represents numbers, and L4 represents special characters.

[0110] S103. For the target sequence of each text, according to the order of each target element in the target sequence, each target element is successively merged with a subsequent preset number of target elements to obtain at least one merged element corresponding to the target element, and a merged element sequence of the text is obtained based on all target elements and at least one merged element.

[0111] As can be seen from the examples in the above embodiments, if the similarity between two sequences is determined only based on the differences in the elements in the two sequences, it is easy to overlook the semantics expressed by each element based on the word order (e.g., "You love me" and "I love you"). After obtaining the target sequence of each text, the embodiments of the present application do not simply mark the order of each target element in the target sequence, but successively merge each target element with a subsequent preset number of target elements to form at least one merged element corresponding to the target element. The merged element can represent the order relationship of each target element in the text more specifically and at a finer granularity.

[0112] Each target element is successively merged with the subsequent 1, 2,..., k target elements. If the number of subsequent target elements of a certain target element in the sequence is less than k, it only needs to be merged until the last target element in the sequence. Taking the above text G as an example, if k is 2, for the target element 'lost', its first subsequent target element is 'fall', and the second target element is 'of', so the obtained merged elements are 'lost' and 'lost of'; for the penultimate target element '!', since there is only one subsequent target element, the obtained merged element is only '!!' one.

[0113] After obtaining the merged element, the embodiments of the present application can obtain a merged element sequence based on the order of the target element in the target sequence and in combination with the merged element.

[0114] In an optional embodiment, the relative order between each target element in the merged element sequence remains the same as the relative order of each target element in the target sequence, and for two adjacent target elements in the target sequence, at least one merged element of the previous target element is further included in the merged element sequence.

[0115] Please refer to Figure 5, which exemplarily illustrates a flow chart of obtaining a merged element sequence based on a target sequence in an embodiment of the present application. As shown in the figure, the text is "I love eating apples," and the target sequence obtained based on the text is ['I', 'love', 'eat', 'apple', 'fruit']. If k is determined to be 2, that is, each target element is merged with the next 1 and next 2 target elements, respectively, the merged elements that can be obtained include:

[0116] 'I love', 'I love to eat',

[0117] 'Love to eat', 'Love to eat apples',

[0118] 'Eat apple', 'Eat apple',

[0119] 'apple',

[0120] The merged element sequence is: ['I', I love', 'I love to eat', 'love', 'love to eat', 'love to eat apples', 'eat', 'eat apples', 'eat apples', 'apple', 'apple', 'fruit'].

[0121] It can be seen from the merged element sequence that the relative order between the target elements in the merged element sequence remains unchanged from the relative order of the target elements in the target sequence. In addition, two adjacent target elements in the target sequence also include at least one merged element of the previous target element in the merged element sequence. The relative order between the merged elements corresponding to the same target element is also fixed - the fewer the number of target elements in the merged element, the closer they are to the front.

[0122] It should be noted that the present application obtains at least one merged element corresponding to the target element by merging each target element in sequence with a subsequent preset number of target elements. Compared with the existing technology, it can more accurately identify the differences between texts with different numbers of words containing reduplications.

[0123] S104 , encoding the merged element sequence of each text, and obtaining the frequency vector of each text according to the encoding result of the merged element sequence of each text.

[0124] It should be understood that after obtaining the merged element sequence of each text, the embodiment of the present application can list the various elements (including target elements and merged elements) in the merged element sequence of all texts to form a total merged sequence, and there are no repeated elements in the total merged sequence. Specifically, the various elements in the merged element sequence of all texts can be placed in a set, the set can be converted into a dictionary, and a key-value pair can be constructed, where the key is the various elements in the set, and the value is the position number of each element in the set, and the position number can be counted starting from 0.

[0125] Each merged element sequence is encoded, that is, each element is converted into the position where it appears in the set, and the frequency statistics of the merged element sequence are performed, that is, the number of times each element appears is calculated, and finally the frequency sequence corresponding to each text is obtained. It should be understood that the number of elements in the frequency sequence is the number of elements in the set, and the elements in the frequency sequence are used to represent the frequency of the corresponding elements in the total merged sequence (which can also be the above-mentioned set) in the corresponding merged element sequence, and the frequency sequence of each text is used as a frequency vector.

[0126] exist Figure 5 Based on the embodiment shown, if there is another text asking "I love eating nuts", the merged element sequence of the text can be determined accordingly as:

[0127] ['I', I love', 'I love to eat', 'love', 'love to eat', 'love to eat nuts', 'eat', 'eat nuts', 'eat nuts', 'nuts', 'nuts'].

[0128] If the merged element sequence of the text "I love apples" is defined as List1, and the merged element sequence of the text "I love nuts" is defined as List2, then all elements in List1 and List2 are counted to form a set:

[0129] set = {'I', 'I love', 'I love to eat', 'love', 'love to eat', 'love to eat apples', 'love to eat nuts', 'eat', 'eat apples', 'eat nuts', 'eat apples', 'eat nuts', 'apples', 'nuts', 'apples', 'nuts', 'fruits'}

[0130] The above set set is converted into a dictionary dict. In one embodiment, when determining the position of each element in the dictionary, the embodiment of the present application can determine the corresponding position in sequence according to the order of the elements in each sequence.

[0131] See Figure 6 , which exemplarily shows a flow chart of determining the position of an element in an embodiment of the present application, as shown in the figure, including:

[0132] S201, number each merge element sequence; traverse from the merge element sequence with the smallest number;

[0133] S202: Determine whether there are any untraversed elements (including target elements and merging elements) in the currently traversed merged element sequence. If so, execute S203; if not, execute S205.

[0134] S203: Determine the element with the highest order among the elements that have not been traversed as the element currently traversed;

[0135] S204, determine whether the element already exists in the dictionary, if not, execute S207; if it exists, execute S205;

[0136] S205: Determine whether the number of the currently traversed sequence is the maximum number. If the number of the currently traversed sequence is not the maximum number, execute S206; if the number of the currently traversed sequence is the maximum number, end the process.

[0137] S206: The next sequence after the currently traversed sequence is used as the new currently traversed sequence, and the process returns to step S202.

[0138] S207: Determine that the element is the last position in the dictionary, and return to step S205.

[0139] Based on the above embodiments, the dictionary obtained for the text "I love to eat apples" and the text "I love to eat nuts" in this embodiment of the application is:

[0140] dict = {'I': 0, 'I love': 1, 'I love to eat': 2, 'love': 3, 'love to eat': 4, 'love to eat apples': 5, 'love to eat nuts': 6, 'eat': 7, 'eat apples': 8, 'eat nuts': 9, 'eat apples': 10, 'eat nuts': 11, 'apples': 12, 'nuts': 13, 'apples': 14, 'nuts': 15, 'fruit': 16}

[0141] Encode List1 and List2 separately, converting each element into its position in the set. After conversion, it becomes:

[0142] List1code=[0,1,2,3,4,5,7,8,10,12,14,16]

[0143] List2code=[0,1,2,3,4,6,7,9,11,13,15,16]

[0144] Performing frequency statistics on list1code and list2code means calculating the number of times each word appears. The resulting frequency sequence is as follows:

[0145] List1codeOneHot=[1,1,1,1,1,1,0,1,1,0,1,0,1,0,1,0,1]

[0146] List2codeOneHot=[1,1,1,1,1,0,1,1,0,1,0,1,0,1,0,1,1]

[0147] The frequency vector of the text “I love apples” is: (1,1,1,1,1,1,0,1,1,0,1,0,1,0,1,0,1);

[0148] The frequency vector of the text “I love nuts” is: (1,1,1,1,1,0,1,1,0,1,0,1,0,1,0,1,1).

[0149] S105 . For any two texts, obtain the text similarity between the two texts according to the frequency vectors of the two texts.

[0150] Specifically, the present application may calculate the cosine similarity of the frequency vectors of the two texts as the text similarity of the two texts.

[0151] The text similarity detection method of the embodiment of the present application obtains at least two texts to be detected, performs word segmentation on each text, and obtains a target sequence of the text according to the word segmentation results. The target element in the target sequence can be the word segmentation at the corresponding position in the word segmentation result, or it can be the attribute information of the word segmentation. It can first detect multiple similarities from different angles. For the target sequence of each text, according to the order of each target element in the target sequence, each target element is merged with a subsequent preset number of target elements in turn to obtain at least one merged element corresponding to the target element. The merged element sequence of the text is obtained according to all the target elements and at least one merged element, which can represent the text more specifically and fine-grainedly. The sequential relationship between the target elements in this method can also identify the subtle differences between texts with reduplication, laying the foundation for more accurate analysis of the semantics of the text. The merged element sequence of each text is encoded, and the frequency vector of each text is obtained based on the encoding results of the merged element sequence of each text; the features of each dimension in the frequency vector are used to represent the frequency of the corresponding element in the total merged sequence in the merged element sequence of the text. For any two texts, the text similarity of any two texts is obtained based on the frequency vectors of any two texts, which has higher accuracy and is more suitable for the network name review scenario. It can effectively mine the behavior of batch malicious registration of nicknames and batch group gatherings, and issue risk warnings.

[0152] The following two specific examples illustrate that the embodiments of the present application pay more attention to word order and reduplication, thereby being able to more accurately determine text similarity.

[0153] Example 1

[0154] For the two texts "you love me" and "I love you", assuming that each target element is sequentially associated with the next two target elements, then:

[0155] The merged element sequence of "you love me" is ['you', 'you love', 'love', 'love me', 'me'];

[0156] The merged element sequence of "I love you" is ['I', 'I love', 'love', 'love you', 'you'];

[0157] All target elements and merged elements include ['you', 'you love', 'love', 'love me', 'I', 'I love', 'love you'], and the frequency vectors are (1, 1, 1, 1, 1, 0, 0) and (1, 0, 1, 0, 1, 1, 1) respectively. The cosine similarity is:

[0158]

[0159] For the nicknames "you love me" and "I love you", assuming that each target element is sequentially combined with the subsequent 3 target elements, then:

[0160] After word segmentation, "you love me" is ['you', 'you love', 'you love me', 'love', 'love me', 'I'];

[0161] After word segmentation, "I love you" is ['I', 'I love', 'I love you', 'love', 'love you', 'you'];

[0162] All target elements and merged elements are ['you', 'you love', 'you love me', 'love', 'love me', 'I', 'I love', 'love you', 'I love you'], and the frequency vectors are (1, 1, 1, 1, 1, 1, 0, 0, 0) and (1, 0, 0, 1, 0, 1, 1, 1, 1) respectively. The cosine similarity is:

[0163]

[0164] It can be seen from Example 1 that in some cases, when merging elements based on target elements, the more subsequent target elements are involved, the lower the obtained cosine similarity, and thus the differences between texts can be more significantly distinguished.

[0165] Example 2

[0166] For example, for two texts: "haha" and "hahaha", if each target element is set to be combined with the subsequent two target elements, then:

[0167] After word segmentation, "haha" is ['ha', 'haha', 'ha'];

[0168] After word segmentation, "hahaha" is ['ha', 'haha', 'ha', 'haha', 'ha', 'haha', 'ha'];

[0169] The elements of all merged element sequences include ['ha', 'haha'], and the frequency vectors of the two texts are (2, 1) and (4, 3) respectively. At this time, the cosine similarity is:

[0170]

[0171] If each target element is set to be merged with the subsequent three target elements, then:

[0172] "哈哈" is segmented into ['哈', '哈哈', '哈'];

[0173] "哈哈哈哈" is segmented into ['哈', '哈哈', '哈哈哈', '哈', '哈哈', '哈哈哈', '哈', '哈哈', '哈'];

[0174] The elements of all merged element sequences include ['哈', '哈哈', '哈哈哈'], and the frequency vectors of the two texts are (2, 1, 0) and (4, 3, 2) respectively. At this time, the cosine similarity is:

[0175]

[0176] Compared with the prior art that determines the text similarity between "哈哈" and "哈哈哈" as 1, this application can discover the differences between the two texts, thereby reducing omissions when determining malicious or batch-registered online names. In some cases, when obtaining merged elements based on target elements, the more subsequent target elements involved, the lower the obtained cosine similarity, and thus the differences between texts can be more significantly distinguished.

[0177] Based on the above embodiments, as an optional embodiment, obtaining the target sequence of the text according to the segmentation result includes:

[0178] Obtaining the segmentation sequence of the text according to the segmentation result;

[0179] Obtaining the target sequence of the text according to the segmentation sequence of the text.

[0180] In the embodiments of this application, the segmentation sequence can be used as the target sequence, or the attributes of each segmentation in the segmentation sequence can be further analyzed, and the target sequence is obtained based on the attributes of each segmentation. That is, the target elements in the target sequence are used to represent the attribute information of the corresponding segmentation.

[0181] In the embodiments of this application, the segmentation sequence of the text is obtained according to the segmentation result of the text, and then the target sequence is obtained according to the segmentation sequence of the text. The segmentation sequence can be directly used as the target sequence, or the attributes of each segmentation in the segmentation sequence can be further analyzed, and the target sequence is obtained based on the attributes of each segmentation, providing multiple feasible solutions to obtain the target sequence and providing flexibility for similarity detection.

[0182] Based on the above embodiments, as an optional embodiment, obtaining the segmentation sequence of the text according to the segmentation result includes:

[0183] Obtaining the initial word segmentation sequence of the text based on the word segmentation results;

[0184] If there are no consecutive segmented words of the first target type in the initial segmented word sequence, the initial segmented word sequence is determined as a segmented word sequence;

[0185] If there are continuous segmented words of the first target type in the initial segmented word sequence, the continuous segmented words of the target type are replaced as a whole with a preset number of preset segmented words, and the replaced initial segmented word sequence is used as the segmented word sequence.

[0186] Considering that maliciously registered online names in the online name review scenario usually present batch information, such as "fruit wholesale xxx", where "xxx" is the sequence information when the batch is generated, which may range from 001 to 999.

[0187] After obtaining the initial word segmentation sequence, the embodiment of the present application further determines whether there are continuous word segmentations of the first target type in the initial word segmentation sequence. It can be understood that the embodiment of the present application can pre-set the number of characters that meet the "continuous" condition. When the number of word segmentations of the first target type exceeds the preset number of characters (for example, 3), it is determined that there are continuous word segmentations of the first target type in the initial word segmentation sequence.

[0188] The first target type of the embodiment of the present application may include a first specified type and a second specified type, and for the first specified type and the second specified type, the embodiment of the present application has a corresponding method for updating the initial word segmentation sequence:

[0189] A) If there are consecutive first-type segmentations in the initial segmentation sequence, the consecutive first-type segmentations are replaced with first target segmentations, where the first target segmentations are used to represent the number of consecutive first-type segmentations.

[0190] B) If there are consecutive segmentations of the second specified type in the initial segmentation sequence, the consecutive segmentations of the second specified type are replaced with second target segmentations, where the second target segmentations are a combination of consecutive segmentations of the second specified type.

[0191] The embodiment of the present application does not limit the specific types of the first specified type and the second specified type. For example, the first specified type may be numbers, and the second specified type may be letters, such as English letters, Latin letters, phonetic letters, Cyrillic letters, etc.

[0192] Taking the text "Lost King 2022!!!" as an example, the embodiment of the present application can decompose all individual characters in the text (including simplified Chinese, traditional Chinese, English letters, Japanese, Korean, numbers, special characters, emoticons, etc.) to obtain the initial word segmentation sequence:

[0193] X=['Lost','Fall','Of','K','i','n','g',”,'2','0','2','2','!','!','!']

[0194] Since the word segmentation sequence contains the first specified type: numbers, and the second specified type: English letters, for numbers, consecutive numbers can be merged until they cannot be merged. At the same time, the character length of the merged number combination is counted. If the length is n, the number string is recorded as NUM_n and replaced. Since the number combination in the above text is "2022" and the length is 4, it is replaced with "NUM_4". For English letters, consecutive English letters can be spliced ​​together and further converted to lowercase "king". The updated word segmentation sequence is:

[0195] X'=['Lost','Fall','Of','King',”,'NUM_4','!','!','!'].

[0196] The embodiment of the present application can identify in the network name detection scenario that there are situations where users are classified by serial numbers when registering through mobile phone numbers or batch accounts, such as "Xinxin Company Xiao Liu 13X11111111", "Xinxin Company A Liang 15X111111111", "Xinxin Customer Service 001", and "Xinxin Customer Service 002".

[0197] Based on the above embodiments, as an optional embodiment, obtaining a target sequence of a text according to a word segmentation sequence of the text includes:

[0198] S301, determining the type of each segmentation in the segmentation sequence;

[0199] S302: Obtain a word type sequence of the text according to the type of each word segment, where each element in the word type sequence is used to represent the type of the word segment at the corresponding position in the word segment sequence;

[0200] S303: Obtain a target sequence of the text according to the word type sequence of the text.

[0201] In the embodiment of the present application, the type of word segmentation may refer to the language type of the word segmentation, such as simplified Chinese, traditional Chinese, English, numbers, and special characters.

[0202] In some embodiments, in response to the situation where the same segmented word may belong to multiple types (for example, a Chinese character can belong to both simplified Chinese and traditional Chinese), the present application sets priority information for each type, so that when a segmented word belongs to multiple types, the type with the higher priority is used as the segmented word type. For example, the priority can be in descending order of simplified Chinese, traditional Chinese, English, numbers, and special characters.

[0203] In the embodiment of the present application, the word type sequence can be directly used as the target sequence, so that the attribute information of the word segmentation represented by the target element in the target sequence is the type of the word segmentation.

[0204] For example, in the text "A fruit wholesale 1", "A" is English, "water", "fruit", "batch", and "delivery" are Chinese, and "1" is a number. If L1 represents Chinese, L2 represents English, and L3 represents numbers, then the word type sequence of this text is ['L2','L1','L1','L1','L1','L3']. If this type sequence is used as the target sequence, the target sequence of this text is also ['L2','L1','L1','L1','L1','L3'].

[0205] Based on the above embodiments, as an optional embodiment, obtaining a target sequence of a text according to a word type sequence of the text includes:

[0206] For each element in the word type sequence, if the type represented by the element belongs to the second target type, then determine the order of the element among the type elements belonging to the second target type in the word type sequence; if the type represented by the element does not belong to the second target type, then determine the order of the element among the elements not belonging to the second target type in the word type sequence;

[0207] According to the order of the corresponding elements of each type in the word type sequence, the attribute sequence is obtained. The attribute information of the word segmentation also includes the order of the elements corresponding to the word segmentation.

[0208] The embodiment of the present application can obtain a target sequence by determining whether each element in the word type sequence belongs to the second target type, and the order of the element in the corresponding type (i.e., either the second target type or not the second target type). In this way, the attribute information represented by the elements in the target sequence also includes the order of the elements corresponding to the word segmentation. That is, the attribute information of the target element of the target sequence of the embodiment of the present application includes both the type of the target element and the order of the target element in the corresponding type, thereby strengthening the word order information of the text.

[0209] Taking the above text "A Fruit Wholesale 1" as an example, based on obtaining the word type sequence ['L2', 'L1', 'L1', 'L1', 'L1', 'L3'], if it is determined that the second target type is Chinese, it can be known that both English and numbers do not belong to the second target type. Further, set the second target type to be represented as A, and the non-second target type to be represented as B. Then, the target sequence can be known as: ['B1', 'A1', 'A2', 'A3', 'A4', 'B2']. Specifically, since "L2" does not belong to the second target type and is the first element that does not belong to the second target type, 'L2' corresponds to 'B1' in the target sequence. Similarly, "fa" corresponds to the 4th element 'L1' in the word type sequence, and "L1" belongs to the second target type, so 'L1' corresponds to 'A4' in the target sequence.

[0210] Please refer to Figure 7a to Figure 7c , which exemplarily shows a schematic diagram of the interface for the detection party to set the detection method in the abnormal online name detection scenario. It should be noted that setting the detection method can be executed after importing the text to be detected, or can be executed before the text to be detected. This application does not make specific limitations. For the convenience of description, the detection method in the embodiments of this application is set to be executed after importing the text to be detected.

[0211] The embodiments of this application can calculate the similarity between any two imported texts, or can also be the similarity between the imported text and multiple sample texts pre-set in the background. This application does not make specific limitations.

[0212] Such as Figure 7a shown, it is a schematic diagram of the initial interface for setting the detection method. This interface has two triggerable controls: the first detection control 201 and the second detection control 202. The embodiments of this application do not make specific limitations on the triggering method. For example, it can be a single click, double click, swipe, etc.

[0213] When the detection party triggers the first detection control 201, it means that the detection party hopes to perform similarity detection based on the word segmentation of the text (detection method 1). In this case, the target elements of the target sequence are used to represent the word segments at the corresponding positions in the word segmentation result. This detection method belongs to the most basic detection method and is relatively faster in detection. For the detection party who sets the detection method for the first time, they can click "Learn More" on the right side of the first detection control 201 to know the specific logic of this detection method.

[0214] In some embodiments, after the detecting party triggers the first detection control 201, the website may further display a prompt message indicating that the setting is completed, and may also further display a prompt message indicating whether to continue setting other detection methods. If the detecting party determines not to continue setting other detection methods, the text similarity detection is performed based on detection method 1, and after the detection is completed, the similarity detection results between the two texts are displayed.

[0215] When the detecting party triggers the second detection control 202, it means that the detecting party wants to perform similarity detection on the attribute information of the base word (detection method 2). In this case, the target element of the target sequence is used to represent the attribute of the word at the corresponding position in the word segmentation result. After the detecting party triggers the second detection control 202, the website jumps to a new interface. Figure 7b , which exemplarily shows an interface diagram of the second detection method set in this application.

[0216] It should be noted that when the testing party decides to continue to set up other testing methods, the website will also be adjusted to Figure 7b As shown in the interface, after the user completes the setting of detection method 2, the website will obtain the similarity between the texts to be tested based on detection method 1 and detection method 2 based on parallel business processes, and after obtaining the similarity detection results, it will display the two similarity detection results at the same time.

[0217] like Figure 7b As shown, the interface further displays the currently supported language types, and the user can further adjust the priority order of each language type. In this embodiment, the initial priority order of each voice type is displayed. When the user wants to adjust a certain language type, the language type control corresponding to the language type can be dragged to move the control before or after another language type control. For example, the initial priority order of simplified Chinese in the left figure is 1, the initial priority order of English is 3, and the priority order of Arabic numerals is 4. Moving the Arabic numeral language type control to the left of the English language type control realizes the update of the English priority order to 4 and the Arabic numeral priority order to 3 as shown in the right figure, thereby realizing a quick and intuitive adjustment of the priority order of the language types. When the detection party determines that the order of the language types has been set, the operation confirmation control 203 means that similarity detection can be performed based on the type of word segmentation.

[0218] See Figure 7c, which exemplarily shows a schematic diagram of the interface after the detection party operates the confirmation control, wherein the left figure shows the prompt information 204, which is used to prompt the detection party whether to further set the second target type. If the second target type is not set, the type of each word segmentation is determined based on the priority of the set language type, and the text similarity is calculated based on the type of each word segmentation. If the detection party sets the second target type, the second target type is further set. The right figure shows that the second target type is set to simplified Chinese. After the setting is completed, it can be determined whether the type of each word segmentation belongs to the second target type based on the set second target type. The attribute information of the word segmentation in the target sequence obtained in this way also includes the order of the elements corresponding to the word segmentation.

[0219] See Figure 8 , which exemplarily shows an interface diagram for displaying text similarity when determining a detection method based on word segmentation (detection method 1) and a detection method based on a specified second target type (detection method 2) in an embodiment of the present application. As shown in the figure, the interface provides a query interface corresponding to different detection methods. When the detection party operates query interface 1 and enters a preset threshold, texts whose similarity obtained based on detection method 1 reaches the preset threshold can be displayed;

[0220] When the detection party operates the query interface 2 and inputs the preset threshold, the texts whose similarity reaches the preset threshold obtained based on the detection method 2 are displayed;

[0221] When the detection party operates the query interface 3 and inputs two preset thresholds (corresponding to detection method 1 and detection method 2 respectively), the texts that meet the similarity obtained by detection method 1 and detection method 2 and reach the corresponding preset thresholds can be displayed.

[0222] For example, there are three texts to be tested:

[0223] A1='コール001';

[0224] A2='コール10000000000';

[0225] A3='Customer Service 002';

[0226] Segment the three texts separately, and set each segmentation to be merged with the next two segmentations in turn; the sequence of segmentations and merged words of the three texts is:

[0227] A11=['コ','コー','ー','ール','ル','ルNUM_3','NUM_3'];

[0228] A12=['コ','コー','ー','ール','ル','ルNUM_11','NUM_11'];

[0229] A13 = ['customer', 'customer service', 'service', 'service NUM_3', 'NUM_3']

[0230] By setting the second target type as Japanese in the above embodiments, that is, the detecting party needs to calculate the similarity of language nicknames in Japanese (assuming that the language library settings include Simplified Chinese, Japanese, numbers, English, and the priorities are the same as those in the above embodiments), determine whether each element in the above sequence is Japanese, and determine the order in the language type to obtain a new sequence:

[0231] L11 = ['A1', 'A1A2', 'A2', 'A2A3', 'A3', 'A3B1', 'B1']

[0232] L12 = ['A1', 'A1A2', 'A2', 'A2A3', 'A3', 'A3B1', 'B1']

[0233] L13 = ['B1', 'B1B2', 'B2', 'B2B3', 'B3']

[0234] It is easy to obtain that the similarity of detection method 1 between A1 and A2 is: 0.7143, and the similarity of detection method 2 is: 1;

[0235] It is easy to obtain that the similarity of detection method 1 between A1 and A3 is: 0.169, and the similarity of detection method 2 is: 0;

[0236] It is easy to obtain that the similarity of detection method 1 between A2 and A3 is: 0, and the similarity of detection method 2 is: 0;

[0237] If it is necessary to select the text pairs with the similarity of detection method 1 exceeding 0.6 and the similarity of detection method 2 exceeding 0.6, then filter and display the text pairs ['call 001', 'call 10000000000'] for subsequent use.

[0238] Based on the above embodiments, as an optional embodiment, the method further includes:

[0239] For any two texts, if it is determined that there is no element belonging to the second target type in the word type sequence corresponding to one of the two texts, and there is an element belonging to the second target type in the word type sequence corresponding to the other text, then it is determined that the text similarity of the two texts is not higher than a preset threshold (for example, 0).

[0240] In an embodiment of the present application, if it is determined that the detecting party has pre-set a second target type, it is determined that the similarity of the text under the second target type needs to be calculated. Then, if the second target type is not involved in one of the texts to be compared but is involved in the other text, the similarity between the two texts can be directly determined to be not higher than a preset threshold, which can greatly improve the efficiency of similarity calculation.

[0241] For example, for the following texts:

[0242] Text 1 = 'Ann Huang';

[0243] Text 2 = 'bob';

[0244] If the second target type is specified as Chinese, that is, the similarity of the simplified Chinese language of Text 1 and Text 2 needs to be calculated. According to the above embodiment, since Text 2 does not contain simplified Chinese, the cosine similarity of the simplified Chinese language of Text 1 and Text 2 is directly set to 0 at this time.

[0245] Based on the above embodiments, as an optional embodiment, determining the type of each token in the token sequence includes:

[0246] For each token in the token sequence, if the token belongs to multiple candidate types, for each candidate type, determine the token frequency of the tokens in the token sequence that belong to the candidate type, and determine the type of the token according to the candidate type corresponding to the highest token frequency.

[0247] In the embodiment of the present application, for a token belonging to multiple candidate types, the candidate type corresponding to the highest token frequency is determined as the type of the token, which can highlight the differences between different types of tokens in the text and more accurately determine the characteristics of the text based on the differences, helping to improve the determination of text similarity.

[0248] For example, if there is a token sequence ['中', '國', '!'], where the token "中" can belong to either simplified Chinese or traditional Chinese. Therefore, the candidate types of the token "中" include simplified Chinese and traditional Chinese. When it belongs to simplified Chinese, there is 1 simplified Chinese token, 1 traditional Chinese token, and 1 special character in this token sequence. When it belongs to traditional Chinese, there are 2 traditional Chinese tokens and 1 special character in this token sequence. It can be seen that when the token "中" belongs to traditional Chinese, the token frequency of traditional Chinese is higher. Therefore, the type sequence ['L2', 'L2', 'L5'] is generated (in this embodiment, L2 represents traditional Chinese and L5 represents a special character).

[0249] When determining the type of a segmentation, in an embodiment of the present application, if a segmentation belongs to multiple candidate types, the word frequency of the segmentation of each candidate type is determined, and the type of the segmentation is determined based on the candidate type corresponding to the highest word frequency, so that the distribution differences of different types in the obtained target sequence are more significant, so that the differences between the two texts can be more significantly distinguished when calculating the similarity.

[0250] Based on the above embodiments, as an optional embodiment, determining the type of word segmentation according to the candidate type corresponding to the highest word frequency includes:

[0251] If the candidate type corresponding to the highest word frequency is unique, the candidate type corresponding to the highest word frequency is used as the type of the segmentation;

[0252] If the candidate type corresponding to the highest word frequency is not unique, the candidate type with the highest priority and the highest word frequency is used as the type of the word segmentation according to the predetermined type priority.

[0253] In the embodiment of the present application, for the case where there may be multiple candidate types corresponding to the highest word frequency, the type of word segmentation is determined according to the situation. When the candidate type corresponding to the highest word frequency is unique, the candidate type corresponding to the highest word frequency is directly used as the type of word segmentation. In particular, when the candidate type corresponding to the highest word frequency is not unique, the priority relationship of the types is pre-set and the type of word segmentation is determined based on the priority relationship. This is beneficial in actual application, as it allows the detection party to dynamically adjust the priority of each type according to the main language environment of the application, and obtain similarity comparison results that conform to the language environment.

[0254] For example, there are differences in the languages ​​and characters used by netizens in different regions, so there will be differences in the characters used by netizens in different regions when registering online names. Therefore, in some embodiments, the priority relationship of different languages ​​and characters can be dynamically set based on the user's registration location (which can be determined by the user's IP address or the mobile phone number used for registration).

[0255] See Figure 9 , Figure 9 A structural diagram of a text similarity detection system applicable to the scenario embodiment of the present application is shown. As shown in the figure, the detection system includes a user terminal, a text server, a similarity detection server and a verification platform.

[0256] The user terminal can be connected to the similarity detection server via a network. The user terminal can be a terminal running any application with a user registration function. The embodiment of the present application does not limit the type of application. It can be an application that the user needs to download and install, a cloud application, or a game application in a mini-program. When a user needs to register or modify a username, the user terminal uploads the new username to the similarity detection server via the network.

[0257] The text server stores a certain number of abnormal usernames registered in batches. Each abnormal username is stored in the text server as a sample text. The text server can classify and store the sample texts according to the language type involved. That is, each storage interval is used to store sample texts with the same main language type. The main language type is the voice type with the largest number of word segments in the text.

[0258] The text similarity tester logs in to the verification platform. The verification platform can provide the testing party with an optional detection method (that is, the target element in the target sequence is used to represent the segmentation at the corresponding position in the segmentation result or the attribute information of the segmentation, and the attribute information can further be at least one of the language type and the order of the corresponding elements). The verification platform sends the detection method determined by the testing party to the similarity detection server. The similarity detection server uses the new user name as the text to be tested, and calculates the similarity between the text to be tested and each sample text according to the determined detection method. If the similarity between the text to be tested and a preset number of sample texts exceeds a threshold, the text to be tested is determined to be an abnormal user name, and a prompt message of name abnormality is sent to the terminal to enable the user to re-edit the user name.

[0259] The similarity detection server can also feed back the detected abnormal user name to the verification platform for manual verification by the detection party. If the manual verification determines that it is correct, the detected abnormal user name can be stored in the text server.

[0260] The present application embodiment provides a device for detecting text similarity, such as Figure 10 As shown, the text similarity detection device may include: a text acquisition module 1001, a target sequence acquisition module 1002, a merging module 1003, a frequency vector module 1004 and a similarity calculation module 1005, wherein:

[0261] The text acquisition module 1001 is used to acquire at least two texts to be detected;

[0262] The target sequence obtaining module 1002 is used to segment each text and obtain a target sequence of the text based on the segmentation results; the target element in the target sequence is used to represent the segmentation at the corresponding position in the segmentation result or at least one of the attribute information of the segmentation;

[0263] Merging module 1003 is configured to, for each target sequence of text, sequentially merge each target element with a subsequent preset number of target elements according to the order of the target elements in the target sequence to obtain at least one merged element corresponding to the target element, and obtain a merged element sequence of the text based on all the target elements and the at least one merged element;

[0264] Frequency vector module 1004 is used to encode the merged element sequence of each text and obtain a frequency vector for each text based on the encoding results of the merged element sequence of each text. The features of each dimension in the frequency vector are used to represent the frequency of the corresponding element in the total merged sequence in the merged element sequence of the text. The total merged sequence is obtained by sorting the target element and the merged elements in the merged element sequences of all texts. There are no repeated elements in the total merged sequence.

[0265] The similarity calculation module 1005 is used to obtain the text similarity between any two texts based on the frequency vectors of the any two texts.

[0266] The device of the embodiment of the present application can execute the method provided by the embodiment of the present application, and its implementation principle is similar. The actions performed by each module in the device of each embodiment of the present application correspond to the steps in the method of each embodiment of the present application. For the detailed functional description of each module of the device, please refer to the description in the corresponding method shown in the previous text, and will not be repeated here.

[0267] An embodiment of the present application provides an electronic device, including a memory, a processor, and a computer program stored in the memory. The processor executes the computer program to implement the steps of the text similarity detection method. Compared with the related art, the following can be achieved:

[0268] By obtaining at least two texts to be detected, each text is segmented, and a target sequence of the text is obtained according to the segmentation results. The target elements in the target sequence can be the segmentations at the corresponding positions in the segmentation results, or the attribute information of the segmentations. The target elements can be detected from different angles and multiple similarities. For the target sequence of each text, each target element is merged with a subsequent preset number of target elements according to the order of each target element in the target sequence to obtain at least one merged element corresponding to the target element. The merged element sequence of the text is obtained according to all the target elements and at least one merged element, which can represent the target elements in the text in a more specific and fine-grained manner. Sequential relationship, and at the same time, it can also identify the subtle differences between texts with reduplication, laying the foundation for more accurate analysis of the semantics of the text, encoding the merged element sequence of each text, and obtaining the frequency vector of each text according to the encoding result of the merged element sequence of each text; the features of each dimension in the frequency vector are used to represent the frequency of the corresponding element in the total merged sequence in the merged element sequence of the text. For any two texts, the text similarity of any two texts is obtained according to the frequency vectors of any two texts, which has higher accuracy and is more suitable for the network name review scenario, and can effectively mine the behavior of batch malicious registration of nicknames and batch group gathering, and issue risk warnings.

[0269] As an optional embodiment, the target sequence acquisition module includes:

[0270] The word segmentation sequence submodule is used to obtain the word segmentation sequence of the text according to the word segmentation results;

[0271] The target sequence submodule is used to obtain the target sequence of the text based on the word segmentation sequence of the text.

[0272] As an optional embodiment, the word segmentation sequence submodule includes:

[0273] The initial word segmentation unit is used to obtain the initial word segmentation sequence of the text according to the word segmentation results;

[0274] a continuous word segmentation judgment module, configured to determine the initial word segmentation sequence as a word segmentation sequence if there are no continuous word segments of the first target type in the initial word segmentation sequence;

[0275] If there are continuous segmentation words of the first target type in the initial segmentation sequence, the continuous segmentation words of the target type are replaced as a whole with a preset number of preset segmentation words, and the replaced initial segmentation sequence is used as the segmentation sequence

[0276] As an optional embodiment, the continuous word segmentation judgment module is specifically used to:

[0277] If there are consecutive first-type segmentations in the initial segmentation sequence, the consecutive first-type segmentations are replaced with first target segmentations, where the first target segmentations are used to represent the number of consecutive first-type segmentations.

[0278] If there are consecutive segmentations of the second specified type in the initial segmentation sequence, the consecutive segmentations of the second specified type are replaced with second target segmentations, where the second target segmentations are a combination of consecutive segmentations of the second specified type.

[0279] As an optional embodiment, the target sequence submodule includes:

[0280] A segmentation type unit, used to determine the type of each segmentation in the segmentation sequence;

[0281] The word type sequence unit is used to obtain the word type sequence of the text according to the type of each word segmentation. Each element in the word type sequence is used to represent the type of the word segmentation at the corresponding position in the word segmentation sequence.

[0282] The target sequence unit is used to obtain the target sequence of the text according to the word type sequence of the text. The attribute information of the word segmentation includes the type of the word segmentation.

[0283] As an optional embodiment, the target sequence unit includes:

[0284] a type determination unit configured to, for each element in the word type sequence, determine, if the type represented by the element belongs to the second target type, the order of the element among the type elements belonging to the second target type in the word type sequence; and, if the type represented by the element does not belong to the second target type, determine the order of the element among the elements not belonging to the second target type in the word type sequence;

[0285] The order unit is used to obtain the target sequence according to the order of each element in the word type sequence. The attribute information of the word segmentation also includes the order of the elements corresponding to the word segmentation.

[0286] As an optional embodiment, the word segmentation type unit is specifically used to:

[0287] For each word in the word segmentation sequence, if the word segmentation belongs to multiple candidate types, then for each candidate type, the word frequency of the word segmentation belonging to the candidate type in the word segmentation sequence is determined, and the type of the word segmentation is determined according to the candidate type corresponding to the highest word frequency.

[0288] As an optional embodiment, the word segmentation type unit includes:

[0289] A first case determination unit is configured to use the candidate type corresponding to the highest word frequency as the type of the segmentation word if the candidate type corresponding to the highest word frequency is unique;

[0290] The second case determination unit is used to determine, if the candidate type corresponding to the highest word frequency is not unique, the candidate type with the highest priority and the highest word frequency as the type of the word segmentation according to a predetermined type priority.

[0291] In an alternative embodiment, an electronic device is provided, such as Figure 11 As shown, Figure 11 The electronic device 4000 shown includes: a processor 4001 and a memory 4003. The processor 4001 and the memory 4003 are connected, for example, via a bus 4002. Optionally, the electronic device 4000 may further include a transceiver 4004, which may be used for data exchange between the electronic device and other electronic devices, such as data transmission and / or data reception. It should be noted that in actual applications, the number of transceivers 4004 is not limited to one, and the structure of the electronic device 4000 does not constitute a limitation on the embodiments of the present application.

[0292] Processor 4001 may be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It may implement or execute the various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of this application. Processor 4001 may also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, and the like.

[0293] Bus 4002 may include a path for transmitting information between the above components. Bus 4002 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus. Bus 4002 may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 11 Only one thick line is used in the diagram, but this does not mean that there is only one bus or one type of bus.

[0294] The memory 4003 can be a ROM (Read Only Memory) or other types of static storage devices that can store static information and instructions, a RAM (Random Access Memory) or other types of dynamic storage devices that can store information and instructions, or an EEPROM (Electrically Erasable Programmable Read Only Memory), a CD-ROM (Compact Disc Read Only Memory) or other optical disk storage, optical disk storage (including compact discs, laser discs, optical discs, digital versatile discs, Blu-ray discs, etc.), magnetic disk storage media, other magnetic storage devices, or any other medium that can be used to carry or store computer programs and can be read by a computer, without limitation here.

[0295] The memory 4003 is used to store the computer program for executing the embodiment of the present application, and the execution is controlled by the processor 4001. The processor 4001 is used to execute the computer program stored in the memory 4003 to implement the steps shown in the above method embodiment.

[0296] An embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the steps and corresponding contents of the aforementioned method embodiment can be implemented.

[0297] An embodiment of the present application also provides a computer program product, including a computer program, which can implement the steps and corresponding contents of the aforementioned method embodiment when executed by a processor.

[0298] The terms "first," "second," "third," "fourth," "1," "2," and the like (if any) in the specification and claims of this application and the accompanying drawings are used to distinguish similar objects and are not necessarily used to describe a particular order or sequential sequence. It should be understood that the terms used in this manner are interchangeable where appropriate, so that the embodiments of the application described herein can be implemented in an order other than that shown or described in the drawings.

[0299] It should be understood that, although each operation step is indicated by arrows in the flowchart of the embodiment of the present application, the order of implementation of these steps is not limited to the order indicated by the arrows. Unless otherwise clearly stated herein, in some implementation scenarios of the embodiment of the present application, the implementation steps in each flowchart can be performed in other orders according to demand. In addition, some or all of the steps in each flowchart can include multiple sub-steps or multiple stages based on actual implementation scenarios. Some or all of these sub-steps or stages can be executed at the same time, and each sub-step or stage in these sub-steps or stages can also be executed at different times respectively. Under different scenarios at the execution time, the execution order of these sub-steps or stages can be flexibly configured according to demand, and the embodiment of the present application does not limit this.

[0300] The above are only optional implementation methods for some implementation scenarios of this application. It should be pointed out that for ordinary technicians in this technical field, without departing from the technical concept of the solution of this application, the use of other similar implementation methods based on the technical ideas of this application also falls within the protection scope of the embodiments of this application.

Claims

1. A method for detecting text similarity, characterized in that: include: Get at least two texts to be detected; For each of the texts, segment the text and obtain a target sequence of the text according to the segmentation result; The target element in the target sequence is used to represent the segmentation at the corresponding position in the segmentation result or at least one item of the attribute information of the segmentation; For the target sequence of each text, merging each target element with a subsequent preset number of target elements in sequence according to the order of the target elements in the target sequence to obtain at least one merged element corresponding to the target element, and obtaining a merged element sequence of the text based on all the target elements and the at least one merged element; Encoding the merged element sequence of each text, and obtaining a frequency vector for each text based on the encoding result of the merged element sequence of each text; the features of each dimension in the frequency vector are used to represent the frequency of the corresponding element in the merged element sequence of the text in the total merged sequence, the total merged sequence being obtained by sorting the target element and the merged element in the merged element sequences of all texts, and the total merged sequence does not contain repeated elements; For any two texts, obtaining the text similarity of the two texts according to the frequency vectors of the two texts; Wherein, obtaining the target sequence of the text according to the word segmentation result includes: Obtaining a word segmentation sequence of the text according to the word segmentation result; Determining the type of each segmentation in the segmentation sequence; Obtaining a word type sequence of the text according to the type of each word segment, wherein each element in the word type sequence is used to represent the type of the word segment at a corresponding position in the word segment sequence; According to the word type sequence of the text, a target sequence of the text is obtained, and the attribute information of the word segmentation includes the type of the word segmentation.

2. The method according to claim 1, characterized in that The step of obtaining a word segmentation sequence of the text according to the word segmentation result includes: Obtaining an initial word segmentation sequence of the text according to the word segmentation result; If there are no consecutive segmented words of the first target type in the initial segmented word sequence, determining the initial segmented word sequence as a segmented word sequence; If there are continuous segmented words of the first target type in the initial segmented word sequence, the continuous segmented words of the first target type are replaced as a whole with a preset number of preset segmented words, and the replaced initial segmented word sequence is used as the segmented word sequence.

3. The method according to claim 2, characterized in that The first target type includes at least one of a first designated type or a second designated type; If there are continuous segmented words of the first target type in the initial segmented word sequence, replacing the continuous segmented words of the first target type as a whole with a preset number of preset segmented words includes at least one of the following: If there are consecutive segmented words of the first specified type in the initial segmented word sequence, the consecutive segmented words of the first specified type are replaced with first target segmented words, where the first target segmented words are used to represent the number of consecutive segmented words of the first specified type; If there are consecutive segmented words of the second specified type in the initial segmented word sequence, the consecutive segmented words of the second specified type are replaced with second target segmented words, where the second target segmented words are a combination of the consecutive segmented words of the second specified type.

4. The method according to claim 1, wherein The step of obtaining a target sequence of the text according to the word type sequence of the text includes: For each element in the word type sequence, if the type represented by the element belongs to the second target type, then determine the order of the element among the type elements belonging to the second target type in the word type sequence; if the type represented by the element does not belong to the second target type, then determine the order of the element among the elements not belonging to the second target type in the word type sequence; The target sequence is obtained according to the order of the elements in the word type sequence, and the attribute information of the word segmentation also includes the order of the elements corresponding to the word segmentation.

5. The method according to claim 1, wherein Determining the type of each word in the word segmentation sequence includes: For each word segment in the word segmentation sequence, if the word segment belongs to multiple candidate types, then for each candidate type, the word frequency of the word segmentations belonging to the candidate type in the word segmentation sequence is determined, and the type of the word segmentation is determined based on the candidate type corresponding to the highest word frequency.

6. The method according to claim 5, characterized in that The determining the type of the word segmentation according to the candidate type corresponding to the highest word frequency includes: If the candidate type corresponding to the highest word frequency is unique, the candidate type corresponding to the highest word frequency is used as the type of the word segmentation; If the candidate type corresponding to the highest word frequency is not unique, then according to a predetermined type priority, the candidate type with the highest priority and the highest word frequency is used as the type of the word segmentation.

7. A device for detecting text similarity, characterized in that: include: A text acquisition module, used to acquire at least two texts to be detected; A target sequence acquisition module is configured to segment each of the texts and obtain a target sequence of the text according to the segmentation results; the target element in the target sequence is used to represent the segmentation at the corresponding position in the segmentation result or at least one of the attribute information of the segmentation; a merging module configured to, for each target sequence of a text, sequentially merge each target element with a subsequent preset number of target elements according to the order of the target elements in the target sequence to obtain at least one merged element corresponding to the target element, and obtain a merged element sequence of the text based on all the target elements and the at least one merged element; A frequency vector module is configured to encode the merged element sequence of each text and obtain a frequency vector for each text based on the encoding result of the merged element sequence of each text; the features of each dimension in the frequency vector are used to represent the frequency of the corresponding element in the merged element sequence of the text in the total merged sequence, wherein the total merged sequence is obtained by sorting the target element and the merged element in the merged element sequences of all texts, and there are no repeated elements in the total merged sequence; A similarity calculation module is used to obtain the text similarity between any two texts based on the frequency vectors of the any two texts; The target sequence obtaining module obtains the target sequence of the text according to the word segmentation result, including: Obtaining a word segmentation sequence of the text according to the word segmentation result; Determining the type of each segmentation in the segmentation sequence; Obtaining a word type sequence of the text according to the type of each word segment, wherein each element in the word type sequence is used to represent the type of the word segment at a corresponding position in the word segment sequence; According to the word type sequence of the text, a target sequence of the text is obtained, and the attribute information of the word segmentation includes the type of the word segmentation.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory, characterized in that: The processor executes the computer program to implement the steps of the text similarity detection method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the text similarity detection method according to any one of claims 1 to 6 are implemented.

10. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the text similarity detection method according to any one of claims 1 to 6 are implemented.

Images