Text similarity detection method, device, computer equipment and readable storage medium

Abstract

The present invention provides a text similarity detection method, apparatus, computer device, and readable storage medium. The method comprises: obtaining a text to be detected and segmenting it to obtain a word sequence; determining, from the word sequence, continuous target words located in the word position information table and whose sequence in the word sequence matches the sequence in the word position information table, based on a word position information table corresponding to the copyright text; counting the total number of words in the copyright text, the text to be detected, and the continuous target words, and calculating the difference between the total number of words in the copyright text and the continuous target words, as well as the ratio of the total number of words in the continuous target words to the text to be detected; inputting the difference in the total number of words into a preset function to obtain a similarity evaluation weight, and multiplying the similarity evaluation weight by the total number of words ratio as the similarity between the text to be detected and the copyright text. The present invention is suitable for long text similarity detection scenarios, improving detection efficiency and accuracy.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a text similarity detection method, apparatus, computer equipment, and readable storage medium. Background Art

[0002] Users can upload their own recorded audios on audio platforms, such as audiobook interpretations of novels. These uploaded audios may not have copyright, which may lead to infringement risks. In order to solve the copyright infringement problem, it is necessary to conduct online review of the sounds uploaded by users to determine whether the uploaded sounds are copyrighted.

[0003] Currently, the way to detect whether audio is copyrighted is: first perform voice recognition on the sound uploaded by the user to obtain the text to be reviewed, and then detect the similarity between the text to be reviewed and the copyright text in the copyright text library to determine whether it is infringing.

[0004] However, in the process of detecting text similarity, the existing methods may have low similarity detection accuracy due to text recognition errors or sentence segmentation errors in speech recognition. It may be impossible to distinguish the similarity between the same text to be reviewed and multiple copyright texts. In addition, the existing detection method cannot quickly determine the similarity between long texts, resulting in low efficiency in long text detection. Summary of the Invention

[0005] One of the objectives of the present invention is to provide a text similarity detection method, apparatus, computer device, and readable storage medium to improve the efficiency and accuracy of long text detection. The technical solution of the present invention can be implemented as follows:

[0006] In a first aspect, the present invention provides a method for detecting text similarity, the method comprising:

[0007] Obtaining a text to be detected, and performing word segmentation on the text to be detected to obtain a word sequence;

[0008] determining, from the word sequence, based on a word position information table corresponding to the copyright text, continuous target words that are located in the word position information table and whose sequence order in the word sequence matches the sequence order in the word position information table; the continuous target words matching the copyright text;

[0009] Counting the total number of words in the copyright text, the text to be detected, and the continuous target words, and calculating the difference between the total number of words in the copyright text and the continuous target words, and the ratio of the total number of words in the continuous target words to the text to be detected;

[0010] The difference in the total number of words is input into a preset function to obtain a similarity evaluation weight, and the product of the similarity evaluation weight and the total number of words ratio is used as the similarity between the text to be detected and the copyright text; the similarity is used to determine whether the text to be detected is a copyright text.

[0011] In a second aspect, the present invention provides a text similarity detection device, comprising: an acquisition module, a determination module and a detection module;

[0012] The acquisition module is used to acquire the text to be detected and perform word segmentation on the text to be detected to obtain a word sequence;

[0013] The determining module is configured to determine, from the word sequence, continuous target words located in the word position information table corresponding to the copyright text, and whose sequence order in the word sequence matches the sequence order in the word position information table; the continuous target words match the copyright text;

[0014] The determination module is further configured to count the total number of words in the copyright text, the text to be detected, and the continuous target words, and calculate the difference between the total number of words in the copyright text and the total number of words in the continuous target words, and the ratio of the total number of words in the continuous target words to the total number of words in the text to be detected;

[0015] The detection module is used to substitute the difference in the total number of words into a preset function to obtain a similarity evaluation weight, and use the product of the similarity evaluation weight and the ratio of the total number of words as the similarity between the text to be detected and the copyright text; the similarity is used to determine whether the text to be detected is a copyright text.

[0016] In a third aspect, the present invention provides a computer device comprising a processor and a memory, wherein the memory stores a computer program executable by the processor, and the processor can execute the computer program to implement the text similarity detection method as described in the first aspect.

[0017] In a fourth aspect, the present invention provides a readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the text similarity detection method as described in the first aspect.

[0018] The present invention provides a text similarity detection method, apparatus, computer device, and readable storage medium. First, a text to be detected is segmented to obtain a word sequence. Since an embodiment of the present invention pre-establishes a word position information table corresponding to each copyrighted text to maintain word position information of each word contained in the copyrighted text, combined with this word position information table, the present embodiment can determine words in the word sequence that plagiarize the copyrighted text, namely, continuous target words located in the word position information table whose sequence order in the word sequence matches the sequence order in the word position information table. The method then calculates the total number of words in the copyrighted text, the text to be detected, and the continuous target words, and calculates the difference between the total number of words in the copyrighted text and the continuous target words, as well as the ratio of the total number of words in the continuous target words to the text to be detected. Then, a similarity evaluation weight between the text to be detected and the copyrighted text is calculated based on the difference between the total number of words in the plagiarized content and the total number of words in the copyrighted text. Finally, similarity is calculated based on the similarity evaluation weight and the ratio of the total number of words in the continuous target words to the text to be detected. The entire method is applicable to long text similarity detection scenarios, improving the efficiency and accuracy of long text detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments. It should be understood that the following drawings only illustrate certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other relevant drawings can be obtained based on these drawings without paying any creative work.

[0020] Figure 1 An example diagram of a word position information table with an inverted word position structure provided by an embodiment of the present invention;

[0021] Figure 2 A schematic flow chart of a text similarity detection method provided by an embodiment of the present invention;

[0022] Figure 3 This is a schematic flowchart of step S203 provided in an embodiment of the present application;

[0023] Figure 4 A schematic diagram of determining whether a word is a continuous target word provided by an embodiment of the present invention;

[0024] Figure 5 A functional module diagram of a text similarity detection device provided by an embodiment of the present invention;

[0025] Figure 6 This is a structural block diagram of a computer device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0026] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions of the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings herein can be arranged and designed in various different configurations.

[0027] Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the invention as claimed, but rather merely represents selected embodiments of the present invention. All other embodiments derived by persons of ordinary skill in the art based on the embodiments of the present invention without creative effort shall fall within the scope of protection of the present invention.

[0028] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, it does not need to be further defined or explained in subsequent drawings.

[0029] In the description of the present invention, it should be noted that if the terms "upper", "lower", "inside", "outside", etc. appear, the orientation or position relationship indicated is based on the orientation or position relationship shown in the accompanying drawings, or is the orientation or position relationship in which the product of the invention is usually placed when in use. It is only for the convenience of describing the present invention and simplifying the description, and does not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation. Therefore, it should not be understood as a limitation on the present invention.

[0030] In addition, the terms "first", "second", etc., if used, are merely used to distinguish and describe, and should not be understood as indicating or implying relative importance.

[0031] It should be noted that, in the absence of conflict, the features in the embodiments of the present invention may be combined with each other.

[0032] Many user-uploaded audio files are not copyrighted, such as audiobook interpretations of novels, which poses a risk of copyright infringement. To address this issue, a system is needed that can perform voice recognition on uploaded audio files to generate a pre-approved text. This pre-approved text can then be used to determine whether the uploaded audio file is copyrighted. The core of this system is a long text similarity matching algorithm, which compares the pre-approved text with copyrighted text in a manually compiled copyright database to determine whether the user's uploaded audio file is plagiarized.

[0033] Among them, the difficulties of the long text similarity matching algorithm are as follows:

[0034] 1. The text converted through speech recognition may contain text recognition errors or sentence segmentation errors, resulting in inaccurate similarity detection results, making it impossible to accurately determine whether there is infringement.

[0035] 2. The audio uploaded by the user may only be partially plagiarized, or the order of the plagiarized paragraphs may be reversed, which can easily lead to misjudgment.

[0036] 3. The existing long text similarity matching method is relatively complex, resulting in very low detection efficiency.

[0037] 4. If a text to be reviewed is completely included in multiple copyrighted texts, existing methods cannot widen the similarity gap.

[0038] In order to solve the above-mentioned technical pain points, an embodiment of the present application provides a text similarity detection method, which is suitable for long text similarity detection scenarios. It can be used to quickly and accurately detect whether user-uploaded resources, such as text, audio, video, and any other resource that can be converted into text, are copyrighted resources to avoid infringement incidents.

[0039] Before introducing the text similarity detection method provided by the embodiments of the present invention, let's first briefly introduce the inverted index algorithm. Search engines use the inverted index algorithm to quickly retrieve results from trillions of data points. The underlying principle is to first segment the text of each document, create a sequential vocabulary for all the words in the document, and then mark the IDs of all documents where each word appears. Finally, when searching, one simply segments the query text, locates each word in the inverted index vocabulary, obtains the IDs of all the documents corresponding to each word, and merges the document IDs to find the corresponding target document.

[0040] The text similarity detection method provided by the embodiment of the present invention draws on the principle of inverted index. The difference is that the inverted index is an inverted structure of words and document IDs first established, while the embodiment of the present invention will establish an inverted structure of words and word positions contained in each copyright text in the copyright library, that is, a word position information table. For example, if a copyright text is "Phoenix painted on the pink wall, red phoenix, pink phoenix, pink phoenix, flower phoenix", then the word position information table pre-established by the embodiment of the present invention can be as follows: Figure 1 As shown, Figure 1 This is an example diagram of a word position information table with a single-word position inverted structure provided by an embodiment of the present invention.

[0041] In combination with the above-mentioned word position information table, the text similarity method provided by the embodiment of the present invention is to detect which content in the text to be reviewed is plagiarized from the copyright text, and the plagiarized content has two characteristics: first, some words in the text to be reviewed are located in the word position information table, and second, these words are continuous, that is, the word position relationship of these words in the text to be reviewed is consistent with the word position relationship in the copyright text. Therefore, the text detection method of the embodiment of the present invention, after multiple sliding scans, first marks the continuously plagiarized words, and then calculates the similarity between the text to be reviewed and the copyright text based on the continuous target words.

[0042] The purpose of setting up the word position information table with an inverted structure in the embodiment of the present application is twofold. One is to solve the problem of detection speed and improve detection efficiency. Assuming that the document length is m and the query length is n, the detection time complexity of the existing technology is O(m*n), while the detection time complexity of the embodiment of the present invention is O(n)); the other is to solve the problem of paragraph plagiarism. Multiple scans can solve the problem of misrecognition of some words in the speech recognition process.

[0043] Combined with the pre-established word position information table, see Figure 2 , Figure 2 This is a schematic flow chart of a text similarity detection method provided in an embodiment of the present application. The method can be applied to a computer device and includes the following steps:

[0044] S201: Obtain a text to be detected, and segment the text to be detected to obtain a word sequence.

[0045] In the embodiment of the present application, the single-word segmentation method is used for text segmentation. The principle of single-word segmentation is that each Chinese character is a word, and each English word is a word. The segmentation results of this segmentation method will directly remove characters such as spaces, symbols, and carriage returns. The words mentioned subsequently are all single-word segmentation results.

[0046] For example, the text to be detected is "tongue twister: red phoenix, pink phoenix, pink phoenix, flower phoenix", and the word sequence obtained by segmenting it into single words is: tongue twister red phoenix pink phoenix pink phoenix flower phoenix.

[0047] In an embodiment of the present invention, the text to be detected can be a long text, which is the content of a document uploaded by a user, or a text converted from audio and video uploaded by a user through voice recognition. Therefore, in an optional implementation, the implementation method of the above step S201 can be: obtain the audio uploaded by the user, perform voice recognition on the audio, obtain the text to be detected, perform single-word segmentation on the text to be detected, and obtain a word sequence.

[0048] S203: According to the word position information table corresponding to the copyright text, determine from the word sequence the continuous target words that are located in the word position information table and whose sequence order in the word sequence matches the sequence order in the word position information table.

[0049] In the embodiment of the present invention, the word position information table corresponding to the copyright text is obtained by segmenting the copyright text into individual words, such as Figure 1 As shown, therefore, the method of establishing a word position information table corresponding to a copyright text may be: obtaining each copyright text in the copyright library, performing word segmentation on each copyright text, and obtaining a copyright word set corresponding to each copyright text; counting the position of each copyright word in the word set in the copyright text, and establishing a word position information table.

[0050] The continuous target words in the embodiments of the present invention are matched with the copyright text, which can be understood as follows: the continuous target words are the words in the text to be detected that infringe the copyright text, and have the following characteristics: in one embodiment, the continuous target word and the first preset number of words after the continuous target word are both located in the word position information table, and there is an increasing ordered sequence between the continuous target word and the word positions of the preset number of words after the continuous target word. In another embodiment, the previous word of the continuous target word is also a continuous target word, and the continuous target word has a word position that is greater than the word position of the previous word and the interval between the word position of the previous word and the word position of the previous word is less than the preset interval. The subsequent embodiments of the present invention will provide a detailed introduction to the continuous target words determined in these two embodiments.

[0051] S205: Counting the total number of words in the copyright text, the text to be detected, and the continuous target words, and calculating the difference between the total number of words in the copyright text and the continuous target words, and the ratio of the total number of words in the continuous target words to the text to be detected.

[0052] S207: Input the difference in the total number of words into a preset function to obtain a similarity evaluation weight, and multiply the similarity evaluation weight by the total number of words ratio as the similarity between the text to be detected and the copyright text; the similarity is used to determine whether the text to be detected is a copyright text.

[0053] In the embodiment of the present application, it is assumed that the total number of words in the copyright text, the text to be detected, and the continuous target words are respectively recorded as docLen, queryLen, and joinLen, the difference between the total number of words in the copyright text and the continuous target words is recorded as x=docLen-joinLen, and the ratio of the total number of words in the continuous target words to the text to be detected is recorded as joinLen / queryLen. The preset function is the Math.pow() function, and the similarity evaluation weight is calculated as: w=(0.2*Math.pow(1.0008,-x)+0.8). Among them, the parameters 0.2, 1.0008, and 0.8 can be set according to actual needs and are not limited here. The final similarity expression is:

[0054] score=joinLen / queryLen*(0.2*Math.pow(1.0008,-x)+0.8).

[0055] Through the above implementation, the similarity between the text to be detected and each copyrighted text is obtained, and then the obtained similarity can be used to assist the reviewer in determining whether the text to be detected is infringing.

[0056] In the above-mentioned text similarity detection method, the text to be detected is first segmented to obtain a word sequence. Since the embodiment of the present invention pre-establishes a word position information table corresponding to each copyright text to maintain the word position information of each word contained in the copyright text, combined with this word position information table, the embodiment of the present invention can determine the words in the word sequence that plagiarize the copyright text, that is, the continuous target words located in the word position information table, whose sequence order in the word sequence matches the sequence order in the word position information table. The total number of words in the copyright text, the text to be detected, and the continuous target words is counted, and the difference between the total number of words in the copyright text and the continuous target words, as well as the ratio of the total number of words in the continuous target words to the text to be detected are calculated. Then, the difference between the total number of words in the plagiarized content and the total number of words in the copyright text is combined to calculate the similarity evaluation weight between the text to be detected and the copyright text. Finally, the similarity is calculated based on the similarity evaluation weight and the ratio of the total number of words in the continuous target words to the text to be detected. The entire method is suitable for long text similarity detection scenarios and improves the efficiency and accuracy of long text detection.

[0057] The following describes step S202 in detail in the embodiment of the present invention.

[0058] In an optional embodiment, the embodiment of the above step S203 can be referred to Figure 3 , Figure 3 The schematic flow chart of step S203 provided in the embodiment of the present application may include the following steps:

[0059] S203-1: If the current word and the first preset number of words after the current word are both in the word position information table, determine whether the current word is a continuous target word based on the word position relationship between the current word and the preset number of words; the current word is any one in the word sequence.

[0060] The continuous target words determined by the above step S203-1 have the following characteristics: the continuous target word and the first preset number of words after the continuous target word are both located in the word position information table, and there is an increasing ordered sequence between the word positions of the continuous target word and the preset number of words after the continuous target word.

[0061] For example, continue with the tongue twister "Red Phoenix Pink Phoenix Pink Phoenix Flower Phoenix" and Figure 1 Taking the word position information table shown as an example, we can first determine that the three words "tongue twister" are not in the word position information table. Assuming that the current word is "red", it is determined through matching that it is located in the word position information table. Assuming that the first preset number is 6, then determine whether there are at least 6 words in the word position information table after "red", and whether there is an increasing ordered queue between the word positions of these 6 or more words and the word position of "red".

[0062] See also Figure 4 , Figure 4 The schematic diagram provided in the embodiment of the present invention for determining whether one of the words is a continuous target word shows that after "red", the seven words "phoenix pink phoenix" are all located in the word position information table, and an increasing ordered sequence can be obtained from the word position in each word box: 7, 9, 10, 12, 13, 14, 15, 16, so it can be determined that "red" is a continuous target word.

[0063] In the process of implementing the above step S203-1, the word sequence may be scanned multiple times through a sliding window of a fixed length to determine continuous target words. That is, the above step S202-1 may be performed as follows:

[0064] Step a1: Scan from the beginning of the word sequence according to a sliding window of a preset length, and take the first word in the sliding window as the current word.

[0065] In an optional embodiment, the preset length can represent the number of words scanned in a single sliding window. The preset length can be customized according to the actual length of the text to be detected, such as 10, that is, 10 words in the word sequence are scanned each time. The subsequent first preset number can be determined based on the length of the sliding window. The first preset number is less than the preset length. For example, if the preset length is 10, then the first preset number can be a value less than 10, such as 6.

[0066] Step a2: Determine whether the current word is in the word position information table;

[0067] Step a3: If not, move the sliding window to the next word position of the current word, and return to execute the step of determining whether the current word is in the word position information table until the current word is in the word position information table;

[0068] Step a4: If so, determine whether there are at least a first preset number of words in the sliding window that are in the word position information table, and there is an increasing ordered sequence between the word position of the current word and the word positions of the first preset number of words.

[0069] Step a5: If not, determine that the current word is a non - consecutive target word, and return to execute the step of moving the sliding window to the next word position of the current word;

[0070] Step a6: If so, determine that the current word is a consecutive target word.

[0071] For example, continuing with the previous example, assume that the length of the sliding window is 10 and the first preset number is 6. Starting from the beginning position of the word sequence, the scanning result is [tongue twister red phoenix pink phoenix pink] red phoenix flower phoenix, where [] represents the sliding window. It can be seen that the first word "rao" is not in the word position information table, so the sliding window is moved backward to become rao [tongue twister red phoenix pink phoenix pink] phoenix flower phoenix, and so on until the moved sliding window becomes tongue twister [red phoenix pink phoenix pink phoenix] flower phoenix. At this time, the first word in the sliding window is "red", which is in the word position information table.紧接着,该滑动窗口内位于“红”之后的词依次为“风凰粉风凰粉红凤凰”,其中有7个词位于词位置信息表中,如 Figure 3 As shown, there is an increasing ordered sequence between the word positions of these 7 words and the word position of "red", so "red" is a consecutive target word. <​​​​​​In the process of implementing the above step S203-2, it can be carried out as follows:

[0075] Step b1: After determining that the current word is a continuous target word, determine whether the next word after the current word meets the preset conditions; wherein, the preset conditions are: the next word has a word position greater than the word position of the current word and the interval between it and the word position of the current word is less than the preset interval.

[0076] In the embodiment of the present application, the preset interval can be defined by oneself, for example, the preset interval is 3.

[0077] Step b2: If satisfied, determine that the next word is a continuous target word, and take the next word as the current word, and return to execute the step of determining whether the next word after the current word meets the preset conditions, until the next word does not meet the preset conditions, or the next word is the last word in the word sequence.

[0078] Step b3: If not satisfied, move the sliding window to the position of the next word, and take the next word as the current word, and return to the step of determining whether the current word is located in the word position information table.

[0079] For example, assuming the preset interval is 3 and we know that "red" is a continuous target word, we can directly check whether the next word "wind" is a continuous word. "Wind" is not in the inverted index and can be skipped. Continuing to check "phoenix," we know that "phoenix" has word positions [6,9,12,16,19] in the word position information table. There is a position 12 that is greater than the position 9 of "red" and the interval between it and the position is less than 3. Therefore, "phoenix" can be considered a continuous target word. Repeat the above method until the next word is not a continuous target word, and then execute steps a1 to a6.

[0080] Through the implementation method, the continuous target words finally determined in the word sequence "tongue twister red phoenix pink phoenix pink red phoenix flower phoenix" are "red", "phoenix", "pink", "phoenix", "pink", "red", "phoenix", "phoenix", "flower", "phoenix" and "phoenix".

[0081] It should be noted that step S203-2 is executed only when a continuous target word is determined in step S203-1. That is to say, assuming that step S203-1 determines that a certain word is a continuous target word, then step S203-2 can be used to determine whether the next word after this word is a continuous word. If it is, step S203-2 is continued to be used to determine the subsequent word. If not, return to step S203-2 for judgment, and repeat this process until all continuous target words are determined.

[0082] In an alternative embodiment, since the text to be detected may be converted from speech recognition and there may be recognition errors, resulting in misjudgment of consecutive target words. To solve this problem, an embodiment provided by the present invention includes the following steps before step S205:

[0083] For non-consecutive target words, if there are consecutive target words within the second preset number of words before and after the non-consecutive target word, update the non-consecutive target word to a consecutive target word.

[0084] That is to say, the initial scan of the word sequence can determine each consecutive target word, and the remaining words are non-consecutive target words. At this time, a secondary scan can be performed on the word sequence. For each non-consecutive target word, if there are consecutive target words within the second preset number (such as 3) of characters on both sides of a non-consecutive word, update this non-consecutive target word to a consecutive target word, which can solve the problem of speech recognition.

[0085] For example, continuing with the above example, through the above embodiment, it is determined that "wind" is a non-consecutive target word. Within 3 characters before and after "wind", there are consecutive target words "red" and "phoenix", "pink" respectively. Then it can be determined that "wind" is actually a misrecognized word, so the present invention also determines it as a consecutive target word.

[0086] The consecutive target words finally determined in the final word sequence "tongue twister red wind phoenix pink wind phoenix pink red phoenix flower phoenix" are "red", "wind", "phoenix", "pink", "wind", "phoenix", "pink", "red", "phoenix", "flower", "phoenix" and "phoenix".

[0087] In summary, the text similarity detection method provided by the embodiments of the present invention is applicable to the long text copyright detection scenario. Compared with the ordinary long text matching method, the embodiments of the present invention are faster and can solve paragraph plagiarism, disordered plagiarism, and compatibility with similarity matching when there are partial text errors.

[0088] Based on the same inventive concept, the embodiments of the present invention also provide a text similarity detection device for performing the corresponding steps in the above method embodiments and various possible embodiments. The following gives an implementation manner of the text similarity detection device. Please refer to Figure 5 , Figure 5 FIG. shows a block diagram of the text similarity detection device provided by the embodiments of the present invention. The text similarity detection device 500 includes: an acquisition module 510, a determination module 520, and a detection module 530;

[0089] The acquisition module 510 is configured to acquire the text to be detected and perform word segmentation on the text to be detected to obtain a word sequence;

[0090] A determination module 520 is configured to determine, from the word sequence, continuous target words in the word position information table corresponding to the copyright text, whose sequence order in the word sequence matches the sequence order in the word position information table; the continuous target words are matched with the copyright text;

[0091] The determination module 520 is further configured to count the total number of words in the copyright text, the text to be detected, and the continuous target words, and calculate the difference between the total number of words in the copyright text and the continuous target words, and the ratio of the total number of words in the continuous target words to the text to be detected;

[0092] Detection module 530 is used to input the difference in the total number of words into a preset function to obtain a similarity evaluation weight, and use the product of the similarity evaluation weight and the total number of words ratio as the similarity between the text to be detected and the copyrighted text; the similarity is used to determine whether the text to be detected is infringing.

[0093] In an optional embodiment, the determination module 520 is specifically used to: if the current word and the first preset number of words after the current word are both located in the word position information table, then determine whether the current word is a continuous target word based on the word position relationship between the current word and the preset number of words; the current word is any one in the word sequence; if it is a continuous target word, then if the next word of the current word is located in the word position information table, then determine whether the next word of the current word is a continuous target word based on the size relationship between the word position of the next word and the word position of the current word.

[0094] In an optional embodiment, the determination module 520 is specifically used to: scan from the starting position of the word sequence according to a sliding window of a preset length, and take the first word in the sliding window as the current word; determine whether the current word is in the word position information table; if not, move the sliding window to the next word position of the current word, and return to execute the step of determining whether the current word is in the word position information table until the current word is in the word position information table; if so, determine whether there are at least a first preset number of words in the word position information table in the sliding window, and there is an increasing ordered sequence between the word position of the current word and the word positions of the first preset number of words; if not, determine that the current word is a non-continuous target word, and return to execute the step of moving the sliding window to the next word position of the current word; if so, determine that the current word is a continuous target word.

[0095] In an optional embodiment, the determination module 520 is specifically configured to: determine whether the next word after the current word meets a preset condition; wherein the preset condition is: the next word has a word position greater than the word position of the current word and the interval between the next word and the word position of the current word is less than a preset interval; if so, determine that the next word is a continuous target word, set the next word as the current word, and return to the step of determining whether the next word after the current word meets the preset condition until the next word does not meet the preset condition, or the next word is the last word in the word sequence. If not, move the sliding window to the position of the next word, set the next word as the current word, and return to the step of determining whether the current word is located in the word position information table.

[0096] In an optional embodiment, the determination module 520 is further configured to: for a non-continuous target word, if there is a continuous target word within a second preset number of words before and after the non-continuous target word, update the non-continuous target word to a continuous target word.

[0097] In an optional embodiment, the text similarity detection device 500 further includes an establishment module for obtaining each copyright text in the copyright library, performing word segmentation on each copyright text, and obtaining a set of copyright words corresponding to each copyright text; and counting the position of each copyright word in the word set in the copyright text to establish a word position information table.

[0098] In an optional embodiment, the acquisition module 520 is specifically used to: obtain the audio uploaded by the user, perform speech recognition on the audio to obtain the text to be detected; and perform word segmentation on the text to be detected to obtain a word sequence.

[0099] It should be noted that the module division in the above embodiments of the present invention is illustrative and merely represents a logical functional division. In actual implementation, other division methods may be employed. Furthermore, the functional units in various embodiments of the present invention may be integrated into a single processing unit, exist as separate physical units, or be integrated into a single unit. The aforementioned integrated units may be implemented in either hardware or software functional units.

[0100] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a readable storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) or a processor to execute all or part of the steps of the various embodiments of the present invention. The aforementioned readable storage medium includes various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0101] The embodiment of the present invention also provides a computer device for executing the text similarity detection method in the embodiment of the present invention, please refer to Figure 6 , Figure 6 The block diagram of a computer device provided by an embodiment of the present invention is shown. The computer device 600 can be a server, a personal computer, an edge gateway, etc., and includes a processor 601, a memory 602, a bus 604, and a communication interface 603. The processor 601 is connected to the memory 602 via the bus 604.

[0102] Optionally, bus 604 may be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus. Buses may be divided into address buses, data buses, control buses, etc. For ease of representation, Figure 6 Only one thick line is used in the diagram, but this does not mean that there is only one bus or one type of bus.

[0103] In the embodiment of the present application, the processor 602 can be a general-purpose processor, a digital signal processor, an application-specific integrated circuit, a field programmable gate array or other programmable logic device, a discrete gate or transistor logic device, or a discrete hardware component, and can implement or execute the various methods, steps, and logic block diagrams disclosed in the embodiment of the present application. The general-purpose processor can be a microprocessor or any conventional processor, etc. The steps of the method disclosed in the embodiment of the present application can be directly embodied as being executed by a hardware processor, or can be executed by a combination of hardware and software modules in the processor. The software module can be located in the memory 601, and the processor 602 reads the program instructions in the memory 601 and completes the steps of the above method in combination with its hardware.

[0104] In the embodiment of the present application, the memory 601 may be a non-volatile memory, such as a hard disk drive (HDD) or a solid-state drive (SSD), or a volatile memory (volatile memory), such as RAM. The memory may also be any other medium that can be used to carry or store desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto. The memory in the embodiment of the present application may also be a circuit or any other device that can implement a storage function, for storing instructions and / or data.

[0105] The memory 601 can be used to store software programs and modules, such as the instructions / modules of the text similarity detection device 500 provided in an embodiment of the present invention, which can be stored in the memory 601 in the form of software or firmware or fixed in the operating system (OS) of the computer device 600. The processor 602 executes the software programs and modules stored in the memory 601 to perform various functional applications and data processing. The communication interface 603 can be used to communicate signaling or data with other node devices.

[0106] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described devices and units can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0107] I understand. Figure 6 The structure shown is for illustration only. The computer device 600 may also include Figure 6 More or fewer components than shown, or with Figure 6 Different configurations shown. Figure 6 The components shown may be implemented in hardware, software, or a combination thereof.

[0108] Based on the above embodiments, the present invention further provides a readable storage medium, in which a computer program is stored. When the computer program is executed by a computer, the computer executes the text similarity detection method provided in the above embodiments.

[0109] Based on the above embodiments, an embodiment of the present invention further provides a computer program. When the computer program is run on a computer, the computer executes the text similarity detection method provided in the above embodiments.

[0110] Based on the above embodiments, an embodiment of the present invention further provides a chip, which is used to read a computer program stored in a memory and to execute the text similarity detection method provided in the above embodiments.

[0111] An embodiment of the present invention further provides a computer program product, comprising instructions, which, when executed on a computer, enables the computer to execute the text similarity detection method provided in the above embodiment.

[0112] The embodiments of the present invention are described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of the processes and / or blocks in the flowchart and / or block diagram, can be implemented by instructions. These instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the process in the flowchart and / or block diagram. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0113] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0114] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0115] The above are merely specific embodiments of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in the present invention should be included in the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be based on the scope of protection of the claims.

Claims

1. A text similarity detection method, characterized in that: The method comprises: Obtaining a text to be detected, and performing word segmentation on the text to be detected to obtain a word sequence; According to the word position information table corresponding to the copyright text, from the word sequence, determining continuous target words that are located in the word position information table and whose sequence order in the word sequence matches the sequence order in the word position information table, including: if the current word and a first preset number of words following the current word are both located in the word position information table, then determining whether the current word is the continuous target word based on the word position relationship between the current word and the first preset number of words; the current word is any one in the word sequence; if it is the continuous target word, then if the next word of the current word is located in the word position information table, then determining whether the next word of the current word is the continuous target word based on the size relationship between the word position of the next word and the word position of the current word; wherein the continuous target words match the copyright text; the size relationship is: the next word has a word position greater than the word position of the current word and the interval between the next word and the word position of the current word is less than a preset interval; Counting the total number of words in the copyright text, the text to be detected, and the continuous target words, and calculating the difference between the total number of words in the copyright text and the continuous target words, and the ratio of the total number of words in the continuous target words to the text to be detected; The difference in the total number of words is input into a preset function to obtain a similarity evaluation weight, and the product of the similarity evaluation weight and the total number of words ratio is used as the similarity between the text to be detected and the copyright text; the similarity is used to determine whether the text to be detected is a copyright text.

2. The text similarity detection method according to claim 1, characterized in that: If the current word and a first preset number of words following the current word are both located in the word position information table, determining whether the current word is the continuous target word based on the word position relationship between the current word and the first preset number of words includes: Scanning from the starting position of the word sequence according to a sliding window of a preset length, and taking the first word in the sliding window as the current word; Determining whether the current word is located in the word position information table; If not, the sliding window is moved to the next word position of the current word, and the step of determining whether the current word is located in the word position information table is returned to be executed until the current word is located in the word position information table; If so, determining whether there are at least a first preset number of words in the word position information table within the sliding window, and whether there is an increasing ordered sequence between the word position of the current word and the word positions of the first preset number of words; If not, the current word is determined to be a non-continuous target word, and the process returns to the step of moving the sliding window to the next word position of the current word; If so, the current word is determined to be the continuous target word.

3. The text similarity detection method according to claim 2, characterized in that: If it is the continuous target word, then if the next word of the current word is located in the word position information table, determining whether the next word of the current word is the continuous target word based on the size relationship between the word position of the next word and the word position of the current word includes: Determine whether the next word after the current word satisfies the size relationship; If so, determining that the next word is the continuous target word, and taking the next word as the current word, returning to the step of determining whether the next word after the current word satisfies the size relationship, until the next word does not satisfy the size relationship, or the next word is the last word in the word sequence; If not satisfied, the sliding window is moved to the position of the next word, and the next word is used as the current word, and the process returns to the step of determining whether the current word is located in the word position information table.

4. The text similarity detection method according to claim 1, characterized in that: The method further comprises: For a non-continuous target word, if the continuous target word exists within a second preset number of words located before and after the non-continuous target word, the non-continuous target word is updated to a continuous target word.

5. The text similarity detection method according to claim 1, characterized in that: Before obtaining the text to be detected and performing word segmentation on the text to be detected to obtain a word sequence, the method further includes: Obtaining each copyright text in the copyright library, performing word segmentation on each copyright text, and obtaining a set of copyright terms corresponding to each copyright text; The position of each copyright word in the copyright word set in the copyright text is counted to establish the word position information table.

6. The text similarity detection method according to claim 1, characterized in that: Obtain the text to be detected and perform word segmentation on the text to be detected to obtain a word sequence, including: Obtaining the audio uploaded by the user, performing speech recognition on the audio, and obtaining the text to be detected; Perform word segmentation on the text to be detected to obtain the word sequence.

7. A text similarity detection device, characterized in that: include: Acquire modules, determine modules and detect modules; The acquisition module is used to acquire the text to be detected and perform word segmentation on the text to be detected to obtain a word sequence; The determination module is configured to determine, from the word sequence, a continuous target word located in the word position information table corresponding to the copyright text, and whose sequence order in the word sequence matches the sequence order in the word position information table, including: if a current word and a first preset number of words following the current word are both located in the word position information table, then determining whether the current word is the continuous target word based on the word position relationship between the current word and the first preset number of words; the current word is any one in the word sequence; if it is the continuous target word, then if the next word of the current word is located in the word position information table, then determining whether the next word of the current word is the continuous target word based on the size relationship between the word position of the next word and the word position of the current word; wherein the continuous target word matches the copyright text; the size relationship is: the next word has a word position greater than the word position of the current word and the interval between the next word and the word position of the current word is less than a preset interval; The determination module is further configured to count the total number of words in the copyright text, the text to be detected, and the continuous target words, and calculate the difference between the total number of words in the copyright text and the total number of words in the continuous target words, and the ratio of the total number of words in the continuous target words to the total number of words in the text to be detected; The detection module is used to substitute the difference in the total number of words into a preset function to obtain a similarity evaluation weight, and use the product of the similarity evaluation weight and the ratio of the total number of words as the similarity between the text to be detected and the copyright text; the similarity is used to determine whether the text to be detected is a copyright text.

8. A computer device, characterized in that: The method comprises a processor and a memory, wherein the memory stores a computer program executable by the processor, and the processor can execute the computer program to implement the text similarity detection method according to any one of claims 1 to 6.

9. A readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the text similarity detection method according to any one of claims 1 to 6 is implemented.

Images