Text similarity detection method, device and system

Abstract

The invention relates to the field of text processing, and discloses a text similarity detection method, device and system, and the method comprises the steps: obtaining a to-be-detected text, a comparison text and a preset comparison length value; segmenting characters in the to-be-detected text into a plurality of to-be-detected character segments; segmenting characters in the comparison text into a plurality of to-be-compared character fields based on a preset comparison length value and the to-be-detected character fields; comparing each to-be-detected character field with all to-be-compared character fields one by one, and when the editing distance between the to-be-detected character field and the compared to-be-compared character field is within a preset editing distance, combining the to-be-detected character field and the compared to-be-compared character field to form a similar character comparison group and comparing; and calculating the number of repeated characters and the number of continuous repeated characters in the to-be-detected character field so as to obtain the text similarity between the repeated characters and the continuous repeated characters. According to the method provided by the invention, the similar text position of the to-be-detected text in the comparison text can be efficiently and accurately positioned, and the text similarity of the to-be-detected text and the comparison text can be detected.

Description

Technical Field

[0001] The present application relates to the field of text processing technology, and in particular to a text similarity detection method, device and system. Background Art

[0002] For application systems that use text elements to calculate manuscript fees, it is necessary to accurately determine the similarity between the manuscript content uploaded by the user and the text data that has been published in the system. Usually, the content of the plagiarized article will not be exactly the same as the plagiarized article, but will select a part of it or simply modify some of the text to evade system detection.

[0003] Among the existing text similarity detection methods, the SimHash algorithm is not good at processing short texts, and the MD5 algorithm is difficult to avoid the impact of punctuation on the detection results. If conventional methods are used, such as using a vocabulary to segment the text and process stop words or pre-set weights, if the vocabulary is not updated in time or lacks vocabulary in the corresponding field, it will affect the detection accuracy, and the complex weight pre-setting and stop word processing will increase the complexity of the calculation; and if the two texts are directly compared one by one, this detection method is not only very slow, but also difficult to detect whether it is plagiarism if the plagiarized content has been modified. Summary of the invention

[0004] The purpose of the embodiments of the present application is to provide a text similarity detection method, device and system to solve the above-mentioned problems.

[0005] In order to achieve the above-mentioned purpose, the first aspect of the present application provides a text similarity detection method, which includes: obtaining a text to be detected, a comparison text and a preset comparison length value; dividing the text in the text to be detected into multiple text segments to be detected; based on the preset comparison length value and the text segments to be detected, dividing the text in the comparison text into multiple text segments to be compared; comparing each text segment to be detected with all text segments to be compared one by one, and when the edit distance between the text segment to be detected and the compared text segment to be compared is within the preset edit distance, combining the text segment to be detected and the compared text segment to be compared to form a similar text comparison group; comparing the text segments to be detected in all similar text comparison groups with the corresponding text segments to be compared, and calculating the number of repeated characters and the number of consecutive repeated characters in the text segment to be detected; and calculating the similarity between the text to be detected and the comparison text based on the number of repeated characters and the number of consecutive repeated characters in all text segments to be detected in the text to be detected.

[0006] In an embodiment of the present application, the step of dividing the text to be detected into multiple text segments to be detected includes: preprocessing the text to be detected to extract the text to be detected; obtaining comparison units and symbol units in the text to be detected, wherein the symbol unit is smaller than the comparison unit; based on the comparison unit and the symbol unit, dividing the text to be detected into multiple text segments to be detected with the comparison unit and the symbol unit as intervals.

[0007] In an embodiment of the present application, based on a preset comparison length value and a text segment to be detected, the step of dividing the text in the comparison text into multiple text segments to be compared includes: preprocessing the comparison text to extract the text in the comparison text; screening out the shortest text segment to be detected among all the text segments to be detected, the length of the shortest text segment to be detected being the minimum text length value; comparing the preset comparison length value with the minimum text length value, and taking the larger one as the comparison length initial value; based on the comparison length initial value, dividing the text in the comparison text into multiple text segments to be compared, the lengths of which are the same as the comparison length initial value.

[0008] In an embodiment of the present application, the method further includes: when comparing each text segment to be detected with all text segments to be compared one by one, if the edit distance between the text segment to be detected and one of the text segments to be compared is greater than the actual length of the text segment to be detected, resetting the initial value of the comparison length until the edit distance between the text segment to be detected and all the text segments to be compared is less than or equal to the actual length of the text segment to be detected.

[0009] In an embodiment of the present application, the step of resetting the preset comparison length value includes: comparing the actual length of the text segment to be detected with the preset comparison length value, updating the initial comparison length value to the larger one, and re-executing the step of dividing the text in the comparison text into multiple text segments to be compared.

[0010] In the embodiment of the present application, the comparison unit is a period-like symbol in the text, and the symbol unit is a comma-like symbol in the text.

[0011] In an embodiment of the present application, the steps of comparing the to-be-detected text segments in all similar text comparison groups with the corresponding to-be-detected text segments, and calculating the number of repeated characters and the number of continuously repeated characters in the to-be-detected text segments include: obtaining a preset displacement step; based on the preset displacement step, performing at least one position correction on the head end or the tail end of the to-be-detected text segments in all similar text comparison groups; calculating the edit distance between the to-be-detected text segment and the corresponding to-be-detected text segment after the position correction, and selecting the to-be-detected text segment corresponding to the to-be-detected text segment with the smallest edit distance as the optimal to-be-detected text segment; comparing the optimal to-be-detected text segment with the corresponding to-be-detected text segment, and calculating the number of repeated characters and the number of continuously repeated characters in the to-be-detected text segment.

[0012] According to a second aspect of the present application, there is provided a text similarity detection device, which includes: an acquisition module, which is used to acquire a text to be detected, a comparison text and a preset comparison length value; a first segmentation module, which is used to segment the text in the text to be detected into a plurality of text segments to be detected; a second segmentation module, which is used to segment the text in the comparison text into a plurality of text segments to be compared based on the preset comparison length value and the text segments to be detected; a combination module, which is used to compare each text segment to be detected with all text segments to be compared one by one, and when the edit distance between the text segment to be detected and the compared text segment to be compared is within the preset edit distance, the text segment to be detected and the compared text segment to be compared are combined to form a similar text comparison group; a comparison module, which is used to compare the text segments to be detected in all similar text comparison groups with the corresponding text segments to be compared, and calculate the number of repeated characters and the number of consecutive repeated characters in the text segment to be detected; and a calculation module, which is used to calculate the similarity between the text to be detected and the comparison text based on the number of repeated characters and the number of consecutive repeated characters in all text segments to be detected in the text to be detected.

[0013] According to a third aspect of the present application, a text similarity detection system is provided. The system includes: a preprocessing component, a comparison component, and a calculation component; the preprocessing component is used to obtain a text to be detected, a comparison text, and a preset comparison length value, and divide the text in the text to be detected into a plurality of text segments to be detected; based on the preset comparison length value and the text segment to be detected, the text in the comparison text is divided into a plurality of text segments to be compared; the comparison component is used to compare each text segment to be detected with all text segments to be compared one by one, and when the edit distance between the text segment to be detected and the compared text segment to be compared is within the preset edit distance, the text segment to be detected and the compared text segment to be compared are combined to form a similar text comparison group; the text segments to be detected in all similar text comparison groups are compared with the corresponding text segments to be compared, and the number of repeated characters and the number of consecutive repeated characters in the text segment to be detected are calculated; the calculation component is used to calculate the similarity between the text to be detected and the comparison text based on the number of repeated characters and the number of consecutive repeated characters in all text segments to be detected in the text to be detected.

[0014] A fourth aspect of the present application provides a machine-readable storage medium having instructions stored thereon, which, when executed by a processor, configures the processor to execute the text similarity detection method of any of the aforementioned embodiments of the first aspect of the present application.

[0015] A fifth aspect of the present application provides a computer program product, including a computer program, which, when executed by a processor, implements the text similarity detection method of any of the aforementioned implementations of the first aspect of the present application.

[0016] Through the above technical solution, the text to be detected and the comparison text can be divided into multiple text segments to be detected and multiple text segments to be compared respectively, and each text segment to be detected is compared with all text segments to be compared one by one and the edit distance between the text segment to be detected and the compared text segment to be compared is calculated to achieve rapid positioning and determine the text segment to be compared that is most similar to the text segment to be detected in the comparison text. The method of the embodiment of the present application is not only applicable to texts of various word counts, but also can accurately determine the similarity between the text to be detected and the comparison text without the help of a vocabulary or complex weight calculation.

[0017] Other features and advantages of the embodiments of the present application will be described in detail in the subsequent specific implementation section. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The accompanying drawings are used to provide a further understanding of the embodiments of the present application and constitute a part of the specification. Together with the following specific implementations, they are used to explain the embodiments of the present application, but do not constitute a limitation on the embodiments of the present application. In the accompanying drawings:

[0019] Figure 1 A schematic diagram of an application environment of a text similarity detection method according to an embodiment of the present application is schematically shown;

[0020] Figure 2 The following is a schematic diagram showing a flow chart of a text similarity detection method according to an embodiment of the present application;

[0021] Figure 3 The flowchart of step S120 of the text similarity detection method according to the embodiment of the present application is schematically shown;

[0022] Figure 4 The flowchart of step S130 of the text similarity detection method according to the embodiment of the present application is schematically shown;

[0023] Figure 5 The flowchart of step S150 of the text similarity detection method according to the embodiment of the present application is schematically shown;

[0024] Figure 6 The structure block diagram of the text similarity detection device according to the embodiment of the present application is schematically shown;

[0025] Figure 7 The structure block diagram of the text similarity detection system according to the embodiment of the present application is schematically shown;

[0026] Figure 8 The internal structure diagram of a computer device according to an embodiment of the present application is schematically shown. DETAILED DESCRIPTION

[0027] In order to make the purpose, technical scheme and advantages of the embodiments of the present application clearer, the technical scheme in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. It should be understood that the specific implementation methods described herein are only used to illustrate and explain the embodiments of the present application, and are not used to limit the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0028] It should be noted that if the embodiments of the present application involve directional indications (such as up, down, left, right, front, back...), the directional indications are only used to explain the relative position relationship, movement status, etc. between the components under a certain specific posture (as shown in the accompanying drawings). If the specific posture changes, the directional indication will also change accordingly.

[0029] In addition, if there are descriptions involving "first", "second", etc. in the embodiments of the present application, the descriptions of "first", "second", etc. are only used for descriptive purposes and cannot be understood as indicating or suggesting their relative importance or implicitly indicating the number of technical features indicated. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In addition, the technical solutions between the various embodiments can be combined with each other, but they must be based on the ability of ordinary technicians in the field to implement them. When the combination of technical solutions is contradictory or cannot be implemented, it should be deemed that such combination of technical solutions does not exist and is not within the scope of protection required by this application.

[0030] The acquisition, transmission, storage, use, and processing of data in the technical solution of this application are in compliance with the relevant provisions of national laws and regulations. In addition, it should be noted that in the embodiments of this application, some existing solutions in the industry such as certain software, components, and models may be mentioned, which should be considered as exemplary, and their purpose is only to illustrate the feasibility of the implementation of the technical solution of this application, but it does not mean that the applicant has or will necessarily use the solution.

[0031] The text similarity detection method provided in the embodiment of the present application can be applied to Figure 1 In the application environment shown. Among them, the terminal 101 communicates with the server 102 through the network, and the user can issue instructions to the server 102 through the terminal 101, so that the server executes the text similarity detection method provided in the embodiment of the present application, and sends the detection result to the terminal 101, so that the user can obtain the detection result. The terminal 101 can be, but is not limited to, various personal computers, laptops, smart phones, tablet computers and portable wearable devices, and the server 102 can be implemented with an independent server or a server cluster composed of multiple servers.

[0032] Figure 2 The flowchart of the text similarity detection method according to the embodiment of the present application is schematically shown. Figure 2 As shown, in one embodiment of the present application, a text similarity detection method is provided. This embodiment mainly applies this method to the above Figure 1 Taking the terminal 101 (or server 102) in FIG. 1 as an example, the text similarity detection method of this embodiment includes steps S110 to S160.

[0033] In step S110, the text to be detected, the comparison text and the preset comparison length value are obtained.

[0034] In this embodiment, the text to be detected is used to compare with the comparison text to obtain the number of repeated characters between the text content of the text to be detected and the text content of the comparison text, and then calculate the similarity between the text to be detected and the comparison text. The preset comparison length value is a preset value, and a suitable length can be selected according to the length of each text segment of the text to be detected. For example, the preset comparison length value can be any integer value such as 5, 6, 7, 10, 15 or 20.

[0035] In step S120, the characters in the text to be detected are divided into a plurality of text segments to be detected.

[0036] like Figure 3 As shown, in some embodiments, step S120 includes steps S121 to S123.

[0037] In step S121, the text to be detected is preprocessed to extract characters from the text to be detected.

[0038] In step S122, the comparison unit and the symbol unit in the text to be detected are obtained, wherein the symbol unit is smaller than the comparison unit.

[0039] In step S123, based on the comparison unit and the symbol unit, the characters in the to-be-detected text are segmented into a plurality of to-be-detected text segments separated by the comparison unit and the symbol unit.

[0040] In this embodiment, when the text to be detected is preprocessed, all the text content in the text to be detected is first obtained, and then the punctuation marks, label symbols, emoticons and other symbols irrelevant to the text are removed to extract the text content in the text to be detected, and the preprocessing process is completed. The text content can be Chinese, English or other languages.

[0041] When preprocessing the text to be detected, the characters in the text to be detected are divided into a plurality of text segments separated by symbol units and comparison units according to the positions of the symbol units and comparison units in the text to be detected, so as to obtain the text segments to be detected.

[0042] The symbol unit and the comparison unit are generally punctuation marks in the text, for example, period symbols include period, exclamation mark and question mark, and comma symbols include comma, colon and semicolon. Specifically, in this embodiment, the comparison unit is the period symbol in the text, and the symbol unit is the comma symbol in the text, that is, the text in the to-be-detected text is segmented by comma symbols and period symbols to form a plurality of to-be-detected text segments.

[0043] In step S130 , based on the preset comparison length value and the text segment to be detected, the characters in the comparison text are divided into a plurality of text segments to be compared.

[0044] like Figure 4 As shown, in some embodiments, step S130 includes steps S131 to S134.

[0045] In step S131, the comparison text is preprocessed to extract the characters in the comparison text.

[0046] In step S132, among all the text segments to be detected, the shortest text segment to be detected is screened and obtained, wherein the length of the shortest text segment to be detected is the minimum character length value.

[0047] In step S133, the preset comparison length value and the minimum character length value are compared, and the larger one is used as the comparison length initial value.

[0048] In step S134 , based on the initial comparison length value, the characters in the comparison text are divided into a plurality of to-be-compared text segments having the same length as the initial comparison length value.

[0049] In this embodiment, the preprocessing process of the comparison text is the same as the preprocessing process of the text to be detected mentioned above. The entire text content in the comparison text is first obtained, and then the punctuation marks, label symbols, emoticons and other symbols unrelated to the text are removed, and the text content in the comparison text is extracted to complete the preprocessing process.

[0050] After the preprocessing of the comparison text is completed, the preset comparison length value and the minimum text length value are compared, and the larger one is used as the comparison length initial value, and then the text of the comparison text is divided into multiple text segments to be compared with each other with a length equal to the comparison length initial value. If there are any remaining texts, they are treated as a single text segment to be compared.

[0051] Specifically, in the embodiment of the present application, for example, the text to be detected is: ; ; &nb sp; ;Finance is a social practice activity, and its purpose is to serve the real economy. Its essence is not to pursue false prosperity, but to be close to society, originate from life, and focus on problems.

[0052] The contrasting text is: New finance needs to return to its service origin. ; ; ; ;Finance is ultimately a social practice activity, and its purpose is to serve the real economy. The essence of finance is not to pursue false prosperity, but to be down-to-earth and close to society, originate from life, and focus on problems. ; ; ; ; Financial institutions face the general public, bear the worries and joys of thousands of families, and come face to face with many social problems and social pain points. Finance is like water. Only by adhering to the foundation of kindness and good governance can it take advantage of the situation to nourish all things.

[0053] After preprocessing the test text and the comparison text, the test text is obtained as follows: "Finance is a social practice activity. The purpose of finance is to serve the real economy. Its essence is not to pursue false prosperity but to be close to society and focus on problems in life."

[0054] The contrasting text is: "New finance needs to return to its origin of service. Finance is, after all, a social practice activity. The purpose of finance is to serve the real economy. The essential form of finance is not to pursue false prosperity, but to be down-to-earth and close to society, originating from life and focusing on problems. Financial institutions face the general public, carry the worries and joys of thousands of families, and face many social problems and social pain points. Finance is like water. Only by adhering to the foundation of kindness and good governance can it take advantage of the situation to nourish all things."

[0055] It can be obtained that the minimum text length value of the text to be detected is the text segment corresponding to "Finance is a social practice activity", which is 11. If the preset comparison length value is 12, the preset comparison length value is larger, the initial value of the comparison length is set to 12, and the comparison text is segmented according to the initial value of the comparison length.

[0056] The text segments to be detected obtained by the text to be detected are:

[0057] A1: Finance is a social practice;

[0058] A2: The purpose of finance is to serve the real economy;

[0059] A3: Its essence is not to pursue false prosperity;

[0060] A4: Instead, we should be close to society and focus on issues arising from daily life;

[0061] The text segments to be compared obtained by comparing the texts are:

[0062] B1: New finance needs to return to its service origin;

[0063] B2: Finance is ultimately a social practice activity;

[0064] B3: The purpose of finance is to serve the real economy;

[0065] B4: The essence of service finance is not to;

[0066] B5: Instead of pursuing false prosperity, we must strive for success;

[0067] B6: Being close to society comes from life;

[0068] B7: Focus on the problems that financial institutions are facing;

[0069] B8: The common people bear the sorrows and joys of thousands of families;

[0070] B9: Face up to many social problems and social pains;

[0071] B10: Point out the problem: Finance is like water, upholding good intentions;

[0072] B11: Only by following the principles of good governance can we take advantage of the situation;

[0073] B12: To nourish all things;

[0074] In step S140, each to-be-detected text segment is compared with all to-be-compared text segments one by one. When the edit distance between the to-be-detected text segment and the compared to-be-compared text segment is within a preset edit distance, the to-be-detected text segment and the compared to-be-compared text segment are combined to form a similar text comparison group.

[0075] In some embodiments, when each text segment to be detected is compared with all text segments to be compared one by one, if the edit distance between the text segment to be detected and one of the text segments to be compared is greater than the actual length of the text segment to be detected, the initial value of the comparison length is reset until the edit distance between the text segment to be detected and all the text segments to be compared is less than or equal to the actual length of the text segment to be detected.

[0076] In this embodiment, the preset edit distance is set to 50% of the initial value of the comparison length, so the preset edit distance is 6. In this application, the preset edit distance takes an integer value, and if it is not an integer value, the nearest integer value is taken one step further.

[0077] In the process of comparing each to-be-detected text segment with all to-be-compared text segments one by one, if the edit distance between the to-be-detected text segment and the compared to-be-compared text segment is 6 or less, the to-be-detected text segment and the compared to-be-compared text segment are combined to form a similar text comparison group.

[0078] For example, in the embodiment of the present application, the to-be-detected text segment A1 is compared one by one with the to-be-compared text segments B1 to B12, and the most similar to-be-compared text segments are screened out and combined to form a similar text comparison group.

[0079] In this embodiment, when the text segment A1 to be detected is compared with the text segment B1 to be compared, the edit distance between "Finance is a social practice activity" and "New finance needs to return to the origin of service" is 10, so B1 is eliminated. When the text segment A1 to be detected is compared with the text segment B2 to be compared, the edit distance between "Finance is a social practice activity" and B2 "Finance is ultimately a social practice activity" is 5, so B2 is retained. By analogy, the text segment B2 to be compared with the smallest edit distance from A1 is finally obtained, that is, the text segment B2 to be compared is the most similar text segment to be compared with the text segment A1 to be detected, and the two are combined to form a similar text comparison group.

[0080] Furthermore, the step of resetting the preset comparison length value includes: comparing the actual length of the text segment to be detected with the preset comparison length value, updating the initial comparison length value to the larger one, and re-executing the step of dividing the text in the comparison text into multiple text segments to be compared.

[0081] In this embodiment, when calculating the to-be-detected text segments A2, A3, and A4, since the character lengths of the to-be-detected text segments A2, A3, and A4 have changed, if the edit distance between the to-be-detected text segment and the to-be-compared text segment is greater than the actual character length of the to-be-detected text segment, it is proved that the character length of the to-be-compared text segment is greater than the actual length of the to-be-detected text segment, which does not conform to the logical rationality when calculating the edit distance. Therefore, it is necessary to reset the preset comparison length value and re-execute the step of dividing the characters in the comparison text into multiple to-be-compared text segments according to the new comparison length initial value.

[0082] Specifically, the method of comparing the to-be-detected text segments A2, A3, and A4 with the to-be-compared text segments and calculating the edit distance is the same as the method of comparing the to-be-detected text segment A1 with the to-be-compared text segments and calculating the edit distance, which will not be repeated here.

[0083] In step S150, all the text segments to be detected in the similar text comparison group are compared with the corresponding text segments to be compared, and the number of repeated characters and the number of consecutive repeated characters in the text segment to be detected are calculated.

[0084] like Figure 5 As shown, in some embodiments, step S150 includes steps S151 to S154.

[0085] In step S151, a preset displacement step length is obtained.

[0086] In step S152, based on a preset displacement step, at least one position correction is performed on the beginning or the end of the to-be-compared text segment in all similar text comparison groups.

[0087] In step S153, the edit distance between the to-be-detected text segment and the corresponding to-be-compared text segment after position correction is calculated, and the to-be-compared text segment corresponding to the to-be-detected text segment with the smallest edit distance is selected as the optimal to-be-compared text segment.

[0088] In step S154, the optimal text segment to be compared is compared with the corresponding text segment to be detected, and the number of repeated characters and the number of consecutive repeated characters in the text segment to be detected are calculated.

[0089] In this embodiment, the preset displacement step may be 1, 2, 3 or any integer value. After determining the preset displacement step, one, two or more position corrections may be performed at the beginning or end of the to-be-compared text segment, and the edit distance between the to-be-detected text segment and the to-be-compared text segment after each position correction is calculated, and the to-be-compared text segment corresponding to the to-be-detected text segment with the smallest edit distance is selected as the optimal to-be-compared text segment.

[0090] Specifically, as shown in the text segment A1 to be detected and the text segment B2 to be compared:

[0091] The edit distance between the text segment A1 to be detected: "Finance is a social practice activity" and the text segment B2 to be compared: "Finance is ultimately a social practice activity" is 5.

[0092] After a position correction is performed at the beginning of the to-be-compared text segment B2, the edit distance between the to-be-detected text segment A1: "Finance is a social practice activity" and the to-be-compared text segment B2: "Finance is ultimately a social practice activity" is 7.

[0093] After two position corrections are performed at the beginning of the to-be-compared text segment B2, the edit distance between the to-be-detected text segment A1: "Finance is a social practice activity" and the to-be-compared text segment B2: "The origin of finance is ultimately a social practice" is 9.

[0094] Finally, it can be obtained that there is no need to correct the position of the text segment B2 to be compared, that is, the current position of the text segment B2 to be compared is the optimal position, and the current text segment of the text segment B2 to be compared is the optimal text segment to be compared.

[0095] Through steps S151 to S154 of the text similarity detection method of the present application, the shortest edit distance between the text segment to be detected and the text segment to be compared can be calculated, the position of the text segment to be detected in the compared text can be quickly located, and the optimal position of the text segment to be detected in the compared text can be obtained through position correction to ensure that the similarity detection result obtained is the most accurate.

[0096] In step S160 , the similarity between the text to be detected and the comparison text is calculated based on the number of repeated characters and the number of consecutively repeated characters in all text segments to be detected in the text to be detected.

[0097] In this embodiment, after obtaining four similar text comparison groups, namely, the text segments A1 to A4 to be detected and the text segments to be compared that are most similar to the text segments A1 to A4 to be detected, the number of repeated characters and the number of continuously repeated characters in the text segments A1, A2, A3, and A4 to be detected in the text segment to be compared that is most similar to them are calculated respectively.

[0098] For example, in this embodiment, the to-be-detected text segment A1: "Finance is a social practice activity" and the to-be-compared text segment B2: "Finance is ultimately a social practice activity". The characters that appear in both text segments are: "finance", "is", "society", "practice", "activity", that is, the number of repeated characters is 9, and the characters that appear continuously in both text segments are: "finance" and "social practice activities", and the larger one is taken, that is, the number of continuously repeated characters is 6.

[0099] After obtaining the number of repeated characters and the number of consecutively repeated characters in all the text segments to be detected, the similarity between the text to be detected and the comparison text can be calculated.

[0100] The text similarity detection method provided by the embodiment of the present application includes: obtaining a text to be detected, a comparison text and a preset comparison length value; dividing the text in the text to be detected into a plurality of text segments to be detected; dividing the text in the comparison text into a plurality of text segments to be compared based on the preset comparison length value and the text segments to be detected; comparing each text segment to be detected with all text segments to be compared one by one, and when the edit distance between the text segment to be detected and the compared text segment to be compared is within the preset edit distance, combining the text segment to be detected with the compared text segment to be compared to form a similar text comparison group; comparing the text segments to be detected in all similar text comparison groups with the corresponding text segments to be compared, and calculating the number of repeated characters and the number of consecutive repeated characters in the text segment to be detected; and calculating the similarity between the text to be detected and the comparison text based on the number of repeated characters and the number of consecutive repeated characters in all text segments to be detected in the text to be detected.

[0101] The text similarity detection method provided in the embodiment of the present application is to divide the text to be detected into multiple text segments to be detected, divide the text in the comparison text into multiple text segments to be compared, and compare each text segment to be detected with all text segments to be compared one by one and calculate the edit distance between the text segment to be detected and the compared text segment to be compared, so as to achieve rapid positioning, determine the text segment to be compared that is most similar to the text segment to be detected in the comparison text, and greatly improve the positioning efficiency compared with the conventional detection method. After obtaining the similar text comparison group, the number of repeated characters and the number of consecutive repeated characters between the text segment to be detected and the corresponding text segment to be compared in each similar text comparison group are calculated, so as to obtain the similarity between the text to be detected and the comparison text. The text similarity detection method provided in the embodiment of the present application avoids the influence of punctuation marks, solves the defects of poor adaptability of the vocabulary and complex weight calculation, and is applicable to long texts and short texts at the same time. Compared with the existing methods, the detection efficiency and applicability to different texts are greatly improved.

[0102] Figures 2 to 5 Schematic diagram of the flow chart of the text similarity detection method in the embodiment of the present application. It should be understood that although Figures 2 to 5 The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Moreover, Figures 2 to 5 At least part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.

[0103] like Figure 6 As shown, the present application also provides a text similarity detection device, including an acquisition module, a first segmentation module, a second segmentation module, a combination module, a comparison module and a calculation module.

[0104] The acquisition module is used to acquire the text to be detected, the comparison text and the preset comparison length value; the first segmentation module is used to segment the text in the text to be detected into multiple text segments to be detected; the second segmentation module is used to segment the text in the comparison text into multiple text segments to be compared based on the preset comparison length value and the text segments to be detected; the combination module is used to compare each text segment to be detected with all text segments to be compared one by one, and when the edit distance between the text segment to be detected and the compared text segment to be compared is within the preset edit distance, the text segment to be detected and the compared text segment to be compared are combined to form a similar text comparison group; the comparison module is used to compare the text segments to be detected in all similar text comparison groups with the corresponding text segments to be compared, and calculate the number of repeated characters and the number of consecutive repeated characters in the text segment to be detected; the calculation module is used to calculate the similarity between the text to be detected and the comparison text based on the number of repeated characters and the number of consecutive repeated characters in all text segments to be detected in the text to be detected.

[0105] The text similarity detection device provided in the embodiment of the present application can realize the text similarity detection method of any of the above embodiments of the present application through the acquisition module, the first segmentation module, the second segmentation module, the combination module, the comparison module and the calculation module, so that the text to be detected can be divided into multiple text segments to be detected, and the text in the comparison text can be divided into multiple text segments to be compared. By comparing each text segment to be detected with all text segments to be compared one by one and calculating the edit distance between the text segment to be detected and the compared text segment to be compared, fast positioning is achieved, and the text segment to be compared that is most similar to the text segment to be detected in the comparison text is determined. Compared with the conventional detection method, the positioning efficiency is greatly improved. After obtaining the similar text comparison group, the number of repeated characters and the number of continuous repeated characters between the text segment to be detected and the corresponding text segment to be compared in each similar text comparison group are calculated, and then the similarity between the text to be detected and the comparison text can be obtained. The text similarity detection device provided in the embodiment of the present application avoids the influence of punctuation marks, and also solves the defects of poor adaptability of the vocabulary and complex weight calculation. It is also applicable to long texts and short texts, and greatly improves the detection efficiency and applicability to different texts.

[0106] The text similarity detection device includes a processor and a memory. The acquisition module, the first segmentation module, the second segmentation module, the combination module, the comparison module and the calculation module are all stored in the memory as program units, and the processor executes the program modules stored in the memory to implement corresponding functions.

[0107] The processor includes a kernel, and the kernel retrieves the corresponding program unit from the memory. One or more kernels can be provided, and the text similarity detection method of any of the aforementioned embodiments of the present application can be implemented by adjusting kernel parameters.

[0108] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM, and the memory includes at least one memory chip.

[0109] An embodiment of the present application also provides a machine-readable storage medium, on which instructions are stored. When the instructions are executed by a processor, the processor is configured to execute the text similarity detection method of any of the aforementioned embodiments of the present application.

[0110] The machine-readable storage medium provided in the embodiment of the present application, when executed by the processor, enables the processor to be configured to execute the text similarity detection method of any of the aforementioned embodiments of the present application, and can divide the text to be detected into multiple text segments to be detected, divide the text in the comparison text into multiple text segments to be compared, and achieve rapid positioning by comparing each text segment to be detected with all text segments to be compared one by one and calculating the edit distance between the text segment to be detected and the compared text segment to be compared, and determine the text segment to be compared that is most similar to the text segment to be detected in the comparison text, which greatly improves the positioning efficiency compared to the conventional detection method. After obtaining the similar text comparison group, the number of repeated characters and the number of consecutive repeated characters between the text segment to be detected and the corresponding text segment to be compared in each similar text comparison group are calculated, and then the similarity between the text to be detected and the comparison text can be obtained. The text similarity detection method provided in the embodiment of the present application, while avoiding the influence of punctuation marks, also solves the defects of poor adaptability of the vocabulary and complex weight calculation, and is applicable to long texts and short texts at the same time. Compared with the existing method, the detection efficiency and applicability to different texts are greatly improved.

[0111] The embodiment of the present application also provides a computer device, which may be a terminal, and its internal structure diagram may be as follows: Figure 8 As shown. The computer device includes a processor A01, a network interface A02, a display screen A04, an input device A05 and a memory (not shown in the figure) connected through a system bus. Among them, the processor A01 of the computer device is used to provide computing and control capabilities. The memory of the computer device includes an internal memory A03 and a non-volatile storage medium A06. The non-volatile storage medium A06 stores an operating system B01 and a computer program B02. The internal memory A03 provides an environment for the operation of the operating system B01 and the computer program B02 in the non-volatile storage medium A06. The network interface A02 of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor A01, a text similarity detection method is implemented. The display screen A04 of the computer device can be a liquid crystal display or an electronic ink display, and the input device A05 of the computer device can be a touch layer covered on the display screen, or a key, trackball or touchpad set on the computer device housing, or an external keyboard, touchpad or mouse, etc.

[0112] Those skilled in the art will understand that Figure 8 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0113] In one embodiment, the text similarity detection device provided by the present application can be implemented in the form of a computer program. The computer program can be Figure 8 The computer device shown in the figure is run. The memory of the computer device can store various program modules constituting the text similarity detection device, for example, Figure 6 The acquisition module, the first segmentation module, the second segmentation module, the combination module, the comparison module and the calculation module shown in the figure. The computer program composed of each program module enables the processor to execute the steps of the text similarity detection method of each embodiment of the present application described in this specification.

[0114] Figure 8 The computer device shown can be Figure 6 The acquisition module in the text similarity detection device shown executes step S110, the first segmentation module executes step S210, the second segmentation module executes step S310, the combination module executes step S410, the comparison module executes step S510, and the calculation module executes step S610.

[0115] like Figure 7 As shown, the present application also provides a text similarity detection system, which includes: a preprocessing component, a comparison component, and a calculation component.

[0116] The preprocessing component is used to obtain the text to be detected, the comparison text and the preset comparison length value, and divide the text in the text to be detected into multiple text segments to be detected; based on the preset comparison length value and the text segments to be detected, divide the text in the comparison text into multiple text segments to be compared; the comparison component is used to compare each text segment to be detected with all text segments to be compared one by one, and when the edit distance between the text segment to be detected and the compared text segment to be compared is within the preset edit distance, the text segment to be detected and the compared text segment to be compared are combined to form a similar text comparison group; the text segments to be detected in all similar text comparison groups are compared with the corresponding text segments to be compared, and the number of repeated characters and the number of consecutive repeated characters in the text segment to be detected are calculated; the calculation component is used to calculate the similarity between the text to be detected and the comparison text based on the number of repeated characters and the number of consecutive repeated characters in all text segments to be detected in the text to be detected.

[0117] The text similarity detection system provided by the embodiment of the present application can realize the text similarity detection method of any of the aforementioned embodiments of the present application through a preprocessing component, a comparison component, and a calculation component, so that the text to be detected can be divided into multiple text segments to be detected, and the text in the comparison text can be divided into multiple text segments to be compared, and each text segment to be detected is compared with all text segments to be compared one by one and the edit distance between the text segment to be detected and the compared text segment to be compared is calculated, so as to realize rapid positioning, and determine the text segment to be compared that is most similar to the text segment to be detected in the comparison text, and compared with the conventional detection method, greatly improve the positioning efficiency. After obtaining the similar text comparison group, the number of repeated characters and the number of continuous repeated characters between the text segment to be detected and the corresponding text segment to be compared in each similar text comparison group are calculated, so as to obtain the similarity between the text to be detected and the comparison text, and the text similarity detection system provided by the embodiment of the present application, while avoiding the influence of punctuation marks, also solves the defects of poor adaptability of the vocabulary and complex weight calculation, and is applicable to long texts and short texts at the same time, greatly improving the detection efficiency and the applicability to different texts.

[0118] The present application also provides a computer program product, which, when executed by a processor, is suitable for executing a program that is initialized with the following method steps: obtaining a text to be detected, a comparison text, and a preset comparison length value; dividing the text in the text to be detected into a plurality of text segments to be detected; based on the preset comparison length value and the text segments to be detected, dividing the text in the comparison text into a plurality of text segments to be compared; comparing each text segment to be detected with all text segments to be compared one by one, and when the edit distance between the text segment to be detected and the compared text segment to be compared is within the preset edit distance, combining the text segment to be detected with the compared text segment to be compared to form a similar text comparison group; comparing the text segments to be detected in all similar text comparison groups with the corresponding text segments to be compared, and calculating the number of repeated characters and the number of consecutive repeated characters in the text segment to be detected; and calculating the similarity between the text to be detected and the comparison text based on the number of repeated characters and the number of consecutive repeated characters in all text segments to be detected in the text to be detected.

[0119] The computer program product provided in the embodiment of the present application, when executed by the processor, can divide the text to be detected into multiple text segments to be detected, divide the text in the comparison text into multiple text segments to be compared, and compare each text segment to be detected with all text segments to be compared one by one and calculate the edit distance between the text segment to be detected and the compared text segment to be compared, so as to achieve rapid positioning, determine the text segment to be compared that is most similar to the text segment to be detected in the comparison text, and greatly improve the positioning efficiency compared with the conventional detection method. After obtaining the similar text comparison group, the number of repeated characters and the number of continuous repeated characters between the text segment to be detected and the corresponding text segment to be compared in each similar text comparison group are calculated, so as to obtain the similarity between the text to be detected and the comparison text. The computer program product provided in the embodiment of the present application, by executing the text similarity detection method of any of the aforementioned embodiments of the present application, avoids the influence of punctuation marks, and solves the defects of poor adaptability of the vocabulary and complex weight calculation, and is applicable to long texts and short texts at the same time. Compared with the existing method, the detection efficiency and the applicability to different texts are greatly improved.

[0120] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0121] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0122] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0123] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0124] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0125] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0126] Computer readable media include permanent and non-permanent, removable and non-removable media, and can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0127] It should also be noted that the term "includes", "comprising" or any other variation thereof is intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of further restrictions, an element defined by the sentence "comprising a ..." does not exclude the existence of other identical elements in the process, method, commodity or device including the element.

[0128] The above are only embodiments of the present application and are not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included within the scope of the claims of the present application.

Claims

1. A text similarity detection method, characterized in that: The method comprises: Get the text to be detected, the comparison text and the preset comparison length value; Segmenting the text in the to-be-detected text into a plurality of to-be-detected text segments; Based on the preset comparison length value and the to-be-detected text segment, the characters in the comparison text are divided into a plurality of to-be-detected text segments; Compare each of the to-be-detected text segments with all of the to-be-compared text segments one by one, and when the edit distance between the to-be-detected text segment and the to-be-compared text segment is within a preset edit distance, combine the to-be-detected text segment and the to-be-compared text segment to form a similar text comparison group; Compare the to-be-detected text segments in all the similar text comparison groups with the corresponding to-be-compared text segments, and calculate the number of repeated characters and the number of consecutively repeated characters in the to-be-detected text segments; Based on the number of repeated characters and the number of continuously repeated characters in all the text segments to be detected in the text to be detected, the similarity between the text to be detected and the comparison text is calculated.

2. The text similarity detection method according to claim 1, characterized in that: The step of dividing the characters in the to-be-detected text into a plurality of to-be-detected text segments comprises: Preprocessing the text to be detected to extract characters from the text to be detected; Acquire a comparison unit and a symbol unit in the text to be detected, wherein the symbol unit is smaller than the comparison unit; Based on the comparison unit and the symbol unit, the characters in the to-be-detected text are divided into a plurality of to-be-detected text segments separated by the comparison unit and the symbol unit.

3. The text similarity detection method according to claim 2, characterized in that: The step of dividing the text in the comparison text into a plurality of text segments to be compared based on the preset comparison length value and the text segment to be detected comprises: Preprocessing the comparison text to extract words from the comparison text; Among all the to-be-detected text segments, the to-be-detected text segment with the shortest length is screened out, and the length of the to-be-detected text segment with the shortest length is the minimum character length value; Compare the preset comparison length value with the minimum character length value, and use the larger one as the comparison length initial value; Based on the initial comparison length value, the characters in the comparison text are divided into a plurality of to-be-compared text segments having the same length as the initial comparison length value.

4. The text similarity detection method according to claim 3, characterized in that: The method further comprises: When comparing each of the to-be-detected text segments with all of the to-be-compared text segments one by one, if the edit distance between the to-be-detected text segment and one of the to-be-compared text segments is greater than the actual length of the to-be-detected text segment, the initial comparison length value is reset until the edit distances between the to-be-detected text segment and all of the to-be-compared text segments are less than or equal to the actual length of the to-be-detected text segment.

5. The text similarity detection method according to claim 4, characterized in that: The step of resetting the preset comparison length value comprises: The actual length of the to-be-detected text segment is compared with the preset comparison length value, the comparison length initial value is updated to the larger one, and the step of dividing the characters in the comparison text into a plurality of to-be-compared text segments is re-executed.

6. The text similarity detection method according to claim 2, characterized in that: The comparison unit is a period-like symbol in the text, and the symbol unit is a comma-like symbol in the text.

7. The text similarity detection method according to claim 1, characterized in that: The step of comparing the to-be-detected text segments in all the similar text comparison groups with the corresponding to-be-compared text segments and calculating the number of repeated characters and the number of continuously repeated characters in the to-be-detected text segments comprises: Get the preset displacement step size; Based on the preset displacement step, performing at least one position correction on the beginning or the end of the to-be-compared text segment in all the similar text comparison groups; Calculating the edit distance between the to-be-detected text segment and the corresponding to-be-compared text segment after position correction, and selecting the to-be-compared text segment corresponding to the to-be-detected text segment with the smallest edit distance as the optimal to-be-compared text segment; The optimal to-be-compared text segment is compared with the corresponding to-be-detected text segment, and the number of repeated characters and the number of continuously repeated characters in the to-be-detected text segment are calculated.

8. A text similarity detection device, characterized in that: The device comprises: An acquisition module is used to acquire the text to be detected, the comparison text, and the preset comparison length value; A first segmentation module, used for segmenting the characters in the to-be-detected text into a plurality of to-be-detected text segments; A second segmentation module, configured to segment the text in the comparison text into a plurality of to-be-compared text segments based on the preset comparison length value and the to-be-detected text segment; A combining module, for comparing each of the to-be-detected text segments with all the to-be-compared text segments one by one, and when the edit distance between the to-be-detected text segment and the to-be-compared text segment is within a preset edit distance, combining the to-be-detected text segment with the to-be-compared text segment to form a similar text comparison group; A comparison module, used for comparing the to-be-detected text segments in all the similar text comparison groups with the corresponding to-be-compared text segments, and calculating the number of repeated characters and the number of consecutively repeated characters in the to-be-detected text segments; The calculation module is used to calculate the similarity between the text to be detected and the comparison text based on the number of repeated characters and the number of continuously repeated characters in all the text segments to be detected in the text to be detected.

9. A text similarity detection system, characterized in that: The system includes: a preprocessing component, a comparison component, and a calculation component; The preprocessing component is used to obtain the text to be detected, the comparison text and the preset comparison length value, and divide the text in the text to be detected into multiple text segments to be detected; based on the preset comparison length value and the text segments to be detected, divide the text in the comparison text into multiple text segments to be compared; The comparison component is used to compare each of the to-be-detected text segments with all the to-be-compared text segments one by one, and when the edit distance between the to-be-detected text segment and the to-be-compared text segment is within a preset edit distance, combine the to-be-detected text segment with the to-be-compared text segment to form a similar text comparison group; compare the to-be-detected text segments in all the similar text comparison groups with the corresponding to-be-compared text segments, and calculate the number of repeated characters and the number of consecutive repeated characters in the to-be-detected text segment; The calculation component is used to calculate the similarity between the text to be detected and the comparison text based on the number of repeated characters and the number of continuously repeated characters in all the text segments to be detected in the text to be detected.

10. A machine-readable storage medium having instructions stored thereon, characterized in that: When the instructions are executed by a processor, the processor is configured to perform the text similarity detection method according to any one of claims 1 to 7.

11. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the computer program implements the text similarity detection method according to any one of claims 1 to 7.

Images