Text similarity detection method, device and system

Abstract

The application relates to the field of text processing and discloses a text similarity detection method, device and system. The method comprises the following steps: obtaining a to-be-detected text, a comparison text and a preset comparison length value; dividing the characters in the to-be-detected text into a plurality of to-be-detected text fields; dividing the characters in the comparison text into a plurality of to-be-compared text fields based on the preset comparison length value and the to-be-detected text fields; comparing each to-be-detected text field with all to-be-compared text fields one by one; when the edit distance between the to-be-detected text field and the compared to-be-compared text field is within a preset edit distance, combining the to-be-detected text field and the compared to-be-compared text field to form a similar character comparison group and performing comparison; calculating the number of repeated characters and the number of continuous repeated characters in the to-be-detected text field, and then obtaining the text similarity of the two. The method provided by the application can efficiently and accurately locate the similar text position of the to-be-detected text in the comparison text and detect the text similarity of the two.

Description

Technical Field

[0001] This application relates to the field of text processing technology, specifically to a text similarity detection method, apparatus, and system. Background Technology

[0002] For application systems that calculate royalties based on textual elements, it's crucial to accurately determine the similarity between user-uploaded manuscripts and publicly available text data within the system. Typically, plagiarized articles won't be identical to the original; instead, they'll select a portion or make minor modifications to evade system detection.

[0003] Among existing text similarity detection methods, the SimHash algorithm is not good at handling short texts, and the MD5 algorithm struggles to avoid the influence of punctuation on the detection results. If conventional methods are used, such as using a dictionary to segment and stop words or pre-setting weights, the accuracy will be affected if the dictionary is not updated in a timely manner or lacks relevant vocabulary. Furthermore, complex weight pre-setting and stop word processing increase computational complexity. On the other hand, directly comparing two texts one by one is not only very slow, but also difficult to detect if the plagiarized content has been modified. Summary of the Invention

[0004] The purpose of this application is to provide a text similarity detection method, apparatus, and system to solve the above-mentioned problems.

[0005] To achieve the above objectives, the first aspect of this application provides a text similarity detection method, the method comprising: acquiring a text to be detected, a comparison text, and a preset comparison length value; segmenting the text in the text to be detected into multiple text fields to be detected; based on the preset comparison length value and the text fields to be detected, segmenting the text in the comparison text into multiple text fields to be compared; comparing each text field to be detected with all text fields to be compared one by one, and when the edit distance between the text field to be detected and the text field to be compared is within a preset edit distance, combining the text field to be detected with the text field to be compared to form a similar text comparison group; comparing the text fields to be detected in all similar text comparison groups with the corresponding text fields to be compared, and calculating the number of repeated characters and the number of consecutive repeated characters in the text fields to be detected; and calculating the similarity between the text to be detected and the comparison text based on the number of repeated characters and the number of consecutive repeated characters in all text fields to be detected in the text to be detected.

[0006] In this embodiment of the application, the step of segmenting the text in the text to be detected into multiple text segments to be detected includes: preprocessing the text to be detected to extract the text; obtaining the comparison unit and symbol unit in the text to be detected, wherein the symbol unit is smaller than the comparison unit; and segmenting the text in the text to be detected into multiple text segments to be detected based on the comparison unit and symbol unit.

[0007] In this embodiment of the application, the step of segmenting the text in the comparison text into multiple text fields to be compared based on a preset comparison length value and the text field to be detected includes: preprocessing the comparison text to extract the text; selecting the shortest text field to be detected from all text fields to be detected, with the length of the shortest text field to be detected being the minimum text length value; comparing the preset comparison length value with the minimum text length value, and taking the larger one as the initial comparison length value; and segmenting the text in the comparison text into multiple text fields to be compared with the same length as the initial comparison length value based on the initial comparison length value.

[0008] In this embodiment of the application, the method further includes: when comparing each text field to be detected with all text fields to be compared one by one, if the edit distance between the text field to be detected and one of the text fields to be compared is greater than the actual length of the text field to be detected, then the initial value of the comparison length is reset until the edit distance between the text field to be detected and all the text fields to be compared is less than or equal to the actual length of the text field to be detected.

[0009] In this embodiment of the application, the step of resetting the preset comparison length value includes: comparing the actual length of the text field to be detected with the size of the preset comparison length value, updating the initial comparison length value to the larger one, and re-executing the step of segmenting the text in the comparison text into multiple text fields to be compared.

[0010] In the embodiments of this application, the comparison unit is a period-like symbol in the text, and the symbol unit is a comma-like symbol in the text.

[0011] In this embodiment of the application, the steps of comparing the text field to be detected in all similar text comparison groups with the corresponding text field to be compared, and calculating the number of repeated characters and the number of consecutive repeated characters in the text field to be detected, include: obtaining a preset displacement step size; based on the preset displacement step size, performing at least one position correction on the beginning or end of the text field to be compared in all similar text comparison groups; calculating the edit distance between the text field to be detected and the corresponding text field to be compared after position correction, and selecting the text field to be compared corresponding to the text field to be detected with the smallest edit distance as the optimal text field to be compared; comparing the optimal text field to be compared with the corresponding text field to be detected, and calculating the number of repeated characters and the number of consecutive repeated characters in the text field to be detected.

[0012] A second aspect of this application provides a text similarity detection device, comprising: an acquisition module for acquiring a text to be detected, a comparison text, and a preset comparison length value; a first segmentation module for segmenting the text in the text to be detected into multiple text fields to be detected; a second segmentation module for segmenting the text in the comparison text into multiple text fields to be compared based on the preset comparison length value and the text fields to be detected; a combination module for comparing each text field to be detected with all text fields to be compared one by one, and combining the text fields to be detected with the text fields to be compared to form a similar text comparison group when the edit distance between the text field to be detected and the text fields to be compared is within a preset edit distance; a comparison module for comparing the text fields to be detected in all similar text comparison groups with the corresponding text fields to be compared, and calculating the number of repeated characters and the number of consecutive repeated characters in the text fields to be detected; and a calculation module for calculating the similarity between the text to be detected and the comparison text based on the number of repeated characters and the number of consecutive repeated characters in all text fields to be detected in the text to be detected.

[0013] A third aspect of this application provides a text similarity detection system, comprising: a preprocessing component, a comparison component, and a calculation component; the preprocessing component is used to acquire the text to be detected, the comparison text, and a preset comparison length value, and to segment the text in the text to be detected into multiple text fields to be detected; based on the preset comparison length value and the text fields to be detected, the text in the comparison text is segmented into multiple text fields to be compared; the comparison component is used to compare each text field to be detected with all text fields to be compared one by one, and when the edit distance between the text field to be detected and the text field to be compared is within a preset edit distance, the text field to be detected and the text field to be compared are combined to form a similar text comparison group; the text fields to be detected in all similar text comparison groups are compared with the corresponding text fields to be compared, and the number of repeated characters and the number of consecutive repeated characters in the text fields to be detected are calculated; the calculation component is used to calculate the similarity between the text to be detected and the comparison text based on the number of repeated characters and the number of consecutive repeated characters in all text fields to be detected in the text to be detected.

[0014] A fourth aspect of this application provides a machine-readable storage medium storing instructions that, when executed by a processor, cause the processor to perform a text similarity detection method according to any of the foregoing embodiments of the first aspect of this application.

[0015] The fifth aspect of this application provides a computer program product, including a computer program that, when executed by a processor, implements the text similarity detection method of any of the foregoing embodiments of the first aspect of this application.

[0016] The above technical solution can segment the text to be detected and the comparison text into multiple text fields to be detected and multiple text fields to be compared, respectively. By comparing each text field to be detected with all text fields to be compared one by one and calculating the edit distance between the text field to be detected and the compared text fields, rapid positioning is achieved to determine the text field to be compared that is most similar to the text field to be detected in the comparison text. The method of this application embodiment is not only applicable to texts of various lengths, but also can accurately determine the similarity between the text to be detected and the comparison text without relying on a dictionary or performing complex weight calculations.

[0017] Other features and advantages of the embodiments of this application will be described in detail in the following detailed description section. Attached Figure Description

[0018] The accompanying drawings are provided to further illustrate the embodiments of this application and form part of the specification. They are used together with the following detailed description to explain the embodiments of this application, but do not constitute a limitation on the embodiments of this application. In the drawings:

[0019] Figure 1 This illustration schematically shows an application environment diagram of the text similarity detection method according to an embodiment of this application;

[0020] Figure 2 The schematic diagram illustrates a flowchart of a text similarity detection method according to an embodiment of this application;

[0021] Figure 3 The illustration shows a flowchart of step S120 of the text similarity detection method according to an embodiment of the present application;

[0022] Figure 4 The schematic diagram illustrates a flowchart of step S130 of the text similarity detection method according to an embodiment of this application;

[0023] Figure 5 The illustration shows a flowchart of step S150 of the text similarity detection method according to an embodiment of this application;

[0024] Figure 6 This schematic diagram illustrates the structural block diagram of a text similarity detection device according to an embodiment of this application;

[0025] Figure 7 This schematic diagram illustrates the structural block diagram of a text similarity detection system according to an embodiment of this application;

[0026] Figure 8 The diagram illustrates the internal structure of a computer device according to an embodiment of this application. Detailed Implementation

[0027] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are only for illustration and explanation of the embodiments of this application and are not intended to limit the embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of this application.

[0028] It should be noted that if the embodiments of this application involve directional indicators (such as up, down, left, right, front, back, etc.), the directional indicators are only used to explain the relative positional relationship and movement of the components in a certain specific posture (as shown in the figure). If the specific posture changes, the directional indicators will also change accordingly.

[0029] Furthermore, if the embodiments of this application involve descriptions such as "first" or "second," these descriptions are for descriptive purposes only and should not be construed as indicating or implying their relative importance or implicitly specifying the number of technical features indicated. Therefore, features defined with "first" or "second" may explicitly or implicitly include at least one of those features. Additionally, the technical solutions of various embodiments can be combined with each other, but this must be based on the ability of those skilled in the art to implement them. If the combination of technical solutions is contradictory or impossible to implement, it should be considered that such a combination of technical solutions does not exist and is not within the scope of protection claimed in this application.

[0030] The acquisition, transmission, storage, use, and processing of data in this application comply with relevant national laws and regulations. Furthermore, it should be noted that existing industry solutions such as software, components, and models may be mentioned in the embodiments of this application. These should be considered exemplary, intended only to illustrate the feasibility of implementing the technical solution of this application, and do not imply that the applicant has already used or necessarily used such solutions.

[0031] The text similarity detection method provided in this application can be applied to, for example... Figure 1 In the application environment shown, terminal 101 communicates with server 102 via a network. The user can issue commands to server 102 through terminal 101, causing the server to execute the text similarity detection method provided in this embodiment and send the detection results to terminal 101, allowing the user to obtain the results. Terminal 101 can be, but is not limited to, various personal computers, laptops, smartphones, tablets, and portable wearable devices. Server 102 can be implemented using a standalone server or a server cluster consisting of multiple servers.

[0032] Figure 2 A schematic flowchart of a text similarity detection method according to an embodiment of this application is shown. Figure 2 As shown, in one embodiment of this application, a text similarity detection method is provided. This embodiment mainly applies this method to the above-mentioned... Figure 1 Taking terminal 101 (or server 102) as an example, the text similarity detection method in this embodiment includes steps S110 to S160.

[0033] In step S110, the text to be detected, the text to be compared, and the preset comparison length value are obtained.

[0034] In this embodiment, the text to be detected is compared with the comparison text to obtain the number of repeated characters in the text content of the text to be detected and the text content of the comparison text, and then calculate the similarity between the text to be detected and the comparison text. The preset comparison length value is a preset value that can be selected according to the length of each text field of the text to be detected. For example, the preset comparison length value can be any integer value such as 5, 6, 7, 10, 15 or 20.

[0035] In step S120, the text in the text to be detected is divided into multiple text fields to be detected.

[0036] like Figure 3 As shown, in some embodiments, step S120 includes steps S121 to S123.

[0037] In step S121, the text to be detected is preprocessed to extract the text.

[0038] In step S122, the comparison unit and symbol unit in the text to be detected are obtained. The symbol unit is smaller than the comparison unit.

[0039] In step S123, based on comparison units and symbol units, the text in the text to be detected is divided into multiple text segments to be detected, with the comparison units and symbol units as the intervals.

[0040] In this embodiment, when preprocessing the text to be detected, the entire text content of the text to be detected is first obtained, and then punctuation marks, tags, emoticons, and other symbols unrelated to the text are removed to extract the text content of the text to be detected, thus completing the preprocessing process. The text content can be Chinese, English, or other languages.

[0041] During the preprocessing of the text to be detected, the text is divided into multiple text fields separated by symbolic units and comparison units, based on the positions of symbolic units and comparison units in the text to be detected.

[0042] The symbol unit and comparison unit are generally punctuation marks in the text. For example, period-type symbols include periods, exclamation marks, and question marks, while comma-type symbols include commas, pauses, and semicolons. Specifically, in this embodiment, the comparison unit is period-type symbols in the text, and the symbol unit is comma-type symbols in the text. That is, the text in the text to be detected is divided into multiple text fields by comma-type symbols and period-type symbols.

[0043] In step S130, based on the preset comparison length value and the text field to be detected, the text in the comparison text is divided into multiple text fields to be compared.

[0044] like Figure 4 As shown, in some embodiments, step S130 includes steps S131 to S134.

[0045] In step S131, the comparison text is preprocessed to extract the text from the comparison text.

[0046] In step S132, among all the text fields to be detected, the shortest text field to be detected is selected, wherein the length of the shortest text field to be detected is the minimum text length value.

[0047] In step S133, the preset comparison length value is compared with the minimum text length value, and the larger one is taken as the initial comparison length value.

[0048] In step S134, based on the initial comparison length value, the text in the comparison text is divided into multiple text fields to be compared, each with the same length as the initial comparison length value.

[0049] In this embodiment, the preprocessing process for the comparison text is the same as the preprocessing process for the text to be detected. In the same way, all the text content in the comparison text is first obtained, and then punctuation marks, tags, emoticons and other symbols that are not related to the text are removed to extract the text content in the comparison text and complete the preprocessing process.

[0050] After preprocessing the comparison text, the preset comparison length value is compared with the minimum text length value. The larger value is used as the initial comparison length value, and the text is then divided into multiple comparison fields with a length equal to the initial comparison length value. Any remaining text is treated as a separate comparison field.

[0051] Specifically, in this embodiment of the application, the text to be detected is, for example: Finance is a social practice, and its purpose is to serve the real economy. Its essence is not to pursue false prosperity, but to be grounded in society, rooted in life, and focused on real-world issues.

[0052] The text to be compared is: New finance needs to return to its core function of providing services. Finance, in essence, is a social practice, and its purpose is to serve the real economy. The true nature of finance is not to pursue false prosperity, but to be grounded in society, rooted in life, and focused on real-world issues. Financial institutions serve the general public, bearing the joys and sorrows of countless families, and directly confronting numerous social problems and pain points. Finance is like water; only by upholding the principles of benevolence and good governance can it guide the flow of resources and nourish all things.

[0053] After preprocessing the text to be detected and the comparison text, the text to be detected is: "Finance is a social practice activity. The purpose of finance is to serve the real economy. Its essence is not to pursue false prosperity, but to be solidly close to society, originate from life, and focus on problems."

[0054] The contrasting text reads: "New finance needs to return to the essence of service. Finance is ultimately a social practice activity. The purpose of finance is to serve the real economy. The essence of finance is not to pursue false prosperity, but to be solidly close to society, originate from life, and focus on problems. Financial institutions face the general public, bear the joys and sorrows of thousands of families, and directly confront many social problems and pain points. Finance is like water. Only by upholding the principles of goodwill and good governance can it guide the situation and nourish all things."

[0055] Therefore, the minimum text length of the text to be detected is the text field corresponding to "finance is a social practice activity", with a length of 11. If the preset comparison length value is 12, then the preset comparison length value is larger, so the initial value of the comparison length is set to 12, and the comparison text is segmented according to the initial value of the comparison length.

[0056] The text fields obtained from the text to be detected are as follows:

[0057] A1: Finance is a social practice;

[0058] A2: The purpose of finance is to serve the real economy;

[0059] A3: Its true nature is not to pursue false prosperity;

[0060] A4: Instead, we should focus on issues that are relevant to society and everyday life.

[0061] The text fields to be compared obtained from the comparison text are as follows:

[0062] B1: New finance needs to return to its core function of service;

[0063] B2: Finance, in essence, is a social practice activity;

[0064] B3: The purpose of finance is to serve the real economy;

[0065] B4: The essential nature of financial services is not to serve;

[0066] B5: Instead of pursuing a false prosperity, we must be grounded;

[0067] B6: Solidly grounded, closely connected to society, and rooted in life;

[0068] B7: Focusing on the problems faced by financial institutions;

[0069] B8: The general public bears the burden of every family's joys and sorrows;

[0070] B9: Confronting numerous social problems and social pains;

[0071] B10: Financial issues are like water, guided by benevolence;

[0072] B11: Only by adhering to the principles of good governance can we guide the situation in a way that is conducive to its development;

[0073] B12: To nourish all things;

[0074] In step S140, each text field to be detected is compared with all text fields to be compared one by one. When the edit distance between the text field to be detected and the text field to be compared is within a preset edit distance, the text field to be detected and the text field to be compared are combined to form a similar text comparison group.

[0075] In some embodiments, when comparing each text field to be detected with all text fields to be compared one by one, if the edit distance between the text field to be detected and one of the text fields to be compared is greater than the actual length of the text field to be detected, the initial value of the comparison length is reset until the edit distance between the text field to be detected and all the text fields to be compared is less than or equal to the actual length of the text field to be detected.

[0076] In this embodiment, the preset edit distance is set to 50% of the initial value of the comparison length, therefore the preset edit distance is 6. In this application, the preset edit distance is an integer value; if it is not an integer value, the nearest integer value is taken.

[0077] In the process of comparing each text field to be detected with all text fields to be compared one by one, if the edit distance between the text field to be detected and the text field to be compared is 6 or less, then the text field to be detected and the text field to be compared are combined to form a similar text comparison group.

[0078] For example, in this embodiment of the application, the text field A1 to be detected is compared with the text fields B1 to B12 to be compared one by one, and the text field to be compared with the most similar one is selected and combined to form a similar text comparison group.

[0079] In this embodiment, when comparing text field A1 to be detected with text field B1 to be compared, the edit distance between "finance is a social practice activity" and "new finance needs to return to the origin of service" is 10, so B1 is discarded. When comparing text field A1 to be detected with text field B2 to be compared, the edit distance between "finance is a social practice activity" and B2 "finance is ultimately a social practice activity" is 5, so B2 is retained. And so on, until B2 is finally obtained as the text field to be compared with A1 with the smallest edit distance, that is, text field B2 is the text field to be compared with text field A1 most similar to text field A1, and the two are combined to form a similar text comparison group.

[0080] Furthermore, the step of resetting the preset comparison length value includes: comparing the actual length of the text field to be detected with the preset comparison length value, updating the initial comparison length value to the larger one, and re-executing the step of segmenting the text in the comparison text into multiple text fields to be compared.

[0081] In this embodiment, when calculating the text fields A2, A3, and A4 to be detected, since the text lengths of the text fields A2, A3, and A4 have changed, if the edit distance between the text field to be detected and the text field to be compared is greater than the actual text length of the text field to be detected, it proves that the text length of the text field to be compared is greater than the actual length of the text field to be detected, which does not conform to the logical rationality of calculating the edit distance. Therefore, it is necessary to reset the preset comparison length value and re-execute the step of dividing the text in the comparison text into multiple text fields to be compared based on the new initial comparison length value.

[0082] Specifically, the comparison and editing distance calculation methods between the text fields A2, A3, and A4 to be detected and the text fields to be compared are the same as those for the comparison and editing distance calculation between the text field A1 to be detected and the text fields to be compared, and will not be repeated here.

[0083] In step S150, the text field to be detected in all similar text comparison groups is compared with the corresponding text field to be compared, and the number of repeated characters and the number of consecutive repeated characters in the text field to be detected are calculated.

[0084] like Figure 5 As shown, in some embodiments, step S150 includes steps S151 to S154.

[0085] In step S151, the preset displacement step size is obtained.

[0086] In step S152, based on a preset displacement step size, the beginning or end of the text to be compared in all similar text comparison groups is corrected at least once.

[0087] In step S153, the edit distance between the text field to be detected and the corresponding text field to be compared after position correction is calculated, and the text field to be compared corresponding to the text field to be detected with the smallest edit distance is selected as the optimal text field to be compared.

[0088] In step S154, the optimal text field to be compared is compared with the corresponding text field to be detected, and the number of repeated characters and the number of consecutive repeated characters in the text field to be detected are calculated.

[0089] In this embodiment, the preset displacement step size can be 1, 2, 3, or any integer value. After determining the preset displacement step size, one, two, or more position corrections can be made at the beginning or end of the text field to be compared, and the edit distance between the text field to be detected and the text field to be compared can be calculated after each position correction. The text field to be compared corresponding to the text field to be detected with the smallest edit distance is selected as the optimal text field to be compared.

[0090] Specifically, as shown in the text field A1 to be detected and the text field B2 to be compared:

[0091] The edit distance between the text field A1 to be detected, "Finance is a social practice activity", and the text field B2 to be compared, "Finance is ultimately a social practice activity", is 5.

[0092] After adjusting the position at the beginning of the text field B2 to be compared, the edit distance between the text field A1 to be tested: "Finance is a social practice activity" and the text field B2 to be compared: "Finance is ultimately a social practice activity" is 7.

[0093] After two position corrections at the beginning of the text field B2 to be compared, the edit distance between the text field A1 to be tested: "Finance is a social practice" and the text field B2 to be compared: "Finance is ultimately a social practice" is 9.

[0094] Ultimately, it can be concluded that there is no need to correct the position of the text field B2 to be compared, that is, the current position of the text field B2 to be compared is the optimal position, and the current text field of the text field B2 to be compared is the optimal text field to be compared.

[0095] Through steps S151 to S154 of the text similarity detection method of this application, the shortest edit distance between the text field to be detected and the text field to be compared can be calculated, the position of the text field to be detected in the comparison text can be quickly located, and the optimal position of the text field to be detected in the comparison text can be obtained through position correction, so as to ensure that the obtained similarity detection result is the most accurate.

[0096] In step S160, the similarity between the text to be detected and the comparison text is calculated based on the number of repeated characters and the number of consecutively repeated characters in all text fields to be detected in the text to be detected.

[0097] In this embodiment, after obtaining four similar text comparison groups, namely the text fields to be detected A1 to A4 and the text fields to be compared that are most similar to the text fields to be detected A1 to A4, the number of repeated characters and the number of consecutive repeated characters of the text fields to be detected A1, A2, A3, and A4 in the text fields to be compared with their most similar text fields are calculated respectively.

[0098] For example, in this embodiment, the text field to be detected, A1, is "Finance is a social practice activity," and the text field to be compared, B2, is "Finance is ultimately a social practice activity." The words appearing in both text fields are: "finance," "is," "society," "practice," and "activity," meaning the number of repeated characters is 9. The words appearing consecutively in both text fields are: "finance" and "social practice activity," and the larger one is taken, meaning the number of consecutively repeated characters is 6.

[0099] After obtaining the number of repeated characters and consecutive repeated characters in all the text fields to be detected, the similarity between the text to be detected and the comparison text can be calculated.

[0100] The text similarity detection method provided in this application includes: acquiring a text to be detected, a comparison text, and a preset comparison length value; segmenting the text in the text to be detected into multiple text fields to be detected; segmenting the text in the comparison text into multiple text fields to be compared based on the preset comparison length value and the text fields to be detected; comparing each text field to be detected with all text fields to be compared one by one, and when the edit distance between the text field to be detected and the text field to be compared is within a preset edit distance, combining the text field to be detected and the text field to be compared to form a similar text comparison group; comparing the text fields to be detected in all similar text comparison groups with the corresponding text fields to be compared, and calculating the number of repeated characters and the number of consecutive repeated characters in the text fields to be detected; and calculating the similarity between the text to be detected and the comparison text based on the number of repeated characters and the number of consecutive repeated characters in all text fields to be detected in the text to be detected.

[0101] The text similarity detection method provided in this application segment the text to be detected into multiple text fields to be detected, and the text in the comparison text into multiple text fields to be compared. By comparing each text field to be detected with all text fields to be compared one by one and calculating the edit distance between the text field to be detected and the text fields to be compared, rapid positioning is achieved, and the text field to be compared with the text field to be detected is determined. Compared with conventional detection methods, this method greatly improves the positioning efficiency. After obtaining the similar text comparison groups, the number of repeated characters and the number of consecutive repeated characters between the text field to be detected and the corresponding text field to be compared are calculated in each similar text comparison group, thereby obtaining the similarity between the text field to be detected and the comparison text. The text similarity detection method provided in this application avoids the influence of punctuation marks and also solves the defects of poor lexicon adaptability and complex weight calculation. It is applicable to both long and short texts, and compared with existing methods, it greatly improves the detection efficiency and applicability to different texts.

[0102] Figures 2 to 5 This is a flowchart illustrating the text similarity detection method in an embodiment of this application. It should be understood that, although... Figures 2 to 5 The steps in the flowchart are shown sequentially as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order in which these steps are executed, and they can be performed in other orders. Figures 2 to 5 At least some of the steps in the process may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least some of the sub-steps or stages of other steps.

[0103] like Figure 6 As shown, this application also provides a text similarity detection device, including an acquisition module, a first segmentation module, a second segmentation module, a combination module, a comparison module, and a calculation module.

[0104] The acquisition module acquires the text to be detected, the comparison text, and a preset comparison length value. The first segmentation module segments the text in the text to be detected into multiple text fields to be detected. The second segmentation module segments the text in the comparison text into multiple text fields to be compared, based on the preset comparison length value and the text fields to be detected. The combination module compares each text field to be detected with all text fields to be compared one by one. When the edit distance between the text field to be detected and the text field to be compared is within a preset edit distance, the text field to be detected and the text field to be compared are combined to form a similar text comparison group. The comparison module compares the text fields to be detected in all similar text comparison groups with the corresponding text fields to be compared, and calculates the number of repeated characters and the number of consecutive repeated characters in the text fields to be detected. The calculation module calculates the similarity between the text to be detected and the comparison text based on the number of repeated characters and the number of consecutive repeated characters in all text fields to be detected in the text to be detected.

[0105] The text similarity detection device provided in this application can implement the text similarity detection method of any of the aforementioned embodiments of this application through an acquisition module, a first segmentation module, a second segmentation module, a combination module, a comparison module, and a calculation module. This allows for the segmentation of the text to be detected into multiple text fields to be detected, and the segmentation of the text in the comparison text into multiple text fields to be compared. By comparing each text field to be detected with all text fields to be compared one by one and calculating the edit distance between the text field to be detected and the compared text fields, rapid positioning is achieved, determining the text field to be compared most similar to the text field to be detected in the comparison text. Compared with conventional detection methods, this significantly improves positioning efficiency. After obtaining similar text comparison groups, the number of repeated characters and the number of consecutively repeated characters between the text field to be detected and the corresponding text field to be compared in each similar text comparison group are calculated, thereby obtaining the similarity between the text to be detected and the comparison text. The text similarity detection device provided in this application avoids the influence of punctuation marks and solves the defects of poor lexicon adaptability and complex weight calculation. It is applicable to both long and short texts, greatly improving detection efficiency and applicability to different texts.

[0106] The text similarity detection device includes a processor and a memory. The acquisition module, the first segmentation module, the second segmentation module, the combination module, the comparison module, and the calculation module are all stored in the memory as program units. The processor executes the program modules stored in the memory to implement the corresponding functions.

[0107] The processor contains a kernel, which retrieves the corresponding program units from memory. One or more kernels can be configured, and the text similarity detection method of any of the foregoing embodiments of this application can be implemented by adjusting the kernel parameters.

[0108] The memory may include non-permanent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM, and the memory includes at least one memory chip.

[0109] This application also provides a machine-readable storage medium storing instructions that, when executed by a processor, cause the processor to be configured to perform the text similarity detection method of any of the foregoing embodiments of this application.

[0110] The machine-readable storage medium provided in this application, when executed by a processor, configures the processor to perform the text similarity detection method of any of the foregoing embodiments of this application. It can segment the text to be detected into multiple text fields to be detected, and segment the text in the comparison text into multiple text fields to be compared. By comparing each text field to be detected with all text fields to be compared one by one and calculating the edit distance between the text field to be detected and the compared text fields, rapid positioning is achieved, determining the text field to be compared most similar to the text field to be detected in the comparison text. Compared with conventional detection methods, this greatly improves positioning efficiency. After obtaining similar text comparison groups, the number of repeated characters and the number of consecutive repeated characters between the text field to be detected and the corresponding text field to be compared in each similar text comparison group are calculated, thereby obtaining the similarity between the text to be detected and the comparison text. The text similarity detection method provided in this application, while avoiding the influence of punctuation marks, also solves the defects of poor dictionary adaptability and complex weight calculation. It is applicable to both long and short texts, and compared with existing methods, it greatly improves detection efficiency and applicability to different texts.

[0111] This application also provides a computer device, which may be a terminal, and its internal structure diagram may be as follows: Figure 8 As shown in the figure, the computer device includes a processor A01, a network interface A02, a display screen A04, an input device A05, and a memory (not shown) connected via a system bus. The processor A01 provides computing and control capabilities. The memory includes internal memory A03 and a non-volatile storage medium A06. The non-volatile storage medium A06 stores an operating system B01 and a computer program B02. The internal memory A03 provides an environment for the operation of the operating system B01 and the computer program B02 stored in the non-volatile storage medium A06. The network interface A02 is used for communication with external terminals via a network connection. When the computer program is executed by the processor A01, it implements a text similarity detection method. The display screen A04 can be a liquid crystal display (LCD) or an e-ink display. The input device A05 can be a touch layer covering the display screen, buttons, a trackball, or a touchpad mounted on the computer device casing, or an external keyboard, touchpad, or mouse.

[0112] Those skilled in the art will understand that Figure 8 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0113] In one embodiment, the text similarity detection device provided in this application can be implemented as a computer program, which can be implemented in the form of, for example, Figure 8 The device runs on the computer shown. The computer's memory can store the various program modules that make up the text similarity detection device, for example, Figure 6 The diagram shows an acquisition module, a first segmentation module, a second segmentation module, a combination module, a comparison module, and a calculation module. The computer program comprised of these modules causes the processor to execute the steps in the text similarity detection methods of the various embodiments of this application described in this specification.

[0114] Figure 8 The computer equipment shown can be used as follows Figure 6 The text similarity detection device shown executes step S110 through the acquisition module, step S210 through the first segmentation module, step S310 through the second segmentation module, step S410 through the combination module, step S510 through the comparison module, and step S610 through the calculation module.

[0115] like Figure 7 As shown, this application also provides a text similarity detection system, which includes a preprocessing component, a comparison component, and a calculation component.

[0116] The preprocessing component acquires the text to be detected, the comparison text, and a preset comparison length value, and segments the text in the text to be detected into multiple text fields to be detected. Based on the preset comparison length value and the text fields to be detected, the text in the comparison text is segmented into multiple text fields to be compared. The comparison component compares each text field to be detected with all text fields to be compared one by one. When the edit distance between the text field to be detected and the text field to be compared is within a preset edit distance, the text field to be detected and the text field to be compared are combined to form a similar text comparison group. The text fields to be detected in all similar text comparison groups are compared with their corresponding text fields to be compared, and the number of repeated characters and consecutive repeated characters in the text fields to be detected are calculated. The calculation component calculates the similarity between the text to be detected and the comparison text based on the number of repeated characters and consecutive repeated characters in all text fields to be detected in the text to be detected.

[0117] The text similarity detection system provided in this application can implement the text similarity detection method of any of the foregoing embodiments of this application through a preprocessing component, a comparison component, and a calculation component. This allows for the segmentation of the text to be detected into multiple text fields to be detected, and the segmentation of the text in the comparison text into multiple text fields to be compared. By comparing each text field to be detected with all text fields to be compared one by one and calculating the edit distance between the text field to be detected and the compared text fields, rapid positioning is achieved, determining the text field to be compared most similar to the text field to be detected in the comparison text. Compared with conventional detection methods, this significantly improves positioning efficiency. After obtaining similar text comparison groups, the number of repeated characters and the number of consecutive repeated characters between the text field to be detected and the corresponding text field to be compared in each similar text comparison group are calculated, thereby obtaining the similarity between the text to be detected and the comparison text. The text similarity detection system provided in this application avoids the influence of punctuation marks and solves the defects of poor lexicon adaptability and complex weight calculation. It is applicable to both long and short texts, greatly improving detection efficiency and applicability to different texts.

[0118] This application also provides a computer program product, which, when executed by a processor, is suitable for performing an initialization program with the following method steps: acquiring a text to be detected, a comparison text, and a preset comparison length value; segmenting the text to be detected into multiple text fields to be detected; based on the preset comparison length value and the text fields to be detected, segmenting the text in the comparison text into multiple text fields to be compared; comparing each text field to be detected with all text fields to be compared one by one, and when the edit distance between the text field to be detected and the text field to be compared is within a preset edit distance, combining the text field to be detected and the text field to be compared to form a similar text comparison group; comparing the text fields to be detected in all similar text comparison groups with the corresponding text fields to be compared, and calculating the number of repeated characters and the number of consecutive repeated characters in the text fields to be detected; and calculating the similarity between the text to be detected and the comparison text based on the number of repeated characters and the number of consecutive repeated characters in all text fields to be detected in the text to be detected.

[0119] When the computer program product provided in this application is executed by a processor, it can segment the text to be detected into multiple text fields and segment the text in the comparison text into multiple text fields to be compared by executing a program that initializes the text similarity detection method of any of the foregoing embodiments of this application. By comparing each text field to be detected with all text fields to be compared one by one and calculating the edit distance between the text field to be detected and the text fields to be compared, rapid positioning is achieved, and the text field to be compared with the text field to be detected that is most similar to the text field to be detected in the comparison text is determined. Compared with conventional detection methods, the positioning efficiency is greatly improved. After obtaining the similar text comparison group, the number of repeated characters and the number of consecutive repeated characters between the text field to be detected and the corresponding text field to be compared in each similar text comparison group are calculated, thereby obtaining the similarity between the text to be detected and the comparison text. The computer program product provided in this application, by executing the text similarity detection method of any of the foregoing embodiments of this application, avoids the influence of punctuation marks and solves the defects of poor lexicon adaptability and complex weight calculation. It is applicable to both long and short texts. Compared with existing methods, it greatly improves detection efficiency and applicability to different texts.

[0120] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0121] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0122] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0123] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0124] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0125] Memory may include non-persistent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0126] Computer-readable media include both permanent and non-permanent, removable and non-removable media, which can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0127] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0128] The above are merely embodiments of this application and are not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. A method for detecting similarity of texts, characterized by, The method comprises: acquiring a text to be detected, a comparison text, and a preset comparison length value; segmenting characters in the text to be detected into a plurality of text fields to be detected; segmenting characters in the comparison text into a plurality of text fields to be compared based on the preset comparison length value and the text fields to be detected; comparing each of the text fields to be detected with all the text fields to be compared one by one, and combining the text field to be detected with the text field to be compared when an edit distance between the text field to be detected and the text field to be compared is within a preset edit distance, to form a similar character comparison group; comparing the text fields to be detected in all the similar character comparison groups with corresponding text fields to be compared, and calculating a number of repeated characters and a number of continuous repeated characters in the text fields to be detected, wherein the step of comparing the text fields to be detected in all the similar character comparison groups with corresponding text fields to be compared, and calculating a number of repeated characters and a number of continuous repeated characters in the text fields to be detected comprises: acquiring a preset displacement step; based on the preset displacement step, performing at least one position correction on a head or tail of the text fields to be compared in all the similar character comparison groups; calculating an edit distance between the text field to be detected and the text field to be compared after the position correction, and selecting the text field to be compared corresponding to the text field to be detected with the smallest edit distance as an optimal text field to be compared; comparing the optimal text field to be compared with the corresponding text field to be detected, and calculating a number of repeated characters and a number of continuous repeated characters in the text field to be detected; based on the number of repeated characters and the number of continuous repeated characters of all the text fields to be detected in the text to be detected, calculating a similarity between the text to be detected and the comparison text.

2. The text similarity detection method of claim 1, wherein, The step of segmenting characters in the text to be detected into a plurality of text fields to be detected comprises: preprocessing the text to be detected to extract characters in the text to be detected; acquiring a comparison unit and a symbol unit in the text to be detected, wherein the symbol unit is smaller than the comparison unit; based on the comparison unit and the symbol unit, segmenting characters in the text to be detected into a plurality of text fields to be detected with the comparison unit and the symbol unit as intervals.

3. The text similarity detection method of claim 2, wherein, The step of segmenting characters in the comparison text into a plurality of text fields to be compared based on the preset comparison length value and the text fields to be detected comprises: preprocessing the comparison text to extract characters in the comparison text; in all the text fields to be detected, screening a text field to be detected with the shortest length, and the length of the text field to be detected with the shortest length is a minimum character length value; comparing the preset comparison length value and the minimum character length value, and taking a larger one as a comparison length initial value; based on the comparison length initial value, segmenting characters in the comparison text into a plurality of text fields to be compared with the same size as the comparison length initial value.

4. The text similarity detection method of claim 3, wherein, The method further comprises: If the edit distance between the text field to be detected and one of the text fields to be compared is greater than the actual length of the text field to be detected, the initial comparison length value is reset until the edit distance between the text field to be detected and all the text fields to be compared is less than or equal to the actual length of the text field to be detected.

5. The text similarity detection method of claim 4, wherein, The step of resetting the preset comparison length value comprises: comparing the actual length of the text field to be detected with the preset comparison length value, updating the initial comparison length value to the larger one, and re-executing the step of dividing the text in the text to be compared into a plurality of text fields to be compared.

6. The text similarity detection method of claim 2, wherein, The comparison unit is a period symbol in the text, and the symbol unit is a comma symbol in the text.

7. A text similarity detecting apparatus characterized by comprising: The device comprises: an acquisition module configured to acquire a text to be detected, a comparison text, and a preset comparison length value; a first division module configured to divide text in the text to be detected into a plurality of text fields to be detected; a second division module configured to divide text in the comparison text into a plurality of text fields to be compared based on the preset comparison length value and the text fields to be detected; a combination module configured to compare each of the text fields to be detected with all the text fields to be compared, combine the text field to be detected with the text field to be compared when the edit distance between the text field to be detected and the text field to be compared is within a preset edit distance, and form a similar character comparison group; a comparison module configured to compare the text fields to be detected in all the similar character comparison groups with corresponding text fields to be compared, calculate the number of repeated characters and the number of continuous repeated characters in the text fields to be detected, wherein the step of comparing the text fields to be detected in all the similar character comparison groups with corresponding text fields to be compared, and calculating the number of repeated characters and the number of continuous repeated characters in the text fields to be detected comprises: acquiring a preset displacement step; based on the preset displacement step, performing at least one position correction on the beginning or end of the text field to be compared in all the similar character comparison groups; calculating the edit distance between the text field to be detected and the text field to be compared after position correction, and selecting the text field to be compared corresponding to the text field to be detected with the smallest edit distance as the optimal text field to be compared; comparing the optimal text field to be compared with the corresponding text field to be detected, and calculating the number of repeated characters and the number of continuous repeated characters in the text field to be detected; a calculation module configured to calculate the similarity between the text to be detected and the comparison text based on the number of repeated characters and the number of continuous repeated characters of all the text fields to be detected in the text to be detected.

8. A text similarity detection system, characterized by, The system comprises a preprocessing component, a comparison component, and a calculation component. The preprocessing component is configured to acquire a to-be-detected text, a comparison text, and a preset comparison length value, and divide characters in the to-be-detected text into a plurality of to-be-detected text fields; and divide characters in the comparison text into a plurality of to-be-compared text fields based on the preset comparison length value and the to-be-detected text fields; The comparison component is configured to compare each to-be-detected text field with all to-be-compared text fields one by one, combine the to-be-detected text field with the to-be-compared text field being compared to form a similar character comparison group when an edit distance between the to-be-detected text field and the to-be-compared text field being compared is within a preset edit distance, and compare the to-be-detected text field in all similar character comparison groups with the corresponding to-be-compared text field, and calculate a number of repeated characters and a number of continuous repeated characters in the to-be-detected text field, The comparison component is configured to compare each to-be-detected text field with all to-be-compared text fields one by one, combine the to-be-detected text field with the to-be-compared text field being compared to form a similar character comparison group when an edit distance between the to-be-detected text field and the to-be-compared text field being compared is within a preset edit distance, and compare the to-be-detected text field in all similar character comparison groups with the corresponding to-be-compared text field, and calculate a number of repeated characters and a number of continuous repeated characters in the to-be-detected text field, acquire a preset displacement step; perform at least one position correction on a head or tail of the to-be-compared text field in all similar character comparison groups based on the preset displacement step; calculate an edit distance between the to-be-detected text field and the to-be-compared text field after position correction, and select the to-be-compared text field corresponding to the to-be-detected text field with the smallest edit distance as an optimal to-be-compared text field; compare the optimal to-be-compared text field with the corresponding to-be-detected text field, and calculate a number of repeated characters and a number of continuous repeated characters in the to-be-detected text field; The calculation component is configured to calculate a similarity between the to-be-detected text and the comparison text based on the number of repeated characters and the number of continuous repeated characters of all to-be-detected text fields in the to-be-detected text.

9. A machine-readable storage medium having stored thereon instructions, the instructions comprising: The instructions, when executed by a processor, cause the processor to be configured to perform the text similarity detection method according to any one of claims 1 to 6.

10. A computer program product comprising a computer program, characterized in that, The computer program, when executed by a processor, implements the text similarity detection method according to any one of claims 1 to 6.

Images